So for me this was learned first in practice, then confirmed by Salesforce knowledge base article 194090. When a sandbox is created/refreshed, custom connected apps configured in production will be not be copied, even though they're part of the configuration/metadata. This is true for full sandboxes as well. The standard connected apps (Salesforce1, Chatter Desktop, etc.) will obviously still be there.
The KB article also pointed out that you can't reuse the same API name when you recreate the apps. It's probably the result of this known issue.
These oddities combined with the still confusing UX with connected apps - create in one place (Create > Apps) and edit in another (Manage Apps > Connected Apps) made Connected Apps still feel like a beta feature.