Hi hschacter ,
You might run this question by Qualtrics support for a definitive answer. What was the domain of the made up address you sent this test distribution to?
Does your Qualtrics account have XM Directory? If so, did you look up the history of the email in it?

Thanks for your reply! I contacted Qualtrics support and so far haven't had any luck with a solution. It's happened both when I intentionally add fake gmail and yahoo accounts to my contact list.
How would I know if my account has an XM Directory? I've downloaded the distribution history directly from my survey's distribution page and it just says "Email sent" without any indication of bounces.

You would see something like the "Directories" tab in the attached screen shot if you have XMD.
Seems odd that an email that is known to be undeliverable would not show up as such in the Distribution History. I'm not sure what goes on between email servers and Qualtrics that would cause this; perhaps data is not coming through on the Qualtrics side to update the reporting in your account for some reason. (I haven't experienced your exact same problem myself, or at least it has not been apparent, but I have noticed other odd inconsistencies in the Distribution History reports from time to time.)

Ahh got it. I do not appear to have directories. So strange. I'll try following up with support again! I really need to know that I'm able to identify when my email did not reach respondents...

Thanks again!