Hi ebclc,
If the payload from your CMS contains the contact info, you can just set up a condition to check the attribute containing their email and (if it is empty) their phone.
Once you've set up the JSON event listener in Qualtrics you'll be able to access the attributes for other events in the workflow. See my comment on this thread for an example (it was for a Javascript problem but you could apply it to a Conditional task as well).
Good luck!

Thanks bgooldfed for your help again. Using the condition was the right way to go, so thank you. For anyone else reading this thread, I'm adding below some helpful additional ideas provided by the Qualtrics support team below.
----
I believe you should be able to accomplish something like this using 2 different workflows (one for email, one for SMS), each with different conditions. The XM Directory task requires you to select the distribution method during the task setup, so this cannot be dynamically determined during the workflow execution.
In one workflow, you will select email and in the second, you will select SMS. This will mean that you will have 2 different JSON event endpoints for the 2 different workflows, and the API call would have to trigger both - then, depending on which conditions are met, it will actually execute the task in only one of the workflows. 
For the email workflow: after the JSON event trigger, you can add a condition so that the XM Directory Distribute Survey task will only be executed if the "email" field is not empty. This would look something like this (you can replace the field name in the condition with whatever your field name is for the email address field)!
On the other hand, for the SMS workflow, you can add a condition so that the task will only be executed if the "email" field is empty.