Basic Transform Task Fails With Missing Header Error

Question

Hello everyone,I’m working on a workflow that pulls a file from Qualtrics’ Automation File Service and passes it through Basic Transform before moving on to the next steps. When I run the call on my machine using the endpoint from the Qualtrics API documentation page, the workflow runs perfectly with no issues. When the client makes the same call with the same file directly from Databricks, we keep getting the following error:TRF_1 - Record 1 has no field brand! Please ensure your source file has the header brand. Funny fact: the file does have the brand header. 🥲 Has anyone experienced this or have any insights that could help?

Lpena · Accepted Answer

Hello, Cami!This error, which occurs during the Basic Transform task in your workflow even though the 'brand' header is clearly in your file, generally points to a file processing mismatch or a subtle formatting issue rather than a genuinely missing header.Common Causes for this MismatchDelimiter Mismatch: The delimiter set in your workflow (e.g., comma) does not match the actual delimiter used in your file (e.g., pipe, tab). This causes the parser to incorrectly read the header row.Extract Data Validation: The Extract data task might not have strict validation enabled. If a file is invalid, it may pass the extraction stage but fail when it hits the stricter Transform stage.Hidden/Non-Printable Characters: There could be hidden characters, stray spaces, or non-printable characters in the header row of your file that make the system unable to recognize the 'brand' header correctly.Inconsistent Configuration: Changes made to the delimiter or other settings in the Extract data task might not have been synchronized and updated across all subsequent Load and Transform tasks.File Encoding or Line-Ending Issues: Non-standard file encoding or line-ending characters can also interfere with the file parsing process.Recommended Troubleshooting StepsVerify Delimiters:Confirm the delimiter used in your CSV file is an exact match for what is set in both the Extract data and Basic Transform tasks.Adjust the task configuration or re-save your source file with the correct delimiter.Enable Strict Validation:Turn on strict validation in your Extract data task. This helps catch file issues much earlier in the workflow.Check for Hidden Characters:Open the file in a text editor (like Notepad++ or Sublime Text) to inspect the header row for any non-printable characters, extra spaces, or unusual formatting.Re-save the file in a clean, standard CSV format.Ensure Configuration Consistency:Make absolutely sure that the delimiter settings in the Load and Transform tasks are consistent with any recent changes made in the Extract data task.Confirm Header Naming:Verify that the header names used in the Load task mappings exactly match the header in the file, including case and spacing.Next Steps if the Problem PersistsIf you have performed all these checks and the problem remains, it suggests a more subtle issue related to file formatting or an internal configuration that requires a deeper look.To properly troubleshoot this issue and securely access your account for a deeper review, please submit a ticket through your Customer Success Hub. This is the fastest and most secure way to get a personalized resolution:➡️ https://support-portal.qualtrics.com/

cami_mq · Answer

Hello, everyone!Thank you so much for the insights and suggestions. It looks like we were able to solve the issue. We had already validated all the basic details before — such as file formatting, delimiters, etc. — but after ​@Lpena's message, I went through the checklist again with the client and… guess what? The file delimiter was wrong.The solution is now in the final testing phase and will be pushed to production very soon.

Sign up

Login to the Community

Scanning file for viruses.

This file cannot be downloaded