Migrating Your Supplemental Data Source Implementation | XM Community
Skip to main content

Table of contents


Introduction

Supplemental Data Sources (SDS) are going through a major update while we prepare to make the feature generally available. This post details what is changing and how to migrate legacy SDS integrations to the new SDS implementation.
 

What is changing?

Key changes include the location of where the SDS sources are stored/managed and the size/usage limits. SDS will be managed and shared via the Library rather than through Data & Analysis on Imported Data and Survey Projects. Limits are listed below:
 

Name

Limit

Notes

Max SDS sources per brand

200 SDS Sources

  

Max SDS size 

10 MB

  

Max records per SDS

100,000 records

At least one search field per record must have a value.

Max fields per record

30 fields

  

Max field size

200 bytes / characters

  

Max non-search field size

 5,000 bytes / characters  

Max usage rate (Brand)

30 requests per minute

For creating and updating sources.

Max usage rate (User)

10 requests per minute

For creating and updating sources.


What is the timeline?

  • November 30, 2023: An email was sent to Brand Admins detailing the migration, along with projects that use legacy SDS.
  • February 29, 2024: Implementations on legacy SDS will no longer function as expected and should be migrated to the new SDS.

What action do I need to take?

If you are using legacy SDS, you will need to take action to migrate existing integrations or they will no longer function as expected.

  • >Required] Enable permissions for users and groups who need access.
  • /Required] Migrate legacy SDS sources from Imported Data and Survey Projects to the Library.
  • /Required] Migrate Survey Flow elements.
  • /Required] Migrate Autocomplete questions in surveys.
  • /Required] Migrate Lookup Tasks in Workflows.
  • /NEW-Optional] Setup automatic updates for a new SDS using the Load to SDS Task in workflows.

What alternatives exist?

Due to the limits on the new SDS, migration may look different for each use case. Below is a list of our current recommendations based on use cases we’ve seen.

  • I want to enrich my response data for use in reporting and dashboarding. I do not need this data during survey taking (i.e. for logic or piped text).
    • If you don't need the lookup data during survey taking, you are performing a JOIN and should look into onboarding onto the Data Modeler. SDS will not support storing embedded data via Survey Flow beyond the maximum of 30 fields.
  • I want to perform lookups against XMD contact data during survey taking.
    • Use the fully launched and supported XM Directory features. XMD supports authenticator lookups allowing XMD data to be pulled into a survey.
  • I want to perform lookups against person data during survey taking.
    • Use EX authenticators for accessing person data.


Enabling permissions

SDS permissions are not enabled by default for all users. Brand Admins can follow the below instructions to enable permissions for users and groups

Note: Once permissions are enabled, it can take several minutes to percolate through the system. If the user does not have access immediately, check again later.
  1. Verify that the Autocomplete with Supplemental Data permission is enabled. This should already be enabled for everyone by default, but it’s possible that permissions were changed. These are required in order to use SDS and with Autocomplete questions.
    I46MyVYshlyQ1BHoDfvmuogJ7TPxsRc-T1k2TGjUItSKGFrGX43MvYIZDI8MAgNiVEXMXOYkHwTWa_qGdYjiOcKE-ziMkR8811KH5YBMEMV_lFTeQejYnC3ks8QLXZ2HYCCor0zPvaGIycD4xPPAgA
    • For users:
      1. Navigate to Admin → Users
      2. Search for the user you want to enable the permissions for and open user settings.
      3. Enable Manage Supplemental Data and Use Supplemental Data and save.

        Zb6gmlj9cIIDMDS59k5WC7437E36YYho_1GaTfje6A8RKYUlyrqLLIZA9aoB8ylGWgm0IevLVPY4eG8m1YDCRFTEPRE1FT-1jgIxkGy13CObldMP9rQdDSjy48bkIgsip0CjQ2zb2Bk77XaZVin6rTQ

    • For groups:
      1. Navigate to Admin → Group Types.
      2. Search for the group type associated with groups you want to give access and open group settings.
      3. Enable Manage Supplemental Data and Use Supplemental Data and save
        Cx4J83-adnY6HPGFjwEuPh78GudaczxbyPzuravsGyfQDpY0e0_Lut5nbrl9Ol8kbsVOjp68iFJfY6Hi_slrELk4GbJQ-ojy9wlQizVoLf4_ghbxrijlIuQUFzs02n8RuFMjU1jasKKoVBHnzttfMxM

 

Migrate supplemental data sources to the library

Prior to migrating Survey Flow, Autocomplete, etc., you will need to migrate the datasets that these features consume from the legacy SDS from Data & Analysis to the new SDS implementation in the Library.

  1. You should have received a list of recently used projects that use Supplemental Data Source for Autocomplete or Survey Flow in an email from Qualtrics or your XM Success Manager.
  2. Identify the dataset that is being used as Supplemental Data Source.
  3. Export Imported Data Projects to CSV or TSV. If you already have access to your original dataset in CSV or TSV format, you may skip to step 4 to upload your dataset as a new Supplemental Data Source. ​​​​​​

    Note: If there are impacted projects that are inactive or you no longer use, you can feel free to disregard migrating these. It is up to you which projects should be migrated. Note, however, that any projects that are not migrated will not work as expected once legacy supplemental data sources are no longer supported.

    Note: Setting up supplemental data sources from response data on another survey is no longer supported. To accomplish similar functionality, you may export response data and import as a supplemental data source.​

    1. Navigate to Projects.
    2. Select the Imported Data Project or Survey Project that contains your Supplemental Data Source.

      Tip: Use the search field to find impacted projects. You can enter the survey name or search by survey ID.

       

    3. If within a Survey Project, navigate to the Data & Analysis tab. Take note of the search fields that are being used.
      1. Select Tools → Supplemental Survey Data…
        SRFJRL2r78DrLT16GQqNvqfdC23hs5P_W09tbZIyDiYCAuED-iSMPz7eOOa21gIqjaPoOhEnmfNB3TeIElxiY6q9jySDWbVIXHXYnCkIzA5sFPYV65kp9kp5GsgNwPnGiHw3XSnE8SHr-N4HjAHketg
      2. Take a screenshot or take note of the search fields. You will need this when setting up the new Supplemental Data Source
        o0lvveoTJ7ck6WO_fpVfDccztSc4mptOkE5fC-M515XjdJCf1Htn13_m1yMfRJEkd1Z-14Zz9GgNCdrle14CNksB4ImzzFF2gRAPTJUp5RiQK8lv7tXxDks_YE2JOM1d94-SuGU2xxVqkTUHYj082bU
    4. Select Export & Import → Export Data…
      zsJzuZMerj0zq5M-thpsOx_Ng_bHyqEAhaKLpCtVMEMn_2HauZ3WHFScgAwhu0UWlr4XvGRhWOMxVFi_GIh4njdsqETUkDB4YRnlcAnGr1qTlRTz_DONzgb-plYoL8l6CsCiCIsIlyiHTzS2sImEm-M
    5. Select to export as either CSV or TSV
      gTwj4Xdq7cClXPvC8hBN841TKGcAM33C1bKPK7oQ2AwST58jHyIsXAc2s4_ByD2VLXcxcFT_MMonFxC9Jh1y2xnBtscZgshD6r_LoJiryp9EzdR_OQwg0ZomCRHBDl7125iDPLjmjFgjHid9TNbmKS4
    6. Select Download
  4. Create a new Supplemental Data Source
    1. Navigate to the Library or the autocomplete question or survey flow element in survey flow.
    2. If necessary, trim down the dataset to be less than 100,000 rows, 30 columns, and 10MB file size. Note that any selected search fields must be fully populated in the dataset, otherwise those records cannot be indexed and will not be used for searching.

       Note: If you want to share the SDS with other users, be sure to create it in a group Library. See more info under Sharing.

    3. In library select New resource → Supplemental data source
      iza95Ob7xreohRYIfdn7Y7YnHI3DkX5gN1Nt_J_sRBXcEE-5tsKU1aI1kvNMwhwNxJoiflkwheWI4w35rV3Yu66OrH2PMAxfzh5NKBVhsqwRaZfdwknHYugLN1WC9vlrgB2eoG7MolXEXz5p2srXHJo
    4. Select a file, name the data source, and select up to 5 search fields. You will want to use the same search fields from your previous SDS source in order to ensure active projects referencing the supplemental data source continue to work as intended (see step 1).

      tvVgRq6VePgbvK5wS6N8Qu6Rv6v1LK5MyyvoAuPUP4XzOAs27ANC_yxEuhU5q0plFWWrDKYHM2f57xtF4-jBxX3SQUchofIWWnN5oL7scuhiJFW2E2dyWR5a32qjfnWcuveokDLgMVPJOmlfVa5BhcU

    5. Select Create
    6. Your Supplemental Data Source is now ready to be used! 


Sharing

Legacy sources operated with project sharing. Sources are now managed in the Library. To share a source, create or move it to a group library. All users in the group will then be able to access the source. For additional guidance on library sharing, see Groups.

 

Migrate Survey Flow

  1. Navigate to Projects.
  2. Select a survey using the legacy Supplemental Data Source in Survey Flow.

    Tip: Use the search field to find impacted projects. You can enter the survey name or search by survey ID.

  3. Navigate to Survey Flow.
  4. Notice the banner at the top describing the change. Selecting these elements will show you which survey flow elements are using legacy supplemental data sources. Each element will also have a banner, making it apparent which ones are impacted.

    skiPji9B6w7XPikfzq3DKJna5l31GG_TCUX56SD7-wJYF4PhqrzgncU9ldjcXC9MdT0-RPaOYT3e29NEQv2ESuZcX043FM-Bu2WvQb3CsP5q-VBuoats03NPwocHJLpFK0iLVBghQqsT2wfHXEi2ZCE

    slQZWNX1gQ1WQcA4V7nKxf9C-5oU1OcZcQ8ampWErBBf4R7saTHhh6vXDNg3ZifxAWRLivzfqZPuQ_sl4hn4itOW8_XRg_Y_MxXU03t60bI7RcWCbKPQ-V2VqWL6qddzKM2GiKV28EDv1IITzL4OmOk

    Tip: Toggle Show flow IDs at the top-right to display the flow ID at the top-right corner of each element. You can then use your browser’s search functionality to quickly find the impacted elements.

  5. From the banner in the survey flow element using a legacy supplemental data source, select Create a new supplemental data survey flow element below. A new element will be created directly below.
    CF_2zLbyqFBoDYdYpiwi3-cML6i0Gr13WTBSoGS-rOJnldGNX_Sk9a5_pABBHsS6F7P8QJhuhsKJnQfwGrAYhe-WkSKmLkun94z9MobVQPijX5-HcGpYkumB_Tjf3-l3gkWM9WMZF2XPtK6Ncw6aCtE
  6. Comparing with the existing survey flow element, update the new element to match your use case with a new supplemental data source, selected from your library. In some cases, you may need to modify which embedded data fields are used and which columns are being searched on. This is particularly true when you’ve needed to modify a legacy data source to fit within the new limits.
    2FT0DfuoaVmQ638YsSSfn38xY0PwLFJJ8eNFyjIbOUc19NCygNfTD_yiyhF4r9RQqJHEvKdV4a1im1AiC8RrlrklUX7TOhg9twm8Ued7HNnNGWNZGzVg5x0m-ZLFQgbltc0UtJ2HK-9lEYM0Kn1nFfw

    Note: All conditions must match a record in the data source.​​​​​

  7. Delete the survey flow element that used the legacy supplemental data source (with the orange warning banner indicating the switch).
  8. You can verify that you have finished migrating all your survey flow elements once the banner at the top from step (4) is no longer showing.
  9. Select Apply to save changes.
  10. Preview the impacted survey and verify everything works as you expect.
  11. Publish the updated survey.

    Tip: The supplemental data source can be created on the fly from within autocomplete or survey flow by using the “Create new supplemental data source” option in the source selection dropdowns. See the full instructions to migrate those features below.


Migrate Autocomplete

Warning: In order to maintain reporting data in dashboards and other survey configuration, it is important that you do not delete the old autocomplete questions and create new ones. Instead, we have provided a convenient way for you to migrate an autocomplete question using legacy SDS to new SDS.

 

  1. Navigate to Projects.
  2. Select a survey using the legacy Supplemental Data Source in an autocomplete question.

    Tip: Use the search field to find impacted projects. You can enter the survey name or search by survey ID.

  3. Notice the banner at the top of the survey describing the change. Selecting these questions will show you which questions are using legacy supplemental data sources. Selecting a question ID will automatically select that question within your survey.
    rx0OnD2IHfG5oeF8bQuLAflh2InCDimUMc1JJumv5h1v9GhclKkKPMnMT5JhFi7vrW2Grat_aUvdTAnhvtfPSPlHqLzgMnP9D-SXgyRbMjDCBG_gvF4jSd0APX9A_jXovGK8T7_An91AfU6Tk1fLOD0
  4. In the sidebar, select Manage autocomplete.
    5RVbybo0xF9uTQDM6kzIMK3Awlrr4WWWUNJpnNoxFJ5NWfN9uG1TbYpto5hSqouky4XaixGV6bF9cPJDMzkUy2uosdD9FQ23CU1_UMxcuBuVgledDzETrs6lEJTXb0zftSDGY6xBNqWuR5IbcNnJZNY
  5. Notice the banner describing the change and select Switch to the new version. 

    YYbGOdNkb-ILFCn6hpbJmSUDCHfHv8j94dWFLvBi55cUg-zPKtq8fL2uG9k6mWsjWpHTHc2K4Zae_I234XzU-bfKWG2YPNKiMJFxrDLPFFffegbL9P0ltKqe3DaY8NBctUJFTRIEdH5KkhFGfqXZEfs

     

  6. A split view will be presented to aid in migrating legacy autocomplete configuration. On the left, is a read-only view of the existing configuration. On the right, is the new configuration.
     

    WyPMIfwOmS1qRSeQvrLbHl9Pct9XUTMlN9CpWbOvs5ar6Ug4pXFw1DLNwXMk7kFI6H7oNJ2202ER8X5YOLh-Pql5ScysR0DSNTnGqVsg6nz76EEjsFkgJ_MgpAQo-tdwS5nrZtKk1uyJ3ghkxRI68D0

  7. Select the supplemental data source you migrated earlier that mirrors the legacy one. If you want the same behavior, select the same configuration options. In our example below, we have a new source that is named the same and has all the same fields as the legacy source. When you are finished, select Migrate and your new configuration will be saved.

    WyPMIfwOmS1qRSeQvrLbHl9Pct9XUTMlN9CpWbOvs5ar6Ug4pXFw1DLNwXMk7kFI6H7oNJ2202ER8X5YOLh-Pql5ScysR0DSNTnGqVsg6nz76EEjsFkgJ_MgpAQo-tdwS5nrZtKk1uyJ3ghkxRI68D0

  8. You can verify that you have finished migrating all your questions once the banner at the top of your survey from step (3) is no longer showing.
  9. Preview the survey and verify everything works as you expect.
  10. Publish the updated survey.

 

Migrate Lookup Task in Workflows

Note: If you don’t have existing workflows that use the Lookup Task, there is no action required.

If you do currently have workflows that use the Lookup Task, the following actions are required. The migration of existing Lookup Task instances in live production workflows involves two key steps:

  1. Update the Lookup Task configuration to point to the new v2SDS tables.
  2. Update piped text names in workflow conditions and tasks consuming the piped text exports generated by the Lookup Task.

Before completing each of the above steps, you will first need to follow the instructions above for creating the new SDS in Library and Enabling Permissions for SDS. After this is completed, you can proceed with completing the steps above using the instructions below.
 

Update the Lookup Task configuration to point to the new v2SDS tables

For each existing workflow that contains the Lookup Task, you will need to update the Lookup Task configuration to point to the new SDS source instead of the legacy Imported Data Project or Survey Project where the lookup data was previously stored. Follow the procedure below to update your Lookup Task configuration to work with new SDS sources you created in previous steps above.

Tip: Alternatively, you can make a copy of an existing workflow that includes the Lookup Task and then update the task configuration in the new copy followed by testing it. After all looks good, you would then disable the original workflow that has the old configuration and start running the new workflow you created by copying.

  1. Navigate to a workflow that includes an instance of the Lookup Task. Workflows can be located in a Project’s Workflows tab or in the Stand-alone Workflows location.

    T0MAt74WWbZLIUN0HtRAHDXcikb-yF7hZoZZ_YkRgYJk0vlA3_fBVUxdQAnfsv2eTsfeTnp_VLrb1szRCmF3Q4JCmoIdVVWqpLoDh2tWW2X0ft_DhO89g3jxVMOlUx7HdQHH6YKQesh9WFqZargOHY8

  2. Open the editor view for the workflow that contains the Lookup Task and then click on the Lookup Task to access the configuration field that allows you to set the source where the lookup data is stored and the search query details.
  3. Modify the ‘Select SDS lookup table’ field to point to your new SDS in the dropdown.

    cfmQ093n28zbckodP_G6J446i-g_BOeBGnSPIFRq1bpRTaRAcqXQQalKhqL_GIs6-Aaf72bw672ZdSGE-ynWd4wd9PBqJEwoIyOg1Yls-OC-dFDfSR3Ju_oyY93k94wgu9SjmqsZFKKXim-S0CUkoa0

  4. Review the search query configuration fields to make sure they are set to the correct columns in your SDS source and are using the correct query operator and lookup key. After you’ve completed this, save and close the Lookup Task.

    5DVjWoHKdmE4gmWM48Rq-2qrieTJ8MhyggHWXhxKXeaQJMyqqrCH7Zlt9HaRwfmiNyXo_9izsN4-x9ZettbLrTEcmWjaf1jy51H9QXgqWfzXsb5cIk44QztVBQwnI60LBsxSB2YtykLAp6WIaJfBPl0

  5. Finally, test the workflow to make sure the Lookup Task can access and query the new SDS source. If you created a new workflow by copying your original that includes the old configuration for Lookup Task, make sure and disable the original workflow so it does not keep running and encountering errors when the legacy SDS sources are turned off.

Update the piped text in workflow conditions and subsequent tasks that reference the  Lookup Task exports

When the Lookup Task runs, it generates piped text variables that can be referenced and used in subsequent conditions and task configurations. The migration to the new v2 SDS tables to store the lookup data will change the piped text variable names generated by the Lookup Task. For example, piped text names generated by the Lookup Task using the legacy IDP/v1SDS include a column number (example: ~{ch://OCAC_rpqQqC7eakALoBL/00001}) whereas newly generated exports using v2 SDS as the lookup data table include the column name instead (example: ~{ch://OCAC_rpqQqC7eakALoBL/age}). If you currently have workflows with tasks or conditions that reference the piped text exports generated by the Lookup Task, you will need to update these to use the new name that includes the v2 SDS column names. Please watch the demo video here that demonstrates an example of updating piped text references that depend on the Lookup Task.


 

Automatic Updates 

SDS sources can receive automatic updates via Workflows using the new Load to SDS Task. For example, you may want to regularly sync a file from an SFTP server or Google Drive. Prior to performing the below steps to update an SDS source, you need to create an SDS source in the Library, following the steps above.

Note: The Load to SDS workflow task is in Preview and available upon request for existing SDS customers.​​

 

  1. Navigate to Workflows.
  2. Select Create a workflow → Started at a specific time (scheduled).
    RcojMqiIOWtOlRkN3BA5cKv1RCX8oCHFUGzV-Pvr61CgCILHrozTFhPY2LqM-TbeYanuIAeAcgirY1wELe0DchTb6W28aElMRYRe6O8tVDJFSO2VZ_42QzUvLiZHFcfEaJ57viIq1WJYETDpdQP2YVU
     
  3. Provide an appropriate Name and select Stand-alone workflows.
    IEnCYu-f142z4rwDt2gN_sPZsYzEHXbk3fkgly3J7c6WYXXzt_0TYBQ6JTDnxbspigekruSkUoGGcOKMep-2Ev4vHh6MtDPeyynjDLCcpwv-3V2u6QyX4g0nK7N1AJ_yyzEQLXrpsDhBeX7Dad-b2VQ
  4. Specify how often you would like to update the SDS source. In the example below, the schedule is set to update weekly on Mondays.

    2VBr89_FtOViaWlMf1MHi0IQZ4dItWTHJCdFDQvAFnl7JhHjrNYG9_neiZgY1CJ-ciOqVsVGrgZu8PHlUin1Ph3DJsZKH9HswPl5KE99awPV1M6ux3HYZ9tqH_lXNQblSHqadAcDX3w4BNRZSoUvtHY

    Note: The Load to SDS workflow task is restricted to run at most one time every 24 hours. We recommend scheduling your workflow to run no more than daily.​​​​​​

  5. Below the schedule select + → Add a task.
  6. The first task should typically be an extractor task to get the data you want to update your SDS with (see available extractor tasks) When selecting an extractor task, it is important to recall that the data you extract will overwrite the prior SDS data (not append). Many extractor tasks allow you to export all data, but if you select a file that only contains new additions (such as Extract responses from survey), your new source will only contain these new additions and not the original data. For this example, we will use the Extract data from SFTP files.
  7. Fill out the form for the task.
  8. Below the extraction task, select + → Add a task.
  9. Select the Load to SDS task.
  10. Select the data source for the task. This is going to be the extraction task we set up earlier. Then select the SDS source we want to overwrite with the data from the extraction task.

    Rsru37GCT8zDBFsUeBNHazMD77ddADY1_K_LWdUMwzOHNIteVqgRTDq3-InQHRSAwkkYqjfAmL3qUfzI1OPESheZKisciiATAJyvt_rlZ9xqa4y9vIhnBol5QeXNPvBAzfKUBlrWzKkbgw1wBnUvxS4

     

  11. Select the fields you want to map from the extraction task file to the SDS source. These will typically be the same unless the datasets differ (for example, the SFTP file may exceed the SDS limit of 30 columns and you would need to select just the columns you want to update the SDS source with).

    5UWUjdHQZxCKgH-baIVyGg9esWuj523aFj2VN7CXFyoKlPfV1jQ-rHI8jdKWYi6tVm_jngLu1q8BuwHS6PNiJRr3nH64vtMwZcq7UWgN7BJKADLPEETbtvwvo5yWPloniabOxKbXmXHocux-kFVaUDE

  12. Select Turn the workflow on.
  13. Your workflow will then look something like this:
    j3AzL8D8KPcq_ZVXnL9yD_kd0hbPedLH3gtXJIOmoBGbOTvzQPUz9o2frNlCBqztzUx5_HKYmwufo1V4fTFAitOWFJKop9ZatxiLyPInptVN7savfjfeGkaul0hs4Zp-y9kEX7TyfE4SJcbQqi37Ab8
  14. You are all set! Your SDS source will now be updated weekly with the data from your SFTP file.

When is this being enabled? I got the email but the feature is not in my brand yet.

When is this being enabled? I got the email but the feature is not in my brand yet.

Hi @TJMeersman999 -- just wanted to let you know we saw your question. The team is looking into this for you and will reply back with more info very soon! 

Hi @TJMeersman999 we have enabled the feature for all brands. You may need your brand administrator to carry out the steps described at the end of the post. 

@asheeshr @Michael_Cooksey 

 

Earlier it was not working, but it is working now!! Thank you!

 

Also, side note, if we have a workflow pulling in a file into an imported data project through an SFTP workflow, are we going to be able to do that with the library message or the survey flow option? Our data for the supplemental data project is updating pretty regularly. 

 

Best, 

TJ

@natmaxwell  I don’t see any APIs to update these SDS in Library. We have workflows to update imported data so SDS but with this new approach there seems no way to update SDS automatically on regular basis? Can you please suggest.

Hey @KimothiSaurabh @TJMeersman999 Work is currently in progress to build a Loader task that will allow for an automated update of Supplemental Data Source available in the library. We expect it to be available in early part of Q1, 2024.  

There seems to be some confusion about the availability of this feature. I sent a request to the QPN assistant to have them enabled in our sandbox and I was answered this :

Hi Vincent,
Thank you for submitting the below request:
manage supplemental data / use supplemental data

I just took a look, and it seems like this is a feature that is not yet generally available. Thus, I can't request enablement of this feature for your sandbox.

@vgayraud the feature is currently under preview. This post is to help customers using legacy SDS to migrate. We expect the feature to be generally available in Q1 2024.

Hi,

We received an e-mail that we need to migrate the SDS to library.

So, we tried doing it.

We are now stuck in AUTOCOMPLETE section, where under MANAGE AUTO COMPLETE, earlier when the checkbox - FITLER AUTOCOMPLETE SUGGESTION was enabled, it let us set condition against embedded field. Now, in the new version, when I enable the checkbox, it doesn’t give the option to select the embedded field. Kindly help.

Regards,

neha Tank

Hey @KimothiSaurabh @TJMeersman999 Work is currently in progress to build a Loader task that will allow for an automated update of Supplemental Data Source available in the library. We expect it to be available in early part of Q1, 2024.  

 

@asheeshr  Can you provide an update on this loader? I have not moved my SDS workflows yet, without the loader It will make manual work for me 

We received the second notice to migrate our work, but in the documentation, it said we would receive a list of projects that needed to be migrated.  I don’t recall ever receiving anything and our other brand administrator doesn’t either.  I put in a ticket/question with our Customer Success Hub and never heard back so submitted a new one today.

Is there is a way for me to find out what projects use SDS as I’m not aware of any, but don’t want to miss anything while I have the opportunity to make this process smooth if we do have any.

 

Thanks for any insight!!

Hi @TJMeersman999, @KimothiSaurabh​​, @twesolek, there is now the Load to SDS task available in workflows to automatically update sources. This post has been updated with details on how to get started.

Hi @MeganZich, I checked and you do not have any legacy sources to migrate.

Hi @Neha Tank, I apologize for the issue you were facing with the Autocomplete question type. This issue should have been resolved since you last saw it. If you are still experiencing issues, please file a ticket through the Customer Success Hub so we can take a deeper look and get this resolved for you!

@natmaxwell  I am currently using the “Get a Record from Imported Data Project” API - https://api.qualtrics.com/96815b42da9fb-get-a-record-from-imported-data-project will this API continue to be available. Can you please confirm this?

 

Thanks,

Rakesh

@rakeshdayalan19 That API will continue to be available. Imported Data Projects will not be altered, nor their APIs, as part of these changes to Supplemental Data Sources.

@wesp , As it is mentioned in tutorial above that in workflow, there is no way I can EXTRACT all responses from survey that had the legacy SDS and load it into library, what do you recommend us to do as we cant use Extract responses from survey (It being delta in nature). We are using this SDS in Auto manage complete so we need the entire dataset available in Library SDS at any time.

Awaiting your reply.

@Neha Tank I apologize for the inconvenience and understand this migration is quite painful for some. I’ve been investigating options and, ultimately, this comes down to each use case. One option is to export the response data and upload it into an SDS in the Library.

In some cases, we’ve seen customers “chain” surveys together. So they might have one survey that takes in data. That survey might act like an internal intake survey. Then a second survey might be used with Autocomplete to lookup information from the first survey. Although, there isn’t explicit support for this, I did find a workaround by setting up multiple workflows. Here’s a potential way you can accomplish this:

  1. Create a workflow to load responses from a survey into an Imported Data Project.
    1. Add the Extract responses from survey task.
    2. oOptional] Add the Basic Transform task if you need to transform the data or if you want to filter out rows of data with validation rules to get within the SDS limits.
    3. Add the Load into a data project task.
  2. Create a second workflow to load the data from the Imported Data Project into a Supplemental Data Source.
    1. Add the Extract data from data project task.
      • Select the option to Export all data. The Load to SDS task completely replaces the data in the SDS rather than adding to it, so we want to do a full update each time.
  3. Add the Load to SDS task.

Note that the Load to SDS task may only be run at most every 24 hours. The Extract responses from survey task only exports responses that are at least an hour old.

Thanks @wesp for the response. Let me try it and let you know if I face any problem.

@wesp , I have made all changes and have created the workflow. You had mentioned that LOAD TO SDS task may only be run at most every 24 hours. I loaded last at 2:20 pm IST yesterday. Does that mean it considers 24 hours from that time or is follows the clock ? I tried running workflow now (11:40 am IST), it gave me same error. Would I be able to load only after 2:20 pm IST today ?

Thanks in advance.

@Neha Tank, I’m glad to hear you were able to get the workflows set up! The 24-hour limit has some a buffer, so if you schedule the Load to SDS task to run at a specific time each day (e.g. 2:20pm), then it will run without issues at that time. The limit only applies to successful runs, so the failed attempt due to the limit at 11:40am will not impact the next run at 2:20pm. Please feel free to reach out if you experience any issues!

@wesp The flow of new SDS works fine except for 1 problem. Whenever I load data into IMPORTED DATA PROJECT, it inserts a blank record by default and doesn’t let me delete this record. And then the LOAD TO SDS fails stating the search field is blank for a record.

Till yesterday, it was letting me delete the blank record in DATA & ANALYSIS. Today it is not letting me delete the record. I also tried populating test data in that record to make it non blank, still it recognizes it as blank record. 

Kindly help ASAP as the data needs to be updated to SDS by EOD. Thanks in advance.

@Neha Tank I’m sorry to hear you’ve run into issues. To get the best customized support for your case, I recommend opening a ticket with Customer Support. Here’s what I’d recommend looking at:

  1. In the Extract responses from survey task, make sure to select the option to Remove extra header fields from exported survey response file. This will remove additional headers in the dataset.
  2. Just prior to using the Load into a data project or Load to SDS tasks, consider adding a Basic Transform task with validation rules to filter out rows that are missing data that you expect.
  3. Because this is time-sensitive, you may also consider updating the SDS manually from the Library each day until you have the automated workflow set up as you expect.

How do I get access to this feature in my instance?

Hi @Aanurag_QC, all brands should have access. Brand Admins will need to give permission to users within their respective brands. If you are needing further assistance, I encourage you to reach out to Customer Support.

@wesp Thanks for your prompt support as always. Appreciate it. 

Leave a Reply


Terms & Conditions