💬  10 Days with Jay: Q&A with Jay Choi, General Manager of Research Core | Q&A | XM Community

💬  10 Days with Jay: Q&A with Jay Choi, General Manager of Research Core | Q&A

  • 18 February 2019
  • 51 replies
  • 82 views
Show first post

51 replies

JayC
Rank
> @alejandra_rojas_voce said:
> Hello @JayC
>
> This is a great opportunity to get to know exciting things about you and Qualtrics. What is the biggest challenge you have faced with Research Core at Qualtrics and how you and your team worked it out?
>
> Thank you 🙂

One of the biggest challenges I’ve faced is staying focused on the most critical priorities that will really move the business. There are a million things that pop up every week, and everyone feels like they need urgency. Creating structural focus on the 3 year plan that will truly transform our industry: that's hard.

We tackle this challenge by having a monthly scorecard of our most important initiatives, and writing our press releases a quarter in advance. At the end of this quarter, we want this press release to be true. And we work toward it every quarter.
JayC
Rank
> @KevinK said:
> Hey, I'm new to Qualtrics and looking to tighten up our hiring profiles - if you were building a job req for a hire that you want to train up as a power user on Qualtrics (entry level), what would you specify in the req? Given the visibility you have on the feature plan for the next 2-3 years, what skill sets etc. should we be looking for? (e.g. is python integration coming, for example?)
>
> Basically, as you've worked across dozens or hundreds of clients, what does the profile (or profile set) of a successful Qualtrics power user look like? Let's assume a perfect unicorn doesn't exist, what should we focus on - business analysis, writing skills, stats, coding background, etc? Likewise, are there any red flags you would call out?

Hmmm...this might not be the answer you’re looking for, but if I were hiring for this type of role, I would think about a couple things:

1. Storytelling (with data) - What matters more than great research is moving an organization to action on an insight. It’s all about finding someone who understands how to craft a story that frames the core issue, the relevant facts, and an inspiring course of action.
2. Business acumen - Answering the right question is important. A lot of times, the key question we need to answer isn’t obvious. Someone who is relentless in finding the root decision to be made will always be an advantage, because they are spending their time on the right areas.
JayC
Rank
> @AnthonyR said:
> Hi Jay,
>
> Two questions for you:
>
> As far as the research core goes, what upcoming additions / changes to the product excite you the most, and why? Would love to get some insight into your thought processes on how to drive the platform forward.
>
> As you work to move the Research Core product forward, what are the things you look for as you hire members on to the engineering side of your team? Is it people who bring fresh ideas to the table, people with tons of experience, some combination of these things, or something different entirely?
>
> Appreciate you taking the time to answer our questions!

Hi Anthony,

One of the projects most exciting to me right now is coming from our ExpertReview family of products. In the past, we’ve broken industry ground by being the first to add artificial intelligence and machine learning into a research platform. We plan to add significantly more to ExpertReview with survey quality analysis and the establishment of brand-wide sensitive data policies, and now we’re continuing to beat top competitors in the industry by developing a product that can help customers parse the quality of their data.

Another area we are investing heavily is our automation platform called XM Solutions. Our goal is to make world class research methodologies simple and accessible to anyone. Similar to how you don’t have to be a tax expert to do your taxes anymore, we hope to do the same in research. We want to enable individuals and organizations to build their own content on top of the platform.

It’s been incredible to see how the ExpertReview and XM Solutions brand has grown in just a few short months.
JayC
Rank
> @jpardicusick said:
> Hi @JayC,
>
> Thanks for taking the time to do this with us.
>
> I have a question about industry XM benchmarking. We're a B2B communications/technology business and I'm struggling with finding meaningful benchmarks and standards to use for goal setting, especially as it relates to response rates and XM-specific metrics. What advice can you give about what KPIs to focus on (besides NPS) and where to find benchmarking information?
>
> Thanks,
>
> Jen

NPS is great for measuring relationship strength over a long period of time, since you’re asking people to recommend a product or service to a relative or friend… aka, to the people who you trust the most, and who matter the most to you. In contrast, CSAT is typically a better measure for transactional engagements based on a single point in time. When the experience is still fresh in your mind, it’s great to have the chance to provide genuine feedback and let brands know how you feel. Customer Effort Score (CES) is another measure that focuses on how easy or difficult a task or engagement is. Obviously, we want to provide customers the easiest, quickest, and most enjoyable experiences when engaging with our brand.

As for benchmarks, we have some very exciting news! At X4, Qualtrics’ incredible customer event, we are announcing several benchmark options to help you better understand how your scores measure up. We’ll have benchmarks from both JD Power and Bain & Company running on Qualtrics very soon. We also have the XM Institute, run by Bruce Temkin, and will soon be making XM Institute Benchmarks (formerly known as Temkin Benchmarks) available to all of our CX customers. Lots of exciting things are happening to help you better understand your customers and your business, and to help you make the best decisions.
JayC
Rank
> @VirginiaM said:
>@JayC thanks for your thoughtful answers! I thought of another question, if you don’t mind.
>What made you want to work at Qualtrics and can you tell us about your journey to get where you are today?
>It seems like an incredible company to work for (and if I ever work there I’ll be a member of your Taco Bell fan club)!

Haha, the Taco Bell fan club here is certainly very welcoming. :)

Qualtrics’ culture of innovation and positivity are really what drew me in. I love seeing a company that treats employees and customers really, really well that can also maintain a breakneck growth.
PhilJohnston
Rank
Userlevel 2
Badge +11
My Research Core question has to do with whether there is a method to mass create multiple projects with the same questions (from a library or pre-existing set of questions) each with a unique project name and unique panel (set of email addresses). Thanks, Phil Johnston
LaurenK
Rank
Userlevel 7
Badge +13
Hey @PhilJohnston! This is a great question; however, our Q&As are less for support purposes and more for asking questions tailored to Jay's work processes and overall career! If you contact our Support team they will be able to dive deeper into this inquiry with you!
M
Hi Jay,
How can we measure customer journey success thru Q research core tool ? and is it possible to measure ( both X and O data) journey wise user experience ?

How can you map product experience with revenue and a method to quality on routine basis?
GeorgeIssa
Rank
Userlevel 4
Badge +21
Hello Jay Choi,

I have a question.

I really like the features of the Side by Side Question since it can really shorten the number of pages in a Survey, but I have a question to you the Qualtrics Team and the Research Core Team, regarding this question.

For Column 1, I would like to ask a simple Yes/No Question.

For Column 2, If the User selects No in Column 1, then the Ratings in Column 2 should not show.

Here is an example:
!
JenCX
Rank
Userlevel 7
Badge +11
Hi @JayC,

Thanks for taking the time to do this with us.

I have a question about industry XM benchmarking. We're a B2B communications/technology business and I'm struggling with finding meaningful benchmarks and standards to use for goal setting, especially as it relates to response rates and XM-specific metrics. What advice can you give about what KPIs to focus on (besides NPS) and where to find benchmarking information?

Thanks,

Jen
KatharineS
Rank
Userlevel 6
Badge +4
Hey @GeorgeIssa! This is a great question; however, our Q&As are less for support purposes and more for asking questions tailored to Jay's work processes and overall career! If you contact our Support team they will be able to dive deeper into this inquiry with you!
GeorgeIssa
Rank
Userlevel 4
Badge +21
Hello @KatherineS, thank you for the comment on my post, but I do not think Q Support can provide assistance on the request I made, trust me, this is not the first time I have asked this question or presented this challenge to this forum. This is an Enhancement / Feature Request for Research Core. Since this forum is to engage Jay Choi who is the General Manager of Research Core, I think this is a good question. Thank you.
JasonHill
Rank
Userlevel 4
Badge +6
Hi @JayC,

We at UVU have built an integration with Canvas LMS. It's working beautifully and provides grade pass back to the Canvas grade book. We accomplish this, in part, using the Scoring option. However, we currently don't have a method to grab the Total Score possible as default embedded data. Instead, we add a custom embedded data field and manually set the value to the Total Score possible. Could I entice you to add the Total Score possible as a default embedded data field?

Also, our integration is currently built using the Qualtrics and Canvas APIs, but we are in the final stages of the LTI development. I can arrange a sneak peak of this awesome sauce for you if you have something equally cool to share with me...could be anything from something awesome coming to the Research Core platform, or some cool skill you got.
C
Badge +4
Hi Jay

Any advice best way to share insights with non-technical users without having to train them on how to be a survey guru?

Is there an easy way in the research core which enables you to see who is looking at the dashboards we share and create?

Regards
PeeyushBansal
Rank
Userlevel 7
Badge +33
Hi Jay,

Qualtrics is said to be "one platform" for all kind of research. How you and your team managed to have all functionalities included within single platform?

Regards
Peeyush
Best Regards, Peeyush
U
Rank
Userlevel 6
Badge +5
Hi @JayC,

Thank you for taking the time to answer our questions. I have two questions for you.

1) Are their any improvements in the works for Results Reporting? Our office still prefers Results Reporting over New Reporting and it would be great if Results Reporting could receive the same features that have rolled out to New Reporting.

2) Are there any plans to better communicate new features for the Research Core as they roll out? Currently some features are communicated through the Community, others are communicated through the Product Update page (https://www.qualtrics.com/product-updates/?solution=rc) and others I just find on my own while using the tool. For example, the new Text IQ was announced in the Community and never showed up on the Product Update page. It is my responsibility to keep my team up to date on new RC features and this has become increasingly more difficult in the last few months.

Thanks again!
VirginiaM
Rank
Userlevel 6
Badge +45
@JayC thanks for your thoughtful answers! I thought of another question, if you don’t mind.

What made you want to work at Qualtrics and can you tell us about your journey to get where you are today?

It seems like an incredible company to work for (and if I ever work there I’ll be a member of your Taco Bell fan club)!
JayC
Rank
> @mattyb513 said:
> Excited to see you here, Jay. Coming at you with a question for a large trade organization.
>
> We have thousands of member organizations that we support. Traditionally all our interactions with them have been via email or phone calls, very personalized interactions, the DC way. We would like to move to a system that allows us to better capture their experience in a measurable way so we can course correct and let them guide our decision making. Essentially, we want to do better at experience management but have no guidelines for how to get there.
>
> 1) How would you think about evolving from a very person/relationship-centric model to a more scalable software solution without losing the personal touch?
>
> 2) What kind of questions do you ask to help people feel like their responses are not just going to be aggregated into a dashboard?
>
> 3) Who are some companies that have successfully done this who might be open to a short guidance call?
>
> Looking forward to picking that big brain of yours!
>
> Matt Bloomfield

Great questions, Matt. I will DM you for more info.
JayC
Rank
> @Cenzi said:
> Jay
>
> How can I get the most out of my dual subscription to CX and EX?
>
> Want to show how my staff metrics impact my customers experience but not sure where to start.
>
> Adam

This is a great question, Adam. I’m glad you recognize the importance of marrying EX and CX, because how we treat our employees very often directly impacts customer experience. There is a lot of theoretical analysis on this, but I thought I would give one of the more pragmatic examples I had seen.

One of our customers is an automotive company who has both CX and EX. At first, these were separate programs run independently at the company. When they first got their CX program, they correlated it with sales data and found a strong correlation between CX and dealership sales. The dealers with the highest sales also had the highest CX scores. They then took this analysis further and found that the dealerships with the highest CX had the highest employee engagement scores. While that was a powerful insight, I was really impressed by how they operationalized that insight.

Here are a few things they’re doing:
1. Aligned their CX reporting to their EX reporting - particularly around customer facing teams. Now managers can see their CX dashboards filtered in the same manner as their EX dashboard.
2. Increased the frequency of the EX pulses to quarterly, because annual measurement didn’t allow them to track changes and intervene with their teams.
3. Held weekly standups on their CX score. They literally sit around the table with every department in the dealership represented and solve problems in the moment
And took immediate action on key EX issues:
* Regular Trainings
* Sales programs
* Holding managers accountable for engagement on their teams

Taking these simple actions completely changed the culture of the dealership. Customer-facing Employees could see that the organization was taking real action to improve their engagement and addressed many of the issues they faced. They could also see immediately that the organization was taking CX seriously through their weekly standups. By seeing that the organization was measuring and managing each of these elements, it drove a cultural shift and, impressively, even created a sales lift.
JayC
Rank
> @mujtanibul said:
> Hi Jay,
> How can we measure customer journey success thru Q research core tool ? and is it possible to measure ( both X and O data) journey wise user experience ?
>
> How can you map product experience with revenue and a method to quality on routine basis?

One example for UX research is the combination of clickstream (O-data) and Customer Effort Score (X-data). Oftentimes we will A/B test a new feature or the location of a button in our UX research. We utilize the drop off rates to understand how many people interacted with a new features followed by an in-product Effort Score study. Customer effort measures how difficult was it to complete your task, whether you accomplished your task, etc. Through this combination we can understand what is happening and why it is happening.
JayC
Rank
> @Shashi said:
> Hi @JayC ,
>
> With the regular updates in the product and new feature request coming in, how do you manage your work along with your personal life ( i.e. your work - life balance ) ?

I think the most important thing you can do for yourself with regards to work-life balance is draw up some boundaries. Some projects are important enough that they are going to take up your time even when the work day has ended; that’s just life, and the results you get from putting in the extra hours can be incredibly rewarding. But it’s important to recognize when there’s nothing substantial that you can contribute anymore. Maybe the teams you need to work on a solution with went home, or it’s time to admit that the project demanding your attention just isn’t that dire. On particularly crazy days, I always make a point to shut off my phone at dinner so I can take some time to decompress with my family.

It also helps that Qualtrics deeply encourages us to take at least one big, exciting vacation every year to just experience something new. I take my vacation time very seriously. I even did a week-long technology cleanse last year, and it was incredible.
JayC
Rank
> @JasonHill said:
> Hi @JayC,
>
> We at UVU have built an integration with Canvas LMS. It's working beautifully and provides grade pass back to the Canvas grade book. We accomplish this, in part, using the Scoring option. However, we currently don't have a method to grab the Total Score possible as default embedded data. Instead, we add a custom embedded data field and manually set the value to the Total Score possible. Could I entice you to add the Total Score possible as a default embedded data field?
>
> Also, our integration is currently built using the Qualtrics and Canvas APIs, but we are in the final stages of the LTI development. I can arrange a sneak peak of this awesome sauce for you if you have something equally cool to share with me...could be anything from something awesome coming to the Research Core platform, or some cool skill you got.

Wow, that’s incredible! I’m always impressed with the integrations our power users manage to make on their own. It makes me proud for what our API team has developed and people like you can leverage it!

I would love to see you share this in the Community when you’re done - I’m sure we can all benefit from it! In terms of “Total Score,” I have to confess I’m not familiar enough with Canvas to make any promises.

As for our roadmap… DM me, and we can trade secrets. 🙂
JayC
Rank
> @Cenzi said:
> Hi Jay
>
> Any advice best way to share insights with non-technical users without having to train them on how to be a survey guru?
>
> Is there an easy way in the research core which enables you to see who is looking at the dashboards we share and create?
>
> Regards

Hello again! For your first question, I’d say the support site and Basecamp are great places for users to stop by if they need help getting started. We have lots of resources for newbies that don’t dive too deep into the product, meaning they can get the basics without necessarily being a guru.

As for your second question, that’s a great feature idea! I’m not sure if we have anything that currently fits that niche, since Research Core doesn’t cover dashboards, but we should have a feature like that if we don’t.
JayC
Rank
> @bansalpeeyush29 said:
> Hi Jay,
>
> Qualtrics is said to be "one platform" for all kind of research. How you and your team managed to have all functionalities included within single platform?
>
> Regards
> Peeyush

Hi, Peeyush! We keep our eyes and ears wide open for the challenges our customers want solutions for, as well as innovations happening in the marketplace around us. But it’s not enough to just do the research and have an enormous flux of information to sift through; I’d say our true key to success is our organization into the pillars of experience (Brand Experience, Customer Experience, Product Experience, Employee Experience, and then the Research Core at the center). This helps us stay broad in scope, while narrowing our focus team-by-team.
JayC
Rank
> @Alexandra_OPecko said:
> Hi @JayC!
>
> I am excited to be attending the summit again this year! I was so inspired last year that I made significant changes in my personal life and my career. Was it Dr. Lovatt? Alex Honnold? Arianna Huffington? Perhaps it was all of them combined...
>
> I started an online JavaScript course (to become a Qualtrics super user one day), and much to the shock of my family...I now take a beginners hip hop class. _(Yes, I did break out with some dance moves when I read what was happening on Fight Night! 😃 )_
>
> So, I was wondering: From where do you draw your inspiration? And how has this resource influenced your life?
>
> All my best,
> Alex

I’m so glad to hear that! I loved our keynotes last year, and I’m so pumped for the incredible guests we managed to get this year. And congrats on the classes!

For inspiration, I look at consumer products and consumer software. While B2B software like Qualtrics has historically been held to a different standard. I believe B2C solutions have made the user experience beautiful and delightful and I think that should translate into what we use for our business tools

Leave a Reply


Terms & Conditions