💬  10 Days with Jay: Q&A with Jay Choi, General Manager of Research Core | Q&A | XM Community

💬  10 Days with Jay: Q&A with Jay Choi, General Manager of Research Core | Q&A

  • 18 February 2019
  • 51 replies
  • 82 views
Show first post

51 replies

A
Hi @JayC!

I am excited to be attending the summit again this year! I was so inspired last year that I made significant changes in my personal life and my career. Was it Dr. Lovatt? Alex Honnold? Arianna Huffington? Perhaps it was all of them combined...

I started an online JavaScript course (to become a Qualtrics super user one day), and much to the shock of my family...I now take a beginners hip hop class. _(Yes, I did break out with some dance moves when I read what was happening on Fight Night! 😃 )_

So, I was wondering: From where do you draw your inspiration? And how has this resource influenced your life?

All my best,
Alex
JayC
Rank
> @VirginiaM said:
> Hi Jay! I am gearing up for my first year at the X4 Summit and I am so excited to be a speaker this year 🙂 I have a couple of questions for you as well.
>
> * What do you find the most challenging and most rewarding about your job as the General Manager of Research Core?
>
> * Knowing what you know now, what career advice would you give to your college-aged self? (This is something I ponder a lot since I currently work at a university).
>
> Thanks so much!
> Ginny

Thank you, Ginny, we’re thrilled to have you!

The most challenging and rewarding are very related. The most challenging is that there are so many different directions we can take our product roadmap. The challenge comes from doing our own research to really find customer needs and pains and what we could do as a platform to solve them. A great example of this is our upcoming ExpertReview data cleansing feature that is coming out. We would watch our users spend hours and hours cleansing their data and as new responses come in they would struggle to incorporate the new data. And we wondered, well, what if we could leverage artificial intelligence to help flag and cleanse the data for our customers? That's also the most rewarding part - when we really dial into something that our customers find valuable and they can't live without. It's an incredibly rewarding and meaningful aspect of my role.

As for the advice I'd give my college-aged self... Always be grateful. Always find something you can be grateful for in any circumstance you are in. It changes your perspective on everything, and changes how people see you. I use this everyday and always try to see the best in things and be grateful for the opportunities I have.
JayC
Rank
> @GillesG said:
> Hi Jay,
>
> What's the most impressive / inspiring questionnaire you've seen recently? Looking for inspiration here (and I bet most people reading this thread could use some for their future projects!)

The most impressive I’ve seen was from a friend of mine in business school, who had spent years in research. Not necessarily for his questionnaire, but for his approach. He started by writing down the 3-5 most important strategic decisions he had to make. Then he literally created a powerpoint deck with charts filled with dummy data that he would use to answer that question. He socialized the deck and made sure the executive team agreed on the mock insights before he collected a single piece of data, incorporating feedback as necessary. Then he designed a precision research study with NO fluff questions. It was always short and clear because the output had been thoroughly designed beforehand. I learned a lot from watching how he did that, and he gained a ton of credibility in our company because he sought buy-in before execution.
JayC
Rank
> @AdamBoston said:
> Hi @JayC
>
> Myself and my colleagues are excited for X4 - this will be our first year and would love to know what to expect? Is it as good as people say? Do you have any insider tips to give? What should we make sure not to miss? If you were attending X4 for the first time what would be your key areas of interest? Maybe this is one of your hidden talents? 🙂

Well, I don’t know anything about a hidden talent, but I do consider myself quite X-perienced with the conference.

There is so much to do at X4 that it can be a little daunting at first, but I always give first time attendees a simple framework to think about:

INSPIRE - Make sure you find the activities and sessions that inspire you to be a better person and better at your job. Some ideas are:
* Mainstage sessions (Obama, Oprah, etc…)
* Key customer breakouts you admire how they operate (e.g., Patagonia CHRO)

EQUIP - Walk out with technical skills, strategic design, tips and tricks to solve your business challenges. Some ideas are:
* Basecamp training - learn how to use the product like a pro, designed around real customer use cases
* Design workshops - breakout sessions run by SMEs on program strategy and design
* Ask the experts - stop by the XM Garage or Ask the experts and ask for a consultation or a specific problem you are facing. You would be surprised at how they can help!

ENGAGE - find time to engage in the things you have a passion for. A few ideas
* Networking - find like minded people and hit built-in networking
* Hit the activities designed for networking like fight-night or the concert
* Choose an experience day and meet others

As for insider tips… the minute you get there, ask someone what the dream team is. You will be pleasantly surprised.
JayC
Rank
> @Sam said:
> Hi Jay,
>
> First, I wanted to let you know that I'm excited for X4! I attended the Research Core base camp classes last year, which were great, and I'll be attending the IQ session this year.
>
> I also have dashboard questions, which I hope fall in your area...
> 1. Have you considered not requiring the default time periods and using fiscal/company specific time frames instead? This would be helpful to us in dashboards and data & analysis.
> 2. Could you add a suggestion box (of sorts) to dashboards? This would be helpful to collect feedback from employees as we launch them further in our organization.
>
> Thank you!
> Sam

Hi Sam,

I’m glad you’re stoked for X4! We here at Qualtrics are working day and night to ensure your X4 experience will be even better than your last. :)

I love both of your suggestions. There’s something to be said for being able to break out data easily by company standards, and while our dashboards do use some quarterly filters, we could always stand to improve on that capability across the product. And as for the suggestion box, what better way to gauge whether an employee experience program is benefiting your workforce than by checking in with them every step of the way? In the meantime, I wonder if a regularly scheduled Lifecycle survey, or a prominent feedback survey link in the dashboard, could empower your employees to provide this sort of feedback. But your insights are really helpful, thank you for suggesting them!
JayC
Rank
> @Mol said:
> Hi Jay!
>
> I'm really excited to be again at the X4 summit and learn about the roadmap and future of the Qualtrics XM platform.
>
> I wanted to ask you two questions and would love to hear your thoughts about it:
>
> 1. Do you think that NPS Benchmarks are really accurate considering that every company implements NPS in a different way and that this might affect the result you get? (i.e. when and how do you ask the question)
>
> 2. How do you think that A.I. would affect the future of research? How long would it take before i will be able to predict the NPS based on the information that i already have from the interaction, without needing to ask the customer?
>
> I hope that you are a great visionary and will help us anticipate the future!!!
>
> Looking forward to learn from you and share great experiences during the event!
>
> Carlos

That’s fantastic to hear, Carlos!

1. Wow. This is a loaded question with great perspectives on both sides. I will try and answer just from my own personal perspective (not trying to reflect the perspective of Qualtrics or the research community). There are many challenges with NPS and all metric forms. But for all the challenges, NPS is an intuitive concept and easy to get started. And for many organizations, that's the hardest part: taking the first step to improve their experience. If NPS helps organizations advance toward being more experience-minded and helps them get started in getting visibility to the experiences they deliver, then I think it serves a noble purpose. And as organizations mature they can add or augment their metric. But the usefulness I see in NPS is its simplicity; just seeing people get started has done wonders for their experience management.

2. A lot of this can be done today, where inferred customer satisfaction can be made based on behavioral metrics. That being said, predicting NPS is a small fraction of the puzzle for organizations who are experience leaders. What to do about NPS is the magic. The breakthrough organizations have built a culture of customer experience excellence. Being an experience leader is not only about measuring experience, but it’s about the action you take to continually improve experience.
JayC
Rank
> @NiC said:
> Hi @JayC,
>
> So the only Question i could think of which hasn't been asked by my fellow community members is
> How do you guys prioritize the features that need to be added , I am sure you all would have a vision as to what features are to be added to the platform and also constant onslaught of feature requests from various stake holders. I wanted to know as to how you guys balance your plans with the feature requests or are there different teams who work on both ?
>
> Regards,
> Nihal Chirayath

Thanks for asking, Nihal! Our ideation process is a multi-faceted one, and like our products themselves, it is always growing and expanding. While we primarily drive our product development on industry needs, we also take the time to carefully read the requests we get to learn where we have gaps, and how we can improve. Often times, the market impulse drives a larger vision, while feature requests act as a practical point of view to fill in the gaps - what in that last release worked, what didn’t work, what would make the process easier, etc. Every product decision also comes with a series of carefully executed peer revisions, research endeavors (that means us asking you what works!), and countless cross-departmental meetings, to ensure that not only company interests, but customer interests are being met as well.

P.S. - We really do read the feature requests you submit to our support team. 🙂
JayC
Rank
> @ana_velez_voce said:
> Hi @JayC,
>
>
> In the company where I work we are in a very important process of growth, facing new challenges and clients. What quality or characteristic do you think has helped you to develop in a great way in a company like Qualtrics with such a rapid growth?

1. Invest in hiring and developing great people, and never, ever lower your hiring bar. Qualtrics growth is fueled by incredible people. Getting the right people in the right jobs and investing in their own personal journey is the single most important things I do, and I am so incredibly proud of what the team can do.

2. Be kind. It's not always possible, but when it is, be known for kindness and integrity.
JayC
Rank
> @alejandra_rojas_voce said:
> Hello @JayC
>
> This is a great opportunity to get to know exciting things about you and Qualtrics. What is the biggest challenge you have faced with Research Core at Qualtrics and how you and your team worked it out?
>
> Thank you 🙂

One of the biggest challenges I’ve faced is staying focused on the most critical priorities that will really move the business. There are a million things that pop up every week, and everyone feels like they need urgency. Creating structural focus on the 3 year plan that will truly transform our industry: that's hard.

We tackle this challenge by having a monthly scorecard of our most important initiatives, and writing our press releases a quarter in advance. At the end of this quarter, we want this press release to be true. And we work toward it every quarter.
JayC
Rank
> @Shashi said:
> Hi @JayC ,
>
> I have heard from many people over the community post about the healthy and happy environment at the Qualtrics (and same I am experiencing over the community 🙂 ) . Please share your working experience at qualtrics and what steps you take to maintain the healthy environment in your team ?

I’ve done a lot of rewarding work in my life, but one of the things I love about Qualtrics is that I actually feel energized when I get up in the morning. I think everyone gets the Monday blues, but for me there’s always that additional thought of, _Well, at least I’m going to Qualtrics_. One thing I’m extremely grateful for at Qualtrics is that it’s a place where I can fit in as my authentic self. I simply can’t go through meetings without laughing or everyone poking fun at each other or just taking a break when it gets too serious. It’s not unusual on my team to spend the first 5 minutes of a meeting sharing a funny youtube video, meme, or story. I think it’s so cool that I work for a company where it’s ok that I don’t run meetings the “traditional way,” and where it’s safe to crack jokes and be real, or have career one-on-ones at Taco Bell.
JayC
Rank
> @KevinK said:
> Hey, I'm new to Qualtrics and looking to tighten up our hiring profiles - if you were building a job req for a hire that you want to train up as a power user on Qualtrics (entry level), what would you specify in the req? Given the visibility you have on the feature plan for the next 2-3 years, what skill sets etc. should we be looking for? (e.g. is python integration coming, for example?)
>
> Basically, as you've worked across dozens or hundreds of clients, what does the profile (or profile set) of a successful Qualtrics power user look like? Let's assume a perfect unicorn doesn't exist, what should we focus on - business analysis, writing skills, stats, coding background, etc? Likewise, are there any red flags you would call out?

Hmmm...this might not be the answer you’re looking for, but if I were hiring for this type of role, I would think about a couple things:

1. Storytelling (with data) - What matters more than great research is moving an organization to action on an insight. It’s all about finding someone who understands how to craft a story that frames the core issue, the relevant facts, and an inspiring course of action.
2. Business acumen - Answering the right question is important. A lot of times, the key question we need to answer isn’t obvious. Someone who is relentless in finding the root decision to be made will always be an advantage, because they are spending their time on the right areas.
JayC
Rank
> @jpardicusick said:
> I thought of another question:
>
> I know many individuals who are successful in their careers have interesting hobbies or talents that while aren't directly related to business, help fortify the characteristics that lead to career success. One example is a colleague of mine who practices martial arts, which helps him build discipline that carries over to his work ethic. @JayC, do you have any hobbies or special talents that at first glance seem completely unrelated but actually do help you achieve your career goals?

Haha, this is a little embarrassing, but…

1. Acting - I almost pursued a career in acting, but ultimately went down a different path. But it's incredible how useful it’s been, as I find myself constantly giving large presentations where I draw upon stage presence skills. Never intended, but always useful.

2. Taco Bell - This isn't a skill... or maybe it is, haha. But I absolutely love Taco Bell, and for some reason there's a small society of people in any company that share this passion. You make friends quick.
JayC
Rank
> @AnthonyR said:
> Hi Jay,
>
> Two questions for you:
>
> As far as the research core goes, what upcoming additions / changes to the product excite you the most, and why? Would love to get some insight into your thought processes on how to drive the platform forward.
>
> As you work to move the Research Core product forward, what are the things you look for as you hire members on to the engineering side of your team? Is it people who bring fresh ideas to the table, people with tons of experience, some combination of these things, or something different entirely?
>
> Appreciate you taking the time to answer our questions!

Hi Anthony,

One of the projects most exciting to me right now is coming from our ExpertReview family of products. In the past, we’ve broken industry ground by being the first to add artificial intelligence and machine learning into a research platform. We plan to add significantly more to ExpertReview with survey quality analysis and the establishment of brand-wide sensitive data policies, and now we’re continuing to beat top competitors in the industry by developing a product that can help customers parse the quality of their data.

Another area we are investing heavily is our automation platform called XM Solutions. Our goal is to make world class research methodologies simple and accessible to anyone. Similar to how you don’t have to be a tax expert to do your taxes anymore, we hope to do the same in research. We want to enable individuals and organizations to build their own content on top of the platform.

It’s been incredible to see how the ExpertReview and XM Solutions brand has grown in just a few short months.
U
Rank
Userlevel 6
Badge +5
Hi @JayC,

Thank you for taking the time to answer our questions. I have two questions for you.

1) Are their any improvements in the works for Results Reporting? Our office still prefers Results Reporting over New Reporting and it would be great if Results Reporting could receive the same features that have rolled out to New Reporting.

2) Are there any plans to better communicate new features for the Research Core as they roll out? Currently some features are communicated through the Community, others are communicated through the Product Update page (https://www.qualtrics.com/product-updates/?solution=rc) and others I just find on my own while using the tool. For example, the new Text IQ was announced in the Community and never showed up on the Product Update page. It is my responsibility to keep my team up to date on new RC features and this has become increasingly more difficult in the last few months.

Thanks again!
VirginiaM
Rank
Userlevel 6
Badge +45
@JayC thanks for your thoughtful answers! I thought of another question, if you don’t mind.

What made you want to work at Qualtrics and can you tell us about your journey to get where you are today?

It seems like an incredible company to work for (and if I ever work there I’ll be a member of your Taco Bell fan club)!
JayC
Rank
> @mattyb513 said:
> Excited to see you here, Jay. Coming at you with a question for a large trade organization.
>
> We have thousands of member organizations that we support. Traditionally all our interactions with them have been via email or phone calls, very personalized interactions, the DC way. We would like to move to a system that allows us to better capture their experience in a measurable way so we can course correct and let them guide our decision making. Essentially, we want to do better at experience management but have no guidelines for how to get there.
>
> 1) How would you think about evolving from a very person/relationship-centric model to a more scalable software solution without losing the personal touch?
>
> 2) What kind of questions do you ask to help people feel like their responses are not just going to be aggregated into a dashboard?
>
> 3) Who are some companies that have successfully done this who might be open to a short guidance call?
>
> Looking forward to picking that big brain of yours!
>
> Matt Bloomfield

Great questions, Matt. I will DM you for more info.
JayC
Rank
> @Cenzi said:
> Jay
>
> How can I get the most out of my dual subscription to CX and EX?
>
> Want to show how my staff metrics impact my customers experience but not sure where to start.
>
> Adam

This is a great question, Adam. I’m glad you recognize the importance of marrying EX and CX, because how we treat our employees very often directly impacts customer experience. There is a lot of theoretical analysis on this, but I thought I would give one of the more pragmatic examples I had seen.

One of our customers is an automotive company who has both CX and EX. At first, these were separate programs run independently at the company. When they first got their CX program, they correlated it with sales data and found a strong correlation between CX and dealership sales. The dealers with the highest sales also had the highest CX scores. They then took this analysis further and found that the dealerships with the highest CX had the highest employee engagement scores. While that was a powerful insight, I was really impressed by how they operationalized that insight.

Here are a few things they’re doing:
1. Aligned their CX reporting to their EX reporting - particularly around customer facing teams. Now managers can see their CX dashboards filtered in the same manner as their EX dashboard.
2. Increased the frequency of the EX pulses to quarterly, because annual measurement didn’t allow them to track changes and intervene with their teams.
3. Held weekly standups on their CX score. They literally sit around the table with every department in the dealership represented and solve problems in the moment
And took immediate action on key EX issues:
* Regular Trainings
* Sales programs
* Holding managers accountable for engagement on their teams

Taking these simple actions completely changed the culture of the dealership. Customer-facing Employees could see that the organization was taking real action to improve their engagement and addressed many of the issues they faced. They could also see immediately that the organization was taking CX seriously through their weekly standups. By seeing that the organization was measuring and managing each of these elements, it drove a cultural shift and, impressively, even created a sales lift.
JayC
Rank
> @mujtanibul said:
> Hi Jay,
> How can we measure customer journey success thru Q research core tool ? and is it possible to measure ( both X and O data) journey wise user experience ?
>
> How can you map product experience with revenue and a method to quality on routine basis?

One example for UX research is the combination of clickstream (O-data) and Customer Effort Score (X-data). Oftentimes we will A/B test a new feature or the location of a button in our UX research. We utilize the drop off rates to understand how many people interacted with a new features followed by an in-product Effort Score study. Customer effort measures how difficult was it to complete your task, whether you accomplished your task, etc. Through this combination we can understand what is happening and why it is happening.
JayC
Rank
> @jpardicusick said:
> Hi @JayC,
>
> Thanks for taking the time to do this with us.
>
> I have a question about industry XM benchmarking. We're a B2B communications/technology business and I'm struggling with finding meaningful benchmarks and standards to use for goal setting, especially as it relates to response rates and XM-specific metrics. What advice can you give about what KPIs to focus on (besides NPS) and where to find benchmarking information?
>
> Thanks,
>
> Jen

NPS is great for measuring relationship strength over a long period of time, since you’re asking people to recommend a product or service to a relative or friend… aka, to the people who you trust the most, and who matter the most to you. In contrast, CSAT is typically a better measure for transactional engagements based on a single point in time. When the experience is still fresh in your mind, it’s great to have the chance to provide genuine feedback and let brands know how you feel. Customer Effort Score (CES) is another measure that focuses on how easy or difficult a task or engagement is. Obviously, we want to provide customers the easiest, quickest, and most enjoyable experiences when engaging with our brand.

As for benchmarks, we have some very exciting news! At X4, Qualtrics’ incredible customer event, we are announcing several benchmark options to help you better understand how your scores measure up. We’ll have benchmarks from both JD Power and Bain & Company running on Qualtrics very soon. We also have the XM Institute, run by Bruce Temkin, and will soon be making XM Institute Benchmarks (formerly known as Temkin Benchmarks) available to all of our CX customers. Lots of exciting things are happening to help you better understand your customers and your business, and to help you make the best decisions.
JayC
Rank
> @Shashi said:
> Hi @JayC ,
>
> With the regular updates in the product and new feature request coming in, how do you manage your work along with your personal life ( i.e. your work - life balance ) ?

I think the most important thing you can do for yourself with regards to work-life balance is draw up some boundaries. Some projects are important enough that they are going to take up your time even when the work day has ended; that’s just life, and the results you get from putting in the extra hours can be incredibly rewarding. But it’s important to recognize when there’s nothing substantial that you can contribute anymore. Maybe the teams you need to work on a solution with went home, or it’s time to admit that the project demanding your attention just isn’t that dire. On particularly crazy days, I always make a point to shut off my phone at dinner so I can take some time to decompress with my family.

It also helps that Qualtrics deeply encourages us to take at least one big, exciting vacation every year to just experience something new. I take my vacation time very seriously. I even did a week-long technology cleanse last year, and it was incredible.
JayC
Rank
> @JasonHill said:
> Hi @JayC,
>
> We at UVU have built an integration with Canvas LMS. It's working beautifully and provides grade pass back to the Canvas grade book. We accomplish this, in part, using the Scoring option. However, we currently don't have a method to grab the Total Score possible as default embedded data. Instead, we add a custom embedded data field and manually set the value to the Total Score possible. Could I entice you to add the Total Score possible as a default embedded data field?
>
> Also, our integration is currently built using the Qualtrics and Canvas APIs, but we are in the final stages of the LTI development. I can arrange a sneak peak of this awesome sauce for you if you have something equally cool to share with me...could be anything from something awesome coming to the Research Core platform, or some cool skill you got.

Wow, that’s incredible! I’m always impressed with the integrations our power users manage to make on their own. It makes me proud for what our API team has developed and people like you can leverage it!

I would love to see you share this in the Community when you’re done - I’m sure we can all benefit from it! In terms of “Total Score,” I have to confess I’m not familiar enough with Canvas to make any promises.

As for our roadmap… DM me, and we can trade secrets. 🙂
JayC
Rank
> @Cenzi said:
> Hi Jay
>
> Any advice best way to share insights with non-technical users without having to train them on how to be a survey guru?
>
> Is there an easy way in the research core which enables you to see who is looking at the dashboards we share and create?
>
> Regards

Hello again! For your first question, I’d say the support site and Basecamp are great places for users to stop by if they need help getting started. We have lots of resources for newbies that don’t dive too deep into the product, meaning they can get the basics without necessarily being a guru.

As for your second question, that’s a great feature idea! I’m not sure if we have anything that currently fits that niche, since Research Core doesn’t cover dashboards, but we should have a feature like that if we don’t.
JayC
Rank
> @bansalpeeyush29 said:
> Hi Jay,
>
> Qualtrics is said to be "one platform" for all kind of research. How you and your team managed to have all functionalities included within single platform?
>
> Regards
> Peeyush

Hi, Peeyush! We keep our eyes and ears wide open for the challenges our customers want solutions for, as well as innovations happening in the marketplace around us. But it’s not enough to just do the research and have an enormous flux of information to sift through; I’d say our true key to success is our organization into the pillars of experience (Brand Experience, Customer Experience, Product Experience, Employee Experience, and then the Research Core at the center). This helps us stay broad in scope, while narrowing our focus team-by-team.
JayC
Rank
> @Alexandra_OPecko said:
> Hi @JayC!
>
> I am excited to be attending the summit again this year! I was so inspired last year that I made significant changes in my personal life and my career. Was it Dr. Lovatt? Alex Honnold? Arianna Huffington? Perhaps it was all of them combined...
>
> I started an online JavaScript course (to become a Qualtrics super user one day), and much to the shock of my family...I now take a beginners hip hop class. _(Yes, I did break out with some dance moves when I read what was happening on Fight Night! 😃 )_
>
> So, I was wondering: From where do you draw your inspiration? And how has this resource influenced your life?
>
> All my best,
> Alex

I’m so glad to hear that! I loved our keynotes last year, and I’m so pumped for the incredible guests we managed to get this year. And congrats on the classes!

For inspiration, I look at consumer products and consumer software. While B2B software like Qualtrics has historically been held to a different standard. I believe B2C solutions have made the user experience beautiful and delightful and I think that should translate into what we use for our business tools
JayC
Rank
> Hi @JayC,
>
> Thank you for taking the time to answer our questions. I have two questions for you.
>
> 1) Are their any improvements in the works for Results Reporting? Our office still prefers Results Reporting over New Reporting and it would be great if Results Reporting could receive the same features that have rolled out to New Reporting.
>
> 2) Are there any plans to better communicate new features for the Research Core as they roll out? Currently some features are communicated through the Community, others are communicated through the Product Update page (https://www.qualtrics.com/product-updates/?solution=rc) and others I just find on my own while using the tool. For example, the new Text IQ was announced in the Community and never showed up on the Product Update page. It is my responsibility to keep my team up to date on new RC features and this has become increasingly more difficult in the last few months.
>
> Thanks again!

Thanks for the feedback! New Reporting mostly got to be the star for a while because it’s the newest reporting feature we offer for RC. We’ll take what you and our other customers have said into account as we proceed with Q2 planning.

I’m sorry to hear it’s been difficult. We as a company are always working on new ways to get product updates out, and while the Product Updates page and the Community are currently the best forums for product improvements, we’ll be working on more channels in the future.

Leave a Reply


Terms & Conditions