Using Research Core for Course Evaluation Surveys

JulieT, I'm the administrator for Michigan State's (homegrown) online course evaluation system and we are slowly moving into the use of Qualtrics. We have a couple of manual pilots going on this semester, without any integration with our SIS, and a pilot next spring that will incorporate only the basics from SIS - course, section and student name. We envision many of the same requirements we are currently using for our online system to be implemented in our use of Qualtrics as we ramp stuff up, but our two pilots will be very simplistic. Our _current _system has automated emails that go out to students and faculty at various times during the response period, notifying/reminding them of the surveys. We have a set up a sequestration period after the surveys close so that if students do not respond, their grades are withheld for 7 days after they are released by the Registrar's Office. Student _can _opt-out of the survey without prejudice for the release of their grades. The surveys require a login by the student with their university email and password and are administered centrally. Faculty have their results available after the grades are released. Student identification is kept confidential and data is anonymized as much as possible. For our pilots this fall and spring, these are being manually created and administered by faculty administrators within the two colleges. The instructors will not have access to the survey data. The responses will be sent to them after the creation of anonymized reports by these administrators. Both colleges will be administering them either via a QR code in class or writing the URL on a chalkboard. Faculty won't have access to these prior to the class in which the surveys are provided to students and the surveys will be closed immediately after the course period. The spring pilot will involve email administration via a contact list created by the integration with our SIS. One of the colleges has been using a different survey tool (the name includes a mammal ;) ) but will be using Qualtrics from now on. They administer all their surveys in class on one day that students are notified in advance. Because the online interface for both the old survey platform and Qualtrics are very mobile friendly and class time is provided for them to do it, response rates are in the 90+%. I hope this is helpful. Let me know if I can provide any other answers. I'm not sure I was able to answer your question from a faculty perspective.

Thanks for the info @MichelleC Currently our course evaluations are conducted via scantron, if you can believe that! So this is a big adjustment. I'll be sharing this info with my team and I'm sure will be asking you other questions. :)

@JulieT LOL, don't feel bad! We still have a lot of departments who use the scantron forms. Our current online system is not mandated - departments can choose to use if they want. Let me know if you have any other questions. I'll try to remember and come back here after our two pilots run this fall and give you an update.

@MichelleC saw a demo of Qualtrics Classroom. It's a new course evaluation package rolling out soon. Pretty slick. Pulls course data in real time and dashboards everything. Unfortunately our university doesn't have $$ for license. But maybe yours has a money tree. :)

@JulieT , money tree!? You and me both, sister! 😃 I'll have to ping Matt Fisler on Qualtrics Classroom. We are going to be discussing an RFP for our eval replacement solution and were considering the Vocalize product, which may have morphed into the Classroom thing. I can't go to X4 this year ( :'( ) so I'll miss any of the general presentations on it, which is a huge bummer.

@MichelleC you'll definitely want to contact your rep. QC is the academic version of Vocalize.

Hi Julie, I've been using Qualtrics for our course eval system since Fall 2015. I'm the Qualtrics admin for our college, which is the nation's largest singly accredited community college system where we have about 250K course enrollments each fall and spring. We started with pilots in Fall 2015 with key courses that required evals for programmatic accreditation, but moved to evaluation all courses every fall and spring starting in Fall 2016. We faced the same faculty push back on response rate, however the cost savings, the quick availability of results, and the ability to load custom questions has helped ease the pain. I administer the evals to the students' email address, and we also post links in our online learning management system (previously Banner, now Canvas). We've had faculty post Course-specific links on their course page, which still allows us to collect the embedded student data from our Contact List. This surely helps with response rates, and also allows the faculty to share some ownership of the process and the response rates. One thing that I think would help boost response rates even more is a well devised communication plan that involves a message on every syllabus, in-class reminders from faculty, and blast pre-communication from either a Dean/Chair, Provost, or President. For reporting results, I've been managing a Vocalize dashboard, which has been a great tool for us! The faculty are able to log in and see only their results, and we have administrator-level access so that Deans can see all responses within their unit. The folks at Qualtrics have been awesome about helping set up of complex functionality!

@JakeW thanks for your feedback! We're in the midst of switching over to Canvas in Fall of 19 and are looking forward to integrating course evals into the LMS. I have a meeting today regarding course evals and will be using this info.

@JulieT Exciting times! It's really a lot of fun once you get going on this. Feel free to reach out if you ever have any questions.

@JulieT , I had an opportunity to sit down with @JakeW last year at the Qualtrics Convention and he was an awesome resource. If you are both going this year, you should take the opportunity to pick his brain! B)

@MichelleC would love to meet @JakeW Perhaps this is my angle to convince the Provost attending X4 is imperative. ;)

@JulieT I hope to attend, but we'll see. If not, we can schedule someone via GoToMeeting sometime. Don't hesitate if you have any questions in the meantime.

@MichelleC or @JakeW How do you handle the opt out link? Since it's internal, are you allowed to send without? Obviously there would be a new contact list generated every semester, but the student doesn't realize this and might think they never have to fill out a course evaluation again.

@JulieT we are so new to using Qualtrics for course surveys, I don't think anyone has thought about it yet. Our current (old, super old) system sequesters the grades for 7 days if students DON'T fill out their surveys. I'm sure at some point, if we go with Qualtrics for the entire U, they'll do the same thing. Our survey system here is pretty well naturalized so all students know there's no such thing as "opting-out". :) I'd be interested to hear what @Jakew has to say. My guess would be that regardless of its destination, you have to include the opt-out.

@JulieT and @MichelleC when I first began sending course evals using Qualtrics, I did not use the opt-out since I wanted the message to imply that the eval was not optional. However, I quickly realized that having an opt-out was a better option. I literally received hundreds and hundreds of emails from students who were adamant that they were not going to complete the evaluation for various reasons, and they typically asked to be taken off our mailing list. So, somewhat selfishly, I added the opt-out link, but made it clear in the text of the link that clicking the 'opt-out' link only removes them from future emails reminding them to evaluate that particular course. If a student has 5 classes and wants to be removed for all, they must opt out 5 times. Since I've had the opt-out option, I've felt that it was justified to remove these opt-outs from my denominator when calculating response rates. Keep in mind that out of a distribution of around 160K, we get less than 100 opt-outs. This number may be low compared to the number of emails from students I received, but at the same time, I added a polite 'do not reply to this email' message unless for technical issues. For the distribution I recently sent for Fall 2017 EOT, I actually tried to send without the opt-out link, and it WOULD NOT let me send the distribution. Since we are used to having it on, I didn't do any investigating as to why I wasn't able to remove it, so we continue to have the opt-out link. I hope this helps! Let me know if you have any questions.

> @JulieT said: > @MichelleC saw a demo of Qualtrics Classroom. It's a new course evaluation package rolling out soon. Pretty slick. Pulls course data in real time and dashboards everything. Unfortunately our university doesn't have $$ for license. But maybe yours has a money tree. :) I'm interested in hearing more about the Qualtrics Classroom--how can I find out about this?

@JakeW I was wondering, how do you get the course specific links out to your faculty in an efficient manner?

@VirginiaM contact your Qualtrics account executive. If you're not sure who this is, ask your Academic Support contact. (Love your profile photo, btw!)

@JulieT we just began using Qualtrics for course evaluations this year (although the implementation process began at this time last year). Thanks to an incredibly helpful implementation team at Qualtrics, we were able to do a pilot evaluation over the summer, iron out the kinks, and do a full launch for Fall 2017. Our Institutional Assessment and Research office handles the administration of the course evaluations. Each semester I look at the calendar of courses and schedule anywhere from 4-8 "phases" for course evaluations (based on the end date of the class). I upload a new contact list for each phase and send out automated emails to the students on day 1, day 4, day 7, and day 8 (final day). We also post announcements on Blackboard, Canvas, and Twitter. Students click the course-specific link in their email and use SSO to access the questions for that class. We have quite a few team-taught courses, so the survey is built out to accommodate up to 11 instructors per course. After the course evaluation period closes, faculty have access to their reports in Vocalize and department chairs/deans have access to their faculty's reports in Vocalize as well. I have built out slightly different dashboards for each of these end users. As far as response rates go, ours were higher than normal during the summer pilot but lower than the last two semesters in the fall administration. However, I think it is too early to tell what the end result will be. I am optimistic that the response rates will go up as students and faculty become accustomed to this new system. I am trying to meet with departments to get the word out to faculty that the new surveys are mobile friendly and can be administered during class like they used to with the pencil and paper surveys (this has won a lot of faculty support). Best of luck to you!

We use Vocalize to share dashboards with faculty, but the comments are creating a tricky situation for me. I have the comments displayed in a response ticker widget, but when faculty try to export their comments page as a PDF, it only shows what is visible on the screen. As a workaround, I instruct them to download the widget as a CSV and then save it as a PDF, but they are not liking this multi-step process. Has anyone found an easier way? Would another widget work better?

@VirginiaM - How often they need to pull the pdf. My recommendation would be if that is weekly then you can set filter on the widget level or page level for the time period. And you will have to expand the widget as such that all responses are shown and then a pdf print can be done. Currently only way is to expand the widget.

> @VirginiaM said: > We use Vocalize to share dashboards with faculty, but the comments are creating a tricky situation for me. I have the comments displayed in a response ticker widget, but when faculty try to export their comments page as a PDF, it only shows what is visible on the screen. As a workaround, I instruct them to download the widget as a CSV and then save it as a PDF, but they are not liking this multi-step process. Has anyone found an easier way? Would another widget work better? Hi Virginia! You may want to ask this as a question in the "Viewing Dashboards" category under Vocalize - see here. That way this will show as an independent question and display in the "Unanswered Questions" list on the home page. That may put more eyes on your question!

@VirginiaM you mentioned team taught classes. I've been told getting a list of multiple instructors names from PeopleSoft is pretty much not going to happen. Just the primary one. Do you run into this issue? If so, how do you handle it. I can't imagine students typing the partner instructors last name is going to lead to clean data. @JakeW or anyone else have something to add? Keep in mind, I don't have access to Vocalize.

@JulieT we use Banner instead of PeopleSoft, which does allow us to pull a list of all instructors for a course. The issue I run into is that the wrong instructors are often listed (for instance, department chairs are listed so they can see the class in Blackboard, but they don't actually teach or a faculty member is listed as 0% responsibility but actually teaches part of the course). I have not figured out a workaround for these issues yet, though have toyed around with the idea of letting a student choose from a drop-down list of all faculty (as opposed to typing in the name) in order to evaluate an instructor that's not listed.

@NK_Tripathi they generally download only once a semester, but if they wanted to include several semesters worth of comments in one download, it would probably be a long widget.