Qualtrics enables us to make use of time information in many different ways
There are many ways in which time is an important factor in running research studies. Qualtrics enables us to make use of time information in many different ways, and this post outlines some of the uses of time information what you might not be aware of. Each of these topics will be developed into a “how to” tutorial at some point, so this post is a bit of a teaser, but also to let researchers know what is possible.
1. Completion time.
By default, Qualtrics records the time that a survey is commenced and then the time that it is completed (generally when the final block and question are presented and the survey exits). By simple subtraction in SPSS this can be turned into the number of seconds or minutes taken to complete the survey. Of course if a person takes a break to watch their favourite TV show in the middle of your questionnaire, then the duration may be hours, or even days.
However, what is often more interesting is those that complete the questionnaire in a very short time. If you think the questionnaire should take 10 minutes to do, and someone completes it in 4 minutes, then that might suggest that the data won’t be meaningful. In some cases you might also want to exclude those that took too long. In either case, the time data collected as standard can be a useful validation tool.
2. Timing the presentation of a stimulus.
For the moment I am not talking about very short presentation times, though I talk about that below. For more simple cases where you want to present stimuli (words or pictures) at a set rate, Qualtrics has a timer element that can be used. More information about the timer element can be found on the qualtrics site here. One way the timing element can be used is to fix a minimum time before the next button appears, or to force presentation of information at a set pace.
3. Timing a response.
The same timing question type can be used to explicitly track how long participants spend on a particular question. This could be used to validate responses at the individual block level (as opposed to the overall completion time – see point 1 above). If respondents are being paid for, then this could be used to screen out these participants early in the survey.
Other interesting research uses are possible. For example if you were interested in what opinions on an issue participants are willing to attend to, you could present a series of views both for and against some issue and time how long each participant took to study each view. Shorter times might imply less interest in reading about that perspective – longer times might be taken as showing more interest and attention.
4. The problem of short timing, and some emerging solutions.
Traditionally, a range of research topics are excluded from being done online or via a easy user-interface like Qualtrics because the studies require very short presentation or response times. For example very short presentation times are needed for subliminal priming studies. Very short response times are needed in some reaction time studies. This has always been a challenge, even in the controlled environment of a lab. Modern computer operating systems are not designed to allow an individual process to know exactly when it has control of the computer hardware. Even screen refresh times are too much unknown variation for some tasks.
So as always, the question the researcher needs to consider is how much noise and variation is acceptable due to the variety of computers that participants might use, and how to measure and control for that.
With that caveat, there are some cases where the differences in reaction times being investigated are in a more intermediate range and researchers are moving toward trying online data collection for these kinds of studies.
For that purpose, sometimes the built in timer may even be sufficient. The timer does give the response time in milliseconds, though it does not claim to be “millisecond accurate”.
If a greater level of accuracy is required, several solutions have now been developed using Javascript, or a combination of Flash and Javascript. I will write a longer post on this topic later, but in the meantime interested users may wish to check out the QRTEngine, which has some supporting validation evidence of short presentation times and finding classic patterns based on short reaction time data. The QRTEngine site is here, and the validation study is available here (DOI 10.3758/s13428-014-0530-7).
These methods remain controversial, but seem promising.
Summary
Although some users may see Qualtrics as a handy tool for simple surveys but of no use for hard-core experiments, there are many researchers who are trying to leverage the ease of design in Qualtrics to assist in experimental studies as much as possible, to allow experiments to be carried out online.
In this post I review a few ways in which timing issues are dealt with in a Qualtrics study in order to expand the range of researchers who can benefit from these convenient features.