how to connect Qualtrics and mTurk

Update: For adding another backup to this method, see this post.

Update: You can download a qualtrics survey and import it that shows an example.

we ran several studies using qualtrics and recruiting participants on mturk. We connect them by linking from mTurk to the survey in qualtrics. Participants fill in the survey at qualtrics, and get in the end a long number code that they copy and paste into the form at mTurk, thereby proofing their participation. I think I read this method somewhere, but cannot remember the original source.

Here are the technical details:

In the survey flow of the qualtrics survey, I added a Web Service that fills an embedded data field with a randomly generated number.

  1. Click Survey Flow
  2. Add a new element of the type Web Service
  3. as its URL, enter http://reporting.qualtrics.com/projects/randomNumGen.php
  4. Add two parameters: one that you call min = 1, and one that you call max = 9999999. Of course you can choose other min or max values.
  5. then click below to get an embedded data field, and specify it to be mTurkCode = random (the name of the embedded data field is up to you, but the right side has to be random – you can see this when you click on Test URL, there you see that the value returned from the web service is called random)
  6. move your web service block to the top of the survey flow. This is crucial, as the data field only gets created after this element has been processed. If you leave the web service element at the end of your experiment, you will not see anything when you try to access it.
  7. Save your flow

It looks like this:

UPDATE. SORRY, I JUST REALIZED THERE IS A MISTAKE. THE SECOND LINE SHOULD BE min, NOT mi

(click to enlarge)

Now, at the end of your survey, you have to tell the code to the participant. You add a last page with a text that says something like this:

Thank you for taking part in this study.

Your validation code for mTurk is ${e://Field/mTurkCode}

please press on the continue button >>> one more time.

the text ${e://Field/mTurkCode} will be replaced by qualtrics with the stored number. If you don’t want to type it, you can generate it while editing the text by clicking Piped Text, then Embedded Data field, then write mTurkCode

Don’t forget to test your survey and to make sure that the code number actually appears.

Then, in mTurk, you design a HIT that only requires the entry of the number code. It will look like this:

(Click to enlarge)

In the end, you will have the code of each participant in your qualtrics results database and in the results file that you get from mTurk. You then have to compare the lists and make sure that all numbers entered at mTurk have a counterpart in qualtrics. All these can be approved.

Advertisements

58 thoughts on “how to connect Qualtrics and mTurk

  1. a big problem I have with this method is that the random number generator OFTEN spews out the same number twice, thereby making it pretty much useless to differentiate which respondents answer seriously among the trolls. any suggestion?

    1. That’s very surprising. I just pooled the codes from 3 short studies, 137 cases in total, and there was not a single number that occurred twice. What was the maximum value you set? And are those cases that are very close together in time? How often is OFTEN? Maybe there is a problem in the random number generator web service.

  2. Hi! I embedded the code generator and did everything as you suggested, but the code was not showing up where it was supposed to at the end of the survey. Why is that so? Thank you!

    1. I have no idea 🙂
      the most common problem is that the name of the embedded data field created in the survey flow and the name in the piped text are not the same – so, in my example, mTurkCode appears twice, and it has to be the same name in both cases. Is that the case? Another problem might be that the web service is not working correctly. Try clicking on the Test URL button in the survey flow and see what it says.
      If it doesn’t work, please post screenshots of both the survey flow and the text where you want to write the code number, so that we can figure out what’s going on.

      UPDATE: we figured out the problem: The web service element was placed at the end of the survey, not in the beginning. To be ready to use, the web service needs to be in the survey flow before the element where the embedded data element is used.

      1. Thank you so much for your prompt reply! I did make sure that mTurkCode was consistent in both places, and the Test URL seemed to be working fine too. I have sent you an email with all the screenshots and I really appreciate your help!

  3. Short addendum: When editing the survey flow and clicking on Test URL, it always gives numbers between 1-99 – i.e., the min and max are ignored when Testing the URL. But when you ignore that, save the flow, and actually use the embedded data field, the output is correct, and the min and max are used.

  4. Hi, I am very new to M-Turk and have read your guidelines for adding MTurk codes when connecting Qualtrics to it. I have a question about how this may work with surveys that have multiple conditions that are randomized within Qualtrics. I have one survey that will be presented under 4 different conditions. In Qualtrics, I have these conditions as 4 blocks. In this case, should I add one common element in the beginning of survey flow for MTurk codes followed by the randomizer with the 4 conditions? Also, should each block have an end page about the validation code? I am a bit lost here and would appreciate any help in this regard. Thank you very much!

    1. Thank you Sir for the prompt reply! If you wouldn’t mind could you suggest how that can be accomplished? It would be a huge help! Thank you again.

      1. Update: Thanks again! 🙂 I was able to resolve this. All I had to do was add an ‘end of survey’ message via survey options which will be common for all conditions.

  5. Hi there, I’ve been using the random number generator for some time now, and it just stopped working today (in the middle of data collection). When I click on the reporting.qualtrics.com link, it says the webpage is not available. Am I the only one having this problem?

  6. They fixed it yesterday … now it’s down again. I emailed Qualtrics but they claim they’ve never had a problem with it. Anyone else seen this?

  7. Does anyone have any advice for an alternative random number generator? I need one quite urgently for a survey I am going to run.

  8. if it’s urgent and they don’t get this fixed, I suggest switching to giving participants the ResponseID in the end of survey message. The responseID is generated AFTER the questionnaire is finished, and you can display it in the end of survey message. when using this in connection with mturk, you can ask participants to copy this id back into the mturk response form. see http://www.qualtrics.com/university/researchsuite/faqs#codenumber the disadvantage compared to the random number is that participants have to copy paste something, and you don’t get a link for those who don’t finish.

  9. I got an email from qualtrics:

    We have been hearing about problems with the random number generator and our engineers are working on it. For now the workaround is to use javascript in any question before the random number is needed. Here is the code:

    var randomnumber=Math.floor(Math.random()*89999) + 10000;
    Qualtrics.SurveyEngine.setEmbeddedData(“random”, randomnumber);

    I have no time to try this tonight. If somebody tries, let us know how it worked. I will try it tomorrow as well.

  10. Hello, I’m having difficulty viewing the randomly generated numbers in my qualtrics reports. Everything was fine for the first 100 results, but now the bottom of the table says “results not tallied.” When I click this it says that “the table has no data available.” I’ve tried downloading the results in an SPSS file to see if there is anyway to view the randomly generated numbers in the file, but the numbers all show as 1. Is there another way I can view the randomly generated numbers?

      1. Hi psykins,
        The generator also stopped working for me after the first 100. It says I have four “[Not Tallied]”. Would you mind explaining what you did to figure it out?

        1. Hi Michael,
          Did you ever figure out the issue with the random generator stopping after the first 100 and then leaving you with “Not Tallied” counts? I am experiencing the same thing and would appreciate the help.

  11. Wouldn’t asking them to submit mTurk worker ID at the end of Qulatrics survey be sufficient proof of their participation?

  12. Okay guys, so I just spent ….well..more time than I’m willing to admit trying to get the code to work. I do not take well to defeat and, considering that I am not the most computer savvy, I figured my issue was likely due to operator error.

    So, here is what I just discovered. If you click “item preview.” You only see the ${e://Field//XXX}

    If you preview your block (in effort to avoid going through your entire survey), you get a little more, but no code. Thus, you no longer see the script, but as I stated, no code.

    AND….if you PREVIEW the ENTIRE survey (emphasis on PREVIEW, not just viewing from the first block) ………VOILA!
    Boy….to think of how much further I would be had I figured that out sooner. Hopefully, I can save someone else this trouble.

    This dissertation will be the death of me. 🙂

  13. Hey there, Hoping one of you might be able to help me. It seems that on MTurk a worker can not get back into the same hit once they have clicked on it. So how can I get a worker back to the page where they have to enter the qualtrics code number? It won’t let me back in to enter it? Thanks a ton!!!!

    1. I guess the problem is that people click the link on mturk, which takes them to qualtrics, and then they cannot go back? We usually change the link in mturk so that it opens the qualtrics survey in a new page (target = _blank, you get this in the context menu of the link in the mturk editor). this way, they don’t have to go back to the mturk hit, the page is still there in the background.

  14. Hi, I need help for generating Mturk Code. I followed the instruction above. When I preview or follow the link of my survery, there was no random number code in the end.
    It only shows the same content (${e://Field/mTurkCode}, not the number code) that I inserted in the last block – as the below. Is it correct that I should the following message in the last block in my survey? Or should I do it in other place? I will appreciate your help !!

    ——————–
    Thank you for taking part in this study.

    Your validation code for mTurk is ${e://Field/mTurkCode}

    please press on the continue button >>> one more time.
    ———————————-

  15. Thank you for this invaluable post! I have a question. I ran 300 subjects on my MTurk HIT, but ended up with only 276 responses on Qualtrics. Yet all MTurk submissions had valid random number codes (I had them all start with the same 3 digits to help catch cheating). There were no duplicate subjects since all the worker IDs collected by MTurk are unique.

    I identified the potential cheaters (8% of my sample) whose random number appeared in MTurk but not Qualtrics, but are they really cheaters? I don’t see how they could have given valid codes without having done the survey, and I’m loathe to reject their HITs if they didn’t actually cheat.

    I also have counterparts: people who were given a random number by Qualtrics but apparently never submitted the HIT. These folks comprise 3% of my sample.

    Any advice or wisdom much appreciated!

  16. Hi Maya,
    that sounds weird.
    In general, if you are not 100% sure a turker did cheat, you should not reject her/his HIT. It is a really serious punishment, and may prevent them from taking other HITs that ask for a high approval rate. Better be lenient if you are not sure.

    specifically about your problem – maybe the people with a valid code in mturk are still “responses in progress” in qualtrics? if the code appears on the last page, and they just copy & paste it and then close the window, they would appear not to have finished (your qualtrics survey has a preference for how many days it waits until closing a response).

    the people who are in qualtrics but not in mturk perhaps did not finish, or did not take part seriously, and then decided not to submit the HIT. or they got to the survey in a different way. I would remove those 3% from the sample.

  17. Hi Thomas. Hmm…I don’t seem to have any responses in progress on Qualtrics, but it’s been a couple days now. The survey is set up to force an answer on every question and to generate and display the code at the end. But you could be right that they copy-pasted the code at the end of the survey but forgot to hit the final >> on Qualtrics. Such a response would then not be recorded at all, right? Or become a response in progress and then disappear?

    In your experience, do you usually have a near-perfect correspondence between your MTurk and Qualtrics data?

    Thanks again for your help!

  18. if they fail to hit the final >>, the response stays open. While it is still open, you will not have it in the dataset you download. However, after it is closed (either automatically according to the preferences, or manually by you), you will have it in your dataset. It would not disappear.

    I usually have a near-perfect finishing rate, yes. I tell them explicitely before the number code to press >> once again to finish the survey.

    I believe it would also be possible to display the number code in the end of survey message. That would avoid this problem.

  19. You were right – after I re-downloaded the results from Qualtrics, those incomplete responses did show up. I really appreciate your help!!

  20. Hello,

    I have tried doing this with a psychology study and am having serious problems. The Qualtrics end works great, but when participants try to put in the code on mturks survey link page they get this error:

    There was a problem submitting your results for this HIT.

    This HIT is still assigned to you. To try this HIT again, click “HITs Assigned To You” in the navigation bar, then click “Continue work on this HIT” for the HIT. If this problem persists, you can contact the Requester for this HIT using the “Contact” link above.

    To return this HIT and continue working on other HITs, click the “Return HIT” button.

    like you mentioned in the article, my HIT is also set up so that theoretically they could enter anything and then I would just reject them later. However, no matter what they put down it gives this error.

    Is there any programming that you have to do with Mturk to get it to work? I worked from the survey link template and didn’t change anything except for the description. Is there additional code that must be added to “let it know” that any code entered is fine?

    Any insight would be great! Thanks
    Taylor

  21. Hi there, thank you for this great article. when I preview my survey, it works property and displays my custom end message with the random code on the very last page.

    When I activate the survey, my custom page has been replaced with a generic “thank you for taking this survey message”. When I view the results, the random codes appear, but they do not appear for the worker on the live survey. Any ideas?

    Thanks

  22. it can be tricky with the survey end messages because you can put them into the survey options, but also into the survey flow. the ones in the survey flow seem to be separate, and override the one in the survey options. maybe you them in both places, and they collide?

    1. Thank you for your reply. I did not realize the end message formed part of the option too. When I checked, my options had “default” end message selected, and my flow has a custom end message. I changed the option to the same custom message as the flow, but the active survey still gives me the default end message. Vexing!

  23. vexing indeed. with changes like this, you have to make sure you start as a new participant; sometimes qualtrics seems to cache previous versions for started sessions. I rarely use the preview survey, and usually open the “distribute” survey in a new incognito window. perhaps that is the problem already. good luck! and please post here when you have found a solution.

  24. Hi Thomas, I recently joined Qualtrics from Amazon and built a wizard that automates the steps of linking Qualtrics to MTurk. After going through the ‘survey builder’ people get a Qualtrics survey that’s already preset with a random survey code piped in to the end of survey message, and options to include other things like attention checks and screen out messages. Check it out, let me know what you think.
    http://www.qualtrics.com/innovation-exchange/mturk/
    jhyde@qualtrics.com

  25. Hi, I was wondering if you could help me come up with a solution to a tech issue I’m having. I am doing an online survey and I want the responses to be anonymous. But the participants have to be paid in person. So I want each participant to get a unique numerical code at the end of the survey. They will then present that code to a research assistant, who will check to see if that code is one of the ones we predetermined. Then they will get paid. Does that make sense? I need 50 unique numbers. Right now I am using the set-up you outlined in this blog post and I have the random number generator set so that the min is 0 and the max is 50, but I am worried that this won’t work because it might produce duplicate numbers. Also, this means I can’t test the survey beforehand because then it will use one of the 50 codes. Is there any way I could have the code taken from a predetermined list of numbers? Thanks for your help.

    Gina

  26. I am not aware of a way to set up a predetermined list from which numbers are drawn, and which is then coordinated across different participants.

    I would solve your problem like this: Follow the setup above, and let the numbers vary between 0 and 100000. You will have the numbers in the datafile, so when a participant comes to be paid, you check her number against the datafile, and if it is there, the person gets paid. It is very unlikely that two people will have the same number, but even that is not problem. It would also be impossible for someone to guess a number that is on the list.

    If you are worried about privacy and cannot have the participant ID in the datafile that contains the answers, you can avoid this. Simply set up a second survey that does nothing else but generate the random number and give it to the participant (and save it, of course, so you probably need at least one question in that survey as well). You then forward participants from your first (anonymous) to the second one by changing the end-of-survey message. This way, your first dataset stays completely anonymous, and the second dataset will give you a list of numbers that you can check to pay out money.

    Hope that helps!

    1. Hi Thomas,

      Thank you for your advice. What you proposed is what I originally wanted to do, but then I changed the payment method so that someone else was handing out the money instead of me. In that case, they would not have access to the data set and it would be a pain to update their list every couple of hours as more people completed the survey. Since it seems there is not a way to pick a number from a predetermined list, I will go back to my original plan of being in charge of the money myself. Thank you again for your help!

      Gina

  27. HELP! I am also getting the [Not tallied] issue for all participants passed 100 in Qualtrics. How did you recover these data?

  28. I wrote to qualtrics about the “Not Tallied” Issue and it turns out, that only appears in the “Initial Report”, if you download the data file you should be able to view all the codes that were given out.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s