How to connect Qualtrics and mturk, Part II

(Updated 15.02., with a description on how to add the confirmation number as a safety net.)

Some time ago, I posted our simple method on how to connect mturk and Qualtrics data. The idea was then to generate a random number in Qualtrics, save that in the dataset as an embedded data field, tell it to the participants at the every end of the study, after which they copy it into a text field in mturk and hit submit.

This method worked reasonably well in many studies, but it has its weaknesses, most importantly that wrong codes end up in the mturk database for probably many reasons. Sometimes, people forget to copy, or don’t see the code, or loose connection to Qualtrics, etc. In short, it would be nice to have a more automated backup connection.

Here I describe how this can be achieved with the workerId. This ID is unique for every MTurk worker, and Amazon makes it available as soon as a worker accepts a HIT (thanks to Winter Mason for suggesting this).

First the technical background:

The workerId is accessible in the source of the frame that presents the HIT. You can try this by accepting a HIT, then (using Firefox) right-click on the frame with the HIT, choose This Frame –> Frame Info, and then look at the location.

Here is an example:

https://s3.amazonaws.com/mturk_bulk/hits/54314710/xIpb71zULpXQALR7MKF2DA.html?assignmentId=27VV0FK0COK270BOW2BKGLRW87KIPY&hitId=2CZ8MZ9O9Z6KKQ9OZC8HYRLFQZ9A1G&workerId=A31DOCT85I******&turkSubmitTo=https%3A%2F%2Fwww.mturk.com

You see the workerId (which is mine, and I masked the end with asterisks) in the middle of the path.

To extract this workerId and submit it to Qualtrics, I wrote a little javascript that has to go into the mturk task.

The html and script:

<p>Go to <a href="http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG"
 target="_blank" onClick="addId()">this website and work on the task there</a>
 (link opens new page).

<p>At the end, you will get a confirmation number. Copy that number and
paste it into the field below, and then submit the HIT.</p>

<script type="text/javascript">
//from here on leave everything as it is
function addId(){
    //setting default values. If the parsing of the workerId fails,
    //99999 will be transmitted.
    var workerId="99999";
    var iFrameURL=document.location.toString();
    var temp="";
    //parsing and extracting the workerId
    if (iFrameURL.indexOf("workerId") > 0){
        if (iFrameURL.indexOf("?") > 0){
            temp=iFrameURL.split("?")[1];
            if (temp.indexOf("&") > 0){
                temp=temp.split("&")[2];
                if (temp.indexOf("=") > 0){
                    workerId=temp.split("=")[1];
                }
            }
        }
    }
    //appending the workerId to the link
    document.links[0].href+="&mtwid=" + workerId;
}
</script>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>confirmation number:</p>
<p><textarea rows="3" cols="80" name="answer"></textarea></p>

The first two paragraphs are HTML and simply add the link. The link contains an onClick event handler that executes the javascript function in the script before the link is followed. If the browser has no javascript activated, the link is followed without the workerId being added.

The javascript function takes the location of the frame, and tries to extract the workerId. If the extraction fails, the workerId keeps the default value of 99999. The workerId is appended to the link, using the syntax that Qualtrics expects.

If it works, the link will call: http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG&mtwid=99999, with the correct workerId instead of 99999.

How to actually do it:

1. Create your HIT in mturk. Make sure you do that via creating a HIT template, not by “create HIT manually” – in the latter case, the workerID does not seem to be available. When designing the layout, switch to “Edit HTML Source” and enter the code above. Obviously, you want to change the link to the actual link to your Qualtrics survey, but be careful to leave all quotes etc. as they are.

When you preview the HIT, it will look like this:

This is also what workers will see.

In Qualtrics, add an embedded data field and call it mtwid – this is what is appended to the link in the form of … .html?mtwid=999999. It will look like this:

That’s already it. The embedded data field is saved in your data file.

For testing purposes, you can ask Qualtrics to display the content of the embedded data field, like this:

When the link is followed, this will produce this qualtrics page:

 

Adding a safety net: Giving a confirmation number.

I still recommend that in the end, Qualtrics should give a confirmation number that has to be pasted back into qualtrics. It directs participants back to mturk and make sure they actually submit and don’t loose their HIT because of going over the time limit. Also, if a worker has javascript turned of, the transmission will fail, and the confirmation number is then a backup to match datasets.

In a nutshell, here is how to do this:

In the survey flow, add a web service element that gets a random number from a qualtrics web service, and stores it in an embedded data field that we call mturkcode

It’s important to place this at the beginning of your survey, before the actual question blocks.

The web service URL is http://reporting.qualtrics.com/projects/randomNumGen.php. You can choose any name for the embedded data field you like (instead of mturkcode), but the “random” has to be exactly that. min and max will be the limits of the returned random number.

Then, at the end of the survey, you tell the generated number to your participants. This is done by using the piped text function, which you can easier click on while editing a text field, or directly entering the text as shown below:

or, in plain text: “Thank you for taking part in this study. Your validation code for mTurk is ${e://Field/mturkcode}. Please press on the continue button >>> one more time.”

In MTurk, you then ask for this confirmation number before people submit the HIT.

For more info, see the earlier post on this.

Some additional notes:

  • I hope this is not violating any mTurk policies. I read them carefully and don’t think it does. If you think otherwise, please let me know.
  • Testing this in the mturk Sandbox does not work, as no workerId is assigned there for a task.
  • While creating this, I learned something that might be useful in the future: To store a value in an embedded data field in Qualtrics from Javascript, it’s possible to use a hidden input field: <input type=hidden id=’myField’ name=’ED~myED’ value=’remember_this’> will save “remember_this” in the embedded data field myED, which has to be created in the survey flow.
Advertisements

38 thoughts on “How to connect Qualtrics and mturk, Part II

  1. Thanks for sharing! I’m actually on the Qualtrics support team and we definitely appreciate having a more comprehensive solution to this than just the random number generator.

    1. Hi Ellis,
      well, what I guess what I did here is “quick and dirty”. A really comprehensive solution would be if Qualtrics would support the mTurk API. That would somehow compete with Qualtrics’ own panel, but I think it would add enough value to Qualtrics to balance that.

  2. Hi, it would be helpful if you updated this post to include the instructions for how to make Qualtrics insert a random number at the end. I know this is in your previous post, but some people will only see this post and follow the instructions verbatim, which could cause no random number to appear at the end.

  3. Thanks! By the way, I tried both methods and actually had more success with the random number generator (>90%) than with the worker ID method (~50%). I’m thinking of simply requesting that workers paste their worker ID in a form field at the beginning of the survey. Other people seem to be doing this.

    1. interesting. that means that the automatic transfer fails for half of the participants? do you have any clue why – javascript turned off maybe? you could see that in qualtrics if you add a meta question.

      re asking workers for their ID: yes, others do that, and because mturk shows it in a prominent location, most people will be able to do this. you could even prepare a qualtrics script such that it already shows the ID if it was transferred, or only asks for the ID if none was transferred. although that might create different feelings of anonymity for your participants.

  4. I recommend doing both, giving participants a code number (a random number or the respondent ID) and this method here. Transmitting the mturk id gives a backup – I always have people who miss the code number, or paste it wrong.

    Another advantage is that people who start your survey several times (if that is possible in qualtrics) can be identified, as the worker ID is available from the beginning, not only in the end.

  5. I am trying to identify where in this JS the code is responding to the “accept HIT” event. I am sending workers to a prescreening first, BEFORE they accept the HIT, but I additionally want to transmit the workerID to that prescreening, and currently the “&mtwid=” code in that URL doesn’t result in the workerID being transmitted.

    1. I have never worked with prescreenings before. not sure how it differs, but I imagine it could break for several reasons – either the id is embedded differently, and not correctly parsed, or the link is different. maybe you can send me a link to the hit, and I can see what it looks like.

      1. You can see the error by just hovering over the hyperlink to the prescreening URL: the workerID does not load. However, after “accept HIT”, it loads fine using the same JS syntax for the next survey link.

          1. Yes, I tested it across several dozen real HITs.

            Your notes notwithstanding, when I test this in the sandbox and press “Accept HIT” and hover over the link, my sandbox workerID shows in the URL.

            1. ok, interesting. 3 things:
              1) mturk has fixed the problem that sandbox does not provide the HIT for accepted HITs, that’s nice.
              2) still, for a HIT that has not been accepted, no workerID is available. I looked at your example, checked the location of the frame, and no, there is no workerID. So, the way you wanted it, it does not work
              3) but, isn’t what you want to achieve with your external prescreening exactly what “qualifications” are for? and the workerID would be there automatically.

              1. 1) swell.
                2) ah. As a backup, I do have access to IP address, fortunately. Just a bit harder to reconcile.
                3) Yes, I could be coding in Python and making the prescreening questionnaire part of MTurk, but I prefer to use the GUI of the survey company page. Also, I’m trying to obscure what I’m screening for, and it’s a work in progress. Finally, I need to save the data from the process of their screening, and that’s more easily done externally.

                Thanks for your help, I wouldn’t ask you to spend any more time on this. I appreciate it.

  6. Thank you for the tool!

    I implemented in our survey. It gives a backup method, but is there way to improve it?

    Here is my result from 1018 cases, so it may provide a better statistics. We got the ID successfully for 42.53% of cases. For those failed ones, some are 99999 and some are empty.

    When Java support is disabled (14.7%), the ID is empty. This is expected. 14.5% cases failed to report meta info, so we don’t know Java is enabled or not. But with those Java enabled (70.1%), we also got 99999 or empty ID for some cases.

    The result is independent of browsers. Firefox, Chorme, MSIE all have successful and failed ID detection, but Firefox seems to have more failed detection.

    If you need more detailed information, I’d be glad to provide. Thanks.

    -Xiangrui

    1. very interesting, thanks. I’m not sure why you check whether Java support is enabled, because what matters is JavaScript. Which one did you check?

      If javascript is enabled and there are still IDs missing, it could be that other privacy/security plugins or tools restrict the transfer of the ID. Or that the ID is not always available in the same way on the MTurk website.

      1. I just have Qualtrics checked the browser meta info, and one of the items is “Java Support”. It seems it may be a good idea to ask workers to enter ID (optional), and use the script as a backup.

  7. I tried using your script and it seems to work for some people, but for others the appending task adds “?workerid=9999?workerid=9999?workerid=9999?workerid=9999″…which causes the Get call to fail (I’m not using qualtrics, but my own php script)

    ….and the code returns the correct workerid in place of each 9999. I just put that there to annonymize the worker.

    Any thoughts?

  8. maybe these workers press the button repeatedly, and each time the code adds the workerid. that can happen because the code actually updates the link.

    So, this is clearly a bug 🙂 I did not think of that possibility.

    Could you please confirm that this is the case? By trying yourself, pressing repeatedly and quickly?

    If that’s the problem, we can easily fix it, by adding a variable that makes sure the appending is only done once.

    thanks for the feedback!

    1. That seems to be what’s going on. Every time I click the link, it appends an additional workerid=9999 to yield a string that doesn’t get parsed by php (b/c instead of concatenating with an &, it’s trying to do it with at ?…which is no good). I also found that if users right click and open the link in a new window, the script fails and doesn’t append anything (this caused a bunch of problems). If you are able to update your script to account for these issues, I would be greatly appreciative! I’m pretty handy at coding so, worst case, I can do it too, but I’m definitely not an expert at JavaScript (I do most things in PhP and Actionscript withing Flash).

      Thank you!
      Jeff

    1. In the example link that I pasted above, the assignmentID and the HITID are also embedded. So, it should be possible to change the script such that these two are also extracted and added. If you can code some javascript, you can give it a try, or you find somebody who can code. I can try to, but unfortunately not before next week. Let us know how it went!

  9. Does the URL still contain the workerID? I’ve tried in both the sandbox and by accepting a real HIT to get the URL, but in neither is there a value for workerID. If you’re still actively using mturk, do you have a way to grab it, or do you just go without?

        1. I suspect you look in the wrong place. I just did accepted a HIT, and when I did view source of the hit, the URL started with 184.72.249.63/render_hit_page_for_turk?assignmentId= and contained the workerID. but I would have to create a HIT like the one we typically use to see whether they changed the structure – this here certainly does not look like the URL above… can you send me an email, maybe we can work on this together tomorrow afternoon?

  10. For one link it works perfect. But what if I would like to append the mtwid to a second link in the same HIT? I tried to change the function name but this did not help.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s