scriptingrt and mturk

I got a question by email and thought it would be best answered by a post:

First, is MTurk the most popular avenue for ScriptingRT?  Or is there another/better platform of access?

Mturk is definitely the best way to recruit participants quickly, and the data quality is reasonable. We also used facebook to spread a link to studies, but it is usually much slower. But note that we always use another platform, Qualtrics, to build a questionnaire around the scriptingRT. I guess it would be possible to embed a scripingRT file directly in a Mturk task, but I never tried that.

Also, is it possible to use either ScriptingRT or MTurk to create a design where a worker must complete several tasks (eg reading three sets of sentences over a span of 3-5 days) before being paid?

I guess yes. You could create one Mturk task that would have such a long duration. Alternatively, you could create one Mturk task for the first measurement point, and then invite your workers to do the second and third measurement points, stepping up the payment if they also participate a second and a third time.

setting an embedded data with javascript instead of a web service in qualtrics

qualtrics’ web service was a bit unreliable during the last days. It’s a problem because we use it to connect qualtrics and mturk. qualtrics support suggested to use javascript instead, in particular the Qualtrics.SurveyEngine.setEmbeddedData() function. It works nice, and I set up an example how it works.

so, if you have trouble with the random number generator, just stick this javascript in the first question (but AFTER the embedded data field has been created). Note that it takes one page break to actually update the value (see the example).

connecting mturk to qualtrics to scriptingrt

After describing how to get the mturk worder ID into qualtrics, how to get qualtrics data into scriptingRT, and how to use a code number to give mturk workers something to submit, I now put it all together in one template.

This qualtrics survey

– takes the mturk worker id as a value in the url (here 888888) and puts it in an embedded data field called mtwid; the full link is http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG&mtwid=888888

– generates a code number mturkcode

– embeds the flash file using swfobject, and transmits mtwid and mturkcode as flashvars

The scriptingRT file imports mtwid and mturkcode, displays it, and appends it to the data it saves.

I exported the qualtrics survey, maybe that’s useful. See also the examples page for for the mturk.swf and mturk.html files.

Survey Flow should look like this. Note that all of these variable names are case sensitive, if you put mturkcode here, it cannot be MTurkCode later.

HTML to put into a text field in HTML View:

<script type="text/javascript" src="http://scriptingrt.net/swf/swfobject.js"></script>
  <script type="text/javascript">
   var flashvars = {};
   flashvars.mtwid = "${e://Field/mtwid}";
   flashvars.mturkcode = "${e://Field/mturkcode}";
   var params = {};
   params.play = "true";
   params.loop = "false";
   params.menu = "true";
   params.allowfullscreen = "true";
   params.allowscriptaccess = "always";
   var attributes = {};
   swfobject.embedSWF("http://dl.dropbox.com/u/1009114/scriptingrt/mturk.swf", "myAlternativeContent", "800", "600", "9.0.0", false, flashvars, params, attributes);
  </script>
 </head>
 <body>
  <div id="myAlternativeContent">
   <a href="http://www.adobe.com/go/getflashplayer">
    <img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player" />
   </a>
  </div>
<p>Only proceed to the next page after you are told that the data are saved.</p>

How to connect Qualtrics and mturk, Part II

(Updated 15.02., with a description on how to add the confirmation number as a safety net.)

Some time ago, I posted our simple method on how to connect mturk and Qualtrics data. The idea was then to generate a random number in Qualtrics, save that in the dataset as an embedded data field, tell it to the participants at the every end of the study, after which they copy it into a text field in mturk and hit submit.

This method worked reasonably well in many studies, but it has its weaknesses, most importantly that wrong codes end up in the mturk database for probably many reasons. Sometimes, people forget to copy, or don’t see the code, or loose connection to Qualtrics, etc. In short, it would be nice to have a more automated backup connection.

Here I describe how this can be achieved with the workerId. This ID is unique for every MTurk worker, and Amazon makes it available as soon as a worker accepts a HIT (thanks to Winter Mason for suggesting this).

First the technical background:

The workerId is accessible in the source of the frame that presents the HIT. You can try this by accepting a HIT, then (using Firefox) right-click on the frame with the HIT, choose This Frame –> Frame Info, and then look at the location.

Here is an example:

https://s3.amazonaws.com/mturk_bulk/hits/54314710/xIpb71zULpXQALR7MKF2DA.html?assignmentId=27VV0FK0COK270BOW2BKGLRW87KIPY&hitId=2CZ8MZ9O9Z6KKQ9OZC8HYRLFQZ9A1G&workerId=A31DOCT85I******&turkSubmitTo=https%3A%2F%2Fwww.mturk.com

You see the workerId (which is mine, and I masked the end with asterisks) in the middle of the path.

To extract this workerId and submit it to Qualtrics, I wrote a little javascript that has to go into the mturk task.

The html and script:

<p>Go to <a href="http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG"
 target="_blank" onClick="addId()">this website and work on the task there</a>
 (link opens new page).

<p>At the end, you will get a confirmation number. Copy that number and
paste it into the field below, and then submit the HIT.</p>

<script type="text/javascript">
//from here on leave everything as it is
function addId(){
    //setting default values. If the parsing of the workerId fails,
    //99999 will be transmitted.
    var workerId="99999";
    var iFrameURL=document.location.toString();
    var temp="";
    //parsing and extracting the workerId
    if (iFrameURL.indexOf("workerId") > 0){
        if (iFrameURL.indexOf("?") > 0){
            temp=iFrameURL.split("?")[1];
            if (temp.indexOf("&") > 0){
                temp=temp.split("&")[2];
                if (temp.indexOf("=") > 0){
                    workerId=temp.split("=")[1];
                }
            }
        }
    }
    //appending the workerId to the link
    document.links[0].href+="&mtwid=" + workerId;
}
</script>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>confirmation number:</p>
<p><textarea rows="3" cols="80" name="answer"></textarea></p>

The first two paragraphs are HTML and simply add the link. The link contains an onClick event handler that executes the javascript function in the script before the link is followed. If the browser has no javascript activated, the link is followed without the workerId being added.

The javascript function takes the location of the frame, and tries to extract the workerId. If the extraction fails, the workerId keeps the default value of 99999. The workerId is appended to the link, using the syntax that Qualtrics expects.

If it works, the link will call: http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG&mtwid=99999, with the correct workerId instead of 99999.

How to actually do it:

1. Create your HIT in mturk. Make sure you do that via creating a HIT template, not by “create HIT manually” – in the latter case, the workerID does not seem to be available. When designing the layout, switch to “Edit HTML Source” and enter the code above. Obviously, you want to change the link to the actual link to your Qualtrics survey, but be careful to leave all quotes etc. as they are.

When you preview the HIT, it will look like this:

This is also what workers will see.

In Qualtrics, add an embedded data field and call it mtwid – this is what is appended to the link in the form of … .html?mtwid=999999. It will look like this:

That’s already it. The embedded data field is saved in your data file.

For testing purposes, you can ask Qualtrics to display the content of the embedded data field, like this:

When the link is followed, this will produce this qualtrics page:

 

Adding a safety net: Giving a confirmation number.

I still recommend that in the end, Qualtrics should give a confirmation number that has to be pasted back into qualtrics. It directs participants back to mturk and make sure they actually submit and don’t loose their HIT because of going over the time limit. Also, if a worker has javascript turned of, the transmission will fail, and the confirmation number is then a backup to match datasets.

In a nutshell, here is how to do this:

In the survey flow, add a web service element that gets a random number from a qualtrics web service, and stores it in an embedded data field that we call mturkcode

It’s important to place this at the beginning of your survey, before the actual question blocks.

The web service URL is http://reporting.qualtrics.com/projects/randomNumGen.php. You can choose any name for the embedded data field you like (instead of mturkcode), but the “random” has to be exactly that. min and max will be the limits of the returned random number.

Then, at the end of the survey, you tell the generated number to your participants. This is done by using the piped text function, which you can easier click on while editing a text field, or directly entering the text as shown below:

or, in plain text: “Thank you for taking part in this study. Your validation code for mTurk is ${e://Field/mturkcode}. Please press on the continue button >>> one more time.”

In MTurk, you then ask for this confirmation number before people submit the HIT.

For more info, see the earlier post on this.

Some additional notes:

  • I hope this is not violating any mTurk policies. I read them carefully and don’t think it does. If you think otherwise, please let me know.
  • Testing this in the mturk Sandbox does not work, as no workerId is assigned there for a task.
  • While creating this, I learned something that might be useful in the future: To store a value in an embedded data field in Qualtrics from Javascript, it’s possible to use a hidden input field: <input type=hidden id=’myField’ name=’ED~myED’ value=’remember_this’> will save “remember_this” in the embedded data field myED, which has to be created in the survey flow.