scriptingrt and mturk

I got a question by email and thought it would be best answered by a post:

First, is MTurk the most popular avenue for ScriptingRT?  Or is there another/better platform of access?

Mturk is definitely the best way to recruit participants quickly, and the data quality is reasonable. We also used facebook to spread a link to studies, but it is usually much slower. But note that we always use another platform, Qualtrics, to build a questionnaire around the scriptingRT. I guess it would be possible to embed a scripingRT file directly in a Mturk task, but I never tried that.

Also, is it possible to use either ScriptingRT or MTurk to create a design where a worker must complete several tasks (eg reading three sets of sentences over a span of 3-5 days) before being paid?

I guess yes. You could create one Mturk task that would have such a long duration. Alternatively, you could create one Mturk task for the first measurement point, and then invite your workers to do the second and third measurement points, stepping up the payment if they also participate a second and a third time.

setting an embedded data with javascript instead of a web service in qualtrics

qualtrics’ web service was a bit unreliable during the last days. It’s a problem because we use it to connect qualtrics and mturk. qualtrics support suggested to use javascript instead, in particular the Qualtrics.SurveyEngine.setEmbeddedData() function. It works nice, and I set up an example how it works.

so, if you have trouble with the random number generator, just stick this javascript in the first question (but AFTER the embedded data field has been created). Note that it takes one page break to actually update the value (see the example).

connecting mturk to qualtrics to scriptingrt

After describing how to get the mturk worder ID into qualtrics, how to get qualtrics data into scriptingRT, and how to use a code number to give mturk workers something to submit, I now put it all together in one template.

This qualtrics survey

– takes the mturk worker id as a value in the url (here 888888) and puts it in an embedded data field called mtwid; the full link is http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG&mtwid=888888

– generates a code number mturkcode

– embeds the flash file using swfobject, and transmits mtwid and mturkcode as flashvars

The scriptingRT file imports mtwid and mturkcode, displays it, and appends it to the data it saves.

I exported the qualtrics survey, maybe that’s useful. See also the examples page for for the mturk.swf and mturk.html files.

Survey Flow should look like this. Note that all of these variable names are case sensitive, if you put mturkcode here, it cannot be MTurkCode later.

HTML to put into a text field in HTML View:

<script type="text/javascript" src="http://scriptingrt.net/swf/swfobject.js"></script>
  <script type="text/javascript">
   var flashvars = {};
   flashvars.mtwid = "${e://Field/mtwid}";
   flashvars.mturkcode = "${e://Field/mturkcode}";
   var params = {};
   params.play = "true";
   params.loop = "false";
   params.menu = "true";
   params.allowfullscreen = "true";
   params.allowscriptaccess = "always";
   var attributes = {};
   swfobject.embedSWF("http://dl.dropbox.com/u/1009114/scriptingrt/mturk.swf", "myAlternativeContent", "800", "600", "9.0.0", false, flashvars, params, attributes);
  </script>
 </head>
 <body>
  <div id="myAlternativeContent">
   <a href="http://www.adobe.com/go/getflashplayer">
    <img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player" />
   </a>
  </div>
<p>Only proceed to the next page after you are told that the data are saved.</p>

How to connect Qualtrics and mturk, Part II

(Updated 15.02., with a description on how to add the confirmation number as a safety net.)

Some time ago, I posted our simple method on how to connect mturk and Qualtrics data. The idea was then to generate a random number in Qualtrics, save that in the dataset as an embedded data field, tell it to the participants at the every end of the study, after which they copy it into a text field in mturk and hit submit.

This method worked reasonably well in many studies, but it has its weaknesses, most importantly that wrong codes end up in the mturk database for probably many reasons. Sometimes, people forget to copy, or don’t see the code, or loose connection to Qualtrics, etc. In short, it would be nice to have a more automated backup connection.

Here I describe how this can be achieved with the workerId. This ID is unique for every MTurk worker, and Amazon makes it available as soon as a worker accepts a HIT (thanks to Winter Mason for suggesting this).

First the technical background:

The workerId is accessible in the source of the frame that presents the HIT. You can try this by accepting a HIT, then (using Firefox) right-click on the frame with the HIT, choose This Frame –> Frame Info, and then look at the location.

Here is an example:

https://s3.amazonaws.com/mturk_bulk/hits/54314710/xIpb71zULpXQALR7MKF2DA.html?assignmentId=27VV0FK0COK270BOW2BKGLRW87KIPY&hitId=2CZ8MZ9O9Z6KKQ9OZC8HYRLFQZ9A1G&workerId=A31DOCT85I******&turkSubmitTo=https%3A%2F%2Fwww.mturk.com

You see the workerId (which is mine, and I masked the end with asterisks) in the middle of the path.

To extract this workerId and submit it to Qualtrics, I wrote a little javascript that has to go into the mturk task.

The html and script:

<p>Go to <a href="http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG"
 target="_blank" onClick="addId()">this website and work on the task there</a>
 (link opens new page).

<p>At the end, you will get a confirmation number. Copy that number and
paste it into the field below, and then submit the HIT.</p>

<script type="text/javascript">
//from here on leave everything as it is
function addId(){
    //setting default values. If the parsing of the workerId fails,
    //99999 will be transmitted.
    var workerId="99999";
    var iFrameURL=document.location.toString();
    var temp="";
    //parsing and extracting the workerId
    if (iFrameURL.indexOf("workerId") > 0){
        if (iFrameURL.indexOf("?") > 0){
            temp=iFrameURL.split("?")[1];
            if (temp.indexOf("&") > 0){
                temp=temp.split("&")[2];
                if (temp.indexOf("=") > 0){
                    workerId=temp.split("=")[1];
                }
            }
        }
    }
    //appending the workerId to the link
    document.links[0].href+="&mtwid=" + workerId;
}
</script>
<p>&nbsp;</p>
<p>&nbsp;</p>
<p>confirmation number:</p>
<p><textarea rows="3" cols="80" name="answer"></textarea></p>

The first two paragraphs are HTML and simply add the link. The link contains an onClick event handler that executes the javascript function in the script before the link is followed. If the browser has no javascript activated, the link is followed without the workerId being added.

The javascript function takes the location of the frame, and tries to extract the workerId. If the extraction fails, the workerId keeps the default value of 99999. The workerId is appended to the link, using the syntax that Qualtrics expects.

If it works, the link will call: http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG&mtwid=99999, with the correct workerId instead of 99999.

How to actually do it:

1. Create your HIT in mturk. Make sure you do that via creating a HIT template, not by “create HIT manually” – in the latter case, the workerID does not seem to be available. When designing the layout, switch to “Edit HTML Source” and enter the code above. Obviously, you want to change the link to the actual link to your Qualtrics survey, but be careful to leave all quotes etc. as they are.

When you preview the HIT, it will look like this:

This is also what workers will see.

In Qualtrics, add an embedded data field and call it mtwid – this is what is appended to the link in the form of … .html?mtwid=999999. It will look like this:

That’s already it. The embedded data field is saved in your data file.

For testing purposes, you can ask Qualtrics to display the content of the embedded data field, like this:

When the link is followed, this will produce this qualtrics page:

 

Adding a safety net: Giving a confirmation number.

I still recommend that in the end, Qualtrics should give a confirmation number that has to be pasted back into qualtrics. It directs participants back to mturk and make sure they actually submit and don’t loose their HIT because of going over the time limit. Also, if a worker has javascript turned of, the transmission will fail, and the confirmation number is then a backup to match datasets.

In a nutshell, here is how to do this:

In the survey flow, add a web service element that gets a random number from a qualtrics web service, and stores it in an embedded data field that we call mturkcode

It’s important to place this at the beginning of your survey, before the actual question blocks.

The web service URL is http://reporting.qualtrics.com/projects/randomNumGen.php. You can choose any name for the embedded data field you like (instead of mturkcode), but the “random” has to be exactly that. min and max will be the limits of the returned random number.

Then, at the end of the survey, you tell the generated number to your participants. This is done by using the piped text function, which you can easier click on while editing a text field, or directly entering the text as shown below:

or, in plain text: “Thank you for taking part in this study. Your validation code for mTurk is ${e://Field/mturkcode}. Please press on the continue button >>> one more time.”

In MTurk, you then ask for this confirmation number before people submit the HIT.

For more info, see the earlier post on this.

Some additional notes:

  • I hope this is not violating any mTurk policies. I read them carefully and don’t think it does. If you think otherwise, please let me know.
  • Testing this in the mturk Sandbox does not work, as no workerId is assigned there for a task.
  • While creating this, I learned something that might be useful in the future: To store a value in an embedded data field in Qualtrics from Javascript, it’s possible to use a hidden input field: <input type=hidden id=’myField’ name=’ED~myED’ value=’remember_this’> will save “remember_this” in the embedded data field myED, which has to be created in the survey flow.

how to setup an online experiment in DMDX

 

J. Forster has devised a version of how to run a DMDX experiment online, in what he calls remote testing mode. It took us a while to exactly how he did it, so we document it here.

We followed instructions for method 3, where results are posted to the server at the University of Arizona and also emailed.

1. Download the package that contains the communication test from http://www.u.arizona.edu/~jforster/dmdx/commstest3.exe. This is a self-extracting zip file. You can simply extract it by opening it in winzip, or, even easier, rename it to .zip and then extract it using Windows. From this package, you need the poster.exe and dmdx.exe. The .bat is the same that is described in the remote testing overview in the section “The final Yacht solution”.

2. Now assemble your own package. You need the poster.exe, dmdx.exe, your finished dmdx .rtf script, and an adapted .bat file. Our .bat file looks like this:

start /wait "DMDX" dmdx.exe -auto -run power.rtf
if not exist power.azk goto end
poster.exe /cgi-bin/unloadazk4web email=schubert@igroup.org subject=unloadazk4web_power -iemailaddr results=power.azk
if errorlevel 1 pause
:end

The steps:

start… starts DMDX in the easiest mode possible (-auto) that simply accepts the desktop screen resolution and refresh rate settings. It executes the experiment script power.rtf and waits until it’s done.

if the results are saved in the power.azk file, it takes those results and posts them to the dmdx server at the university of Arizona, in addition to emailing a copy to my email address. If there was a problem, it will pause to show the error messages.

You can test your .bat file by simply executing it on your computer. The results should show up when you visit the server. You will see that your experiment file name includes a long number – the hash tag that is created automatically.

3. Now you need to pack it the four files (your .bat, dmdx.exe, poster.exe, and your .rtf) in a self-extracting zip file that automatically executes one element after unpacking. Winzip’s self-extractor is one option to create them. Apparently, there are also open source alternatives. Put the 4 files into a .zip file, but be careful to only pack the files themselves, not the folder that contains them – otherwise, the folder name is included in the path and creates all kinds of problems.

4. If you use winzip self-extractor, you choose to create a “Self-extracing Zip file for Software Installation”, and specify the zip’s file name. We left the Message box empty. Then you choose “Unzip automatically” and “Run as a user”. The command to issue after unzipping is ./dmdx.bat, or whatever name you gave your .bat. As the message for the Software Installation Dialog, we entered: “The experimental software (DMDX) will not actually install. It will just run, send the results to the server and then be deleted. Just click the Setup button.” In the About box, we entered ” This experimental software DMDX is a tachistoscopic experimental display presentation and experiment controller program written at the University of Arizona by K.I.Forster and J.C. Forster”. This should be all to create your .exe file. Test it.

5. Now you only need to put this online and convince your participants that they can trust you enough to download an .exe file and execute it.

 

how to connect Qualtrics and mTurk

Update: For adding another backup to this method, see this post.

Update: You can download a qualtrics survey and import it that shows an example.

we ran several studies using qualtrics and recruiting participants on mturk. We connect them by linking from mTurk to the survey in qualtrics. Participants fill in the survey at qualtrics, and get in the end a long number code that they copy and paste into the form at mTurk, thereby proofing their participation. I think I read this method somewhere, but cannot remember the original source.

Here are the technical details:

In the survey flow of the qualtrics survey, I added a Web Service that fills an embedded data field with a randomly generated number.

  1. Click Survey Flow
  2. Add a new element of the type Web Service
  3. as its URL, enter http://reporting.qualtrics.com/projects/randomNumGen.php
  4. Add two parameters: one that you call min = 1, and one that you call max = 9999999. Of course you can choose other min or max values.
  5. then click below to get an embedded data field, and specify it to be mTurkCode = random (the name of the embedded data field is up to you, but the right side has to be random – you can see this when you click on Test URL, there you see that the value returned from the web service is called random)
  6. move your web service block to the top of the survey flow. This is crucial, as the data field only gets created after this element has been processed. If you leave the web service element at the end of your experiment, you will not see anything when you try to access it.
  7. Save your flow

It looks like this:

UPDATE. SORRY, I JUST REALIZED THERE IS A MISTAKE. THE SECOND LINE SHOULD BE min, NOT mi

(click to enlarge)

Now, at the end of your survey, you have to tell the code to the participant. You add a last page with a text that says something like this:

Thank you for taking part in this study.

Your validation code for mTurk is ${e://Field/mTurkCode}

please press on the continue button >>> one more time.

the text ${e://Field/mTurkCode} will be replaced by qualtrics with the stored number. If you don’t want to type it, you can generate it while editing the text by clicking Piped Text, then Embedded Data field, then write mTurkCode

Don’t forget to test your survey and to make sure that the code number actually appears.

Then, in mTurk, you design a HIT that only requires the entry of the number code. It will look like this:

(Click to enlarge)

In the end, you will have the code of each participant in your qualtrics results database and in the results file that you get from mTurk. You then have to compare the lists and make sure that all numbers entered at mTurk have a counterpart in qualtrics. All these can be approved.