Slides from the workshop

Winter Mason’s slides from Day 1 on online data collection as crowd sourcing

Thomas’ slides from Day 2 on scriptingRT


how to setup an online experiment in DMDX


J. Forster has devised a version of how to run a DMDX experiment online, in what he calls remote testing mode. It took us a while to exactly how he did it, so we document it here.

We followed instructions for method 3, where results are posted to the server at the University of Arizona and also emailed.

1. Download the package that contains the communication test from This is a self-extracting zip file. You can simply extract it by opening it in winzip, or, even easier, rename it to .zip and then extract it using Windows. From this package, you need the poster.exe and dmdx.exe. The .bat is the same that is described in the remote testing overview in the section “The final Yacht solution”.

2. Now assemble your own package. You need the poster.exe, dmdx.exe, your finished dmdx .rtf script, and an adapted .bat file. Our .bat file looks like this:

start /wait "DMDX" dmdx.exe -auto -run power.rtf
if not exist power.azk goto end
poster.exe /cgi-bin/unloadazk4web subject=unloadazk4web_power -iemailaddr results=power.azk
if errorlevel 1 pause

The steps:

start… starts DMDX in the easiest mode possible (-auto) that simply accepts the desktop screen resolution and refresh rate settings. It executes the experiment script power.rtf and waits until it’s done.

if the results are saved in the power.azk file, it takes those results and posts them to the dmdx server at the university of Arizona, in addition to emailing a copy to my email address. If there was a problem, it will pause to show the error messages.

You can test your .bat file by simply executing it on your computer. The results should show up when you visit the server. You will see that your experiment file name includes a long number – the hash tag that is created automatically.

3. Now you need to pack it the four files (your .bat, dmdx.exe, poster.exe, and your .rtf) in a self-extracting zip file that automatically executes one element after unpacking. Winzip’s self-extractor is one option to create them. Apparently, there are also open source alternatives. Put the 4 files into a .zip file, but be careful to only pack the files themselves, not the folder that contains them – otherwise, the folder name is included in the path and creates all kinds of problems.

4. If you use winzip self-extractor, you choose to create a “Self-extracing Zip file for Software Installation”, and specify the zip’s file name. We left the Message box empty. Then you choose “Unzip automatically” and “Run as a user”. The command to issue after unzipping is ./dmdx.bat, or whatever name you gave your .bat. As the message for the Software Installation Dialog, we entered: “The experimental software (DMDX) will not actually install. It will just run, send the results to the server and then be deleted. Just click the Setup button.” In the About box, we entered ” This experimental software DMDX is a tachistoscopic experimental display presentation and experiment controller program written at the University of Arizona by K.I.Forster and J.C. Forster”. This should be all to create your .exe file. Test it.

5. Now you only need to put this online and convince your participants that they can trust you enough to download an .exe file and execute it.


how to connect Qualtrics and mTurk

Update: For adding another backup to this method, see this post.

Update: You can download a qualtrics survey and import it that shows an example.

we ran several studies using qualtrics and recruiting participants on mturk. We connect them by linking from mTurk to the survey in qualtrics. Participants fill in the survey at qualtrics, and get in the end a long number code that they copy and paste into the form at mTurk, thereby proofing their participation. I think I read this method somewhere, but cannot remember the original source.

Here are the technical details:

In the survey flow of the qualtrics survey, I added a Web Service that fills an embedded data field with a randomly generated number.

  1. Click Survey Flow
  2. Add a new element of the type Web Service
  3. as its URL, enter
  4. Add two parameters: one that you call min = 1, and one that you call max = 9999999. Of course you can choose other min or max values.
  5. then click below to get an embedded data field, and specify it to be mTurkCode = random (the name of the embedded data field is up to you, but the right side has to be random – you can see this when you click on Test URL, there you see that the value returned from the web service is called random)
  6. move your web service block to the top of the survey flow. This is crucial, as the data field only gets created after this element has been processed. If you leave the web service element at the end of your experiment, you will not see anything when you try to access it.
  7. Save your flow

It looks like this:


(click to enlarge)

Now, at the end of your survey, you have to tell the code to the participant. You add a last page with a text that says something like this:

Thank you for taking part in this study.

Your validation code for mTurk is ${e://Field/mTurkCode}

please press on the continue button >>> one more time.

the text ${e://Field/mTurkCode} will be replaced by qualtrics with the stored number. If you don’t want to type it, you can generate it while editing the text by clicking Piped Text, then Embedded Data field, then write mTurkCode

Don’t forget to test your survey and to make sure that the code number actually appears.

Then, in mTurk, you design a HIT that only requires the entry of the number code. It will look like this:

(Click to enlarge)

In the end, you will have the code of each participant in your qualtrics results database and in the results file that you get from mTurk. You then have to compare the lists and make sure that all numbers entered at mTurk have a counterpart in qualtrics. All these can be approved.