scriptingRT
blog of the flashdmdx project – We develop and use tools for online psychological research.
Posted by …
120 Hz flatscreens for priming studies
120 Hz flatscreens can now do what CRTs were used for: precise, short presentation of stimuli. check out this paper by Wang and Nikolic.
Posters on scriptingRT at APS in Chicago and APPE in Lisboa
We will present scriptingRT with a poster at APPE in Lisboa. Carla and Elizabeth will also present a poster on scriptingRT at the 2012 APS Annual Convention, May 24 – 27, 2012 in Chicago
Murteira, C.; Schubert, T., Lopes, D., & Collins, E. (2012). ScriptingRT: A new tool to collect reaction time data online. Poster to be presented at the 7. Encontro Nacional da Assoiação Portuguesa de Psicologia Experimental, March 16 – 17, 2012, Lisboa, Portugal.
Abstract:
Online research is a standard tool of psychological research. However, collecting reaction time data online currently requires specialized programming skills. ScriptingRT is a free open source software library that supports the development of such studies. ScriptingRT was developed using Adobe Flex. Experiments are programmed in an XML-based syntax, run as Flash applications in any Internet browser with a Flash plugin, and can be combined with HTML surveys.
Three studies tested the performance of ScriptingRT. Standard effects (e.g. Stroop, Simon) were reliably replicated using ScriptingRT, when run in the lab and online. In direct comparison to DMDX and E-Prime, effect sizes were slightly smaller, but nevertheless significant. The results confirm the validity of ScriptingRT (http://reactiontimes.wordpress.com/) to measure reaction times.
Murteira, C.; Schubert, T., Lopes, D., & Collins, E. (2012). ScriptingRT: A new tool to collect reaction time data online. Poster to be presented at the 2012 Association for Psychological Science Annual Convention, May 24 – 27, 2012, Chicago, IL, USA.
Abstract:
The internet is a powerful tool for collecting data for psychological research, with a large pool of diverse participants. As the number and variety of people using the internet has expanded, and specialized websites for data collection have emerged, the value of tools to simplify the transition of traditionally laboratory computer based psychology experiments to an online environment has increased. This stimulated the development of ScriptingRT, a free open source software library that supports the easy development of studies collecting reaction time measures. ScriptingRT is a Flash application developed with Adobe Flex. Experiments are programmed using a simple XML-based syntax. More advanced or adventurous coders can include ActionScript 3.0 code to add advanced features. ScriptingRT runs on any Internet browser with a Flash plugin installed. The program measures only reaction times (RT), but can be combined with HTML based surveys which can collect many other types of data.
Although ScriptingRT (http://reactiontimes.wordpress.com/) is in development, it is currently available for public use, and we have replicated strong standard reaction time paradigms using this tool (all data were analysed using hierarchical mixed models). A Stroop task, programmed in ScriptingRT, linked with a survey created in Qualtrics and shared on Facebook (N=19) showed the standard Stroop effect. The same procedure was used to conduct a replication of an experiment on the interference of verticality in decisions about power. This used Amazon’s Mechanical Turk (www.mturk.com) to gather participants. Powerful and powerless words were displayed in two font sizes (large and small). For each word participants decided if the word represented a powerful or a powerless group/person. The results(N=83) replicated the interference of font size with power judgment previously found in the laboratory, shown by longer reaction times on incongruent trials (large font/powerless word or small font/powerful word combinations) compared to congruent trials (large font/powerful word or small font/powerless word combinations).
In addition to the online tests of the software, laboratory tests were conducted to compare ScriptingRT with other reaction time measurement software. In one experiment a Stroop and a Simon task were programmed in both ScriptingRT and DMDX. Participants (N=28) completed all four tasks (order counterbalanced). Response latencies were compared for congruency effects and across software. In both cases we replicated the standard effect, with no interaction with the software, although ScriptingRT indicated significantly longer latencies overall. In another study, participants (N=27) completed two versions of a Stroop task programmed in ScriptingRT and E-prime respectively. Again, we found a standard Stroop main effect without effects of software. Overall, in direct comparison to DMDX and E-Prime, effect sizes were slightly smaller, but nevertheless significant.
Tests with a wider variety of tasks, including an IAT and a Shooter Paradigm are in progress. However, even with only the current results, it seems that ScriptingRT is a useful tool for creating online experiments that collect reliable reaction time measures.
smartphone reaction times
if you are interested in collecting RT data with smartphones (something scriptingRT cannot do yet), check this out ScienceXL, including one publication in PlosOne
embedding scriptingRT SWFs in HTML
in recent weeks, we switched to embedding SWFs using the swf object, and it works well. here is how to do it:
the page describing swfobject, and providing all downloads: http://code.google.com/p/swfobject/
We put a copy of swfobject.js on our server scriptinrt.net: http://scriptingrt.net/swf/swfobject.js
and also a copy of the handy generator html page
in Qualtrics, we paste this code, which also transmits two flashvars (You can delete these two lines if you don’t need them)
<script type="text/javascript" src="http://scriptingrt.net/swf/swfobject.js"></script>
<script type="text/javascript">
var flashvars = {};
flashvars.mtwid = "${e://Field/mtwid}";
flashvars.mturkcode = "${e://Field/mturkcode}";
var params = {};
params.play = "true";
params.loop = "false";
params.menu = "true";
params.allowfullscreen = "true";
params.allowscriptaccess = "always";
var attributes = {};
swfobject.embedSWF("http://dl.dropbox.com/u/1009114/scriptingrt/mturk.swf", "myAlternativeContent", "800", "600", "9.0.0", false, flashvars, params, attributes);
</script>
<div id="myAlternativeContent">
<a href="http://www.adobe.com/go/getflashplayer">
<img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player">
</a>
</div>
<p>Click on the START button above to begin the task.</p>
<p>Only proceed to the next page after you are told that the data are saved.</p>
careful with mx:Text width in scriptingRT
We had a problem today that I want alert users to. A compiled scriptingRT study was somehow larger than it was supposed to be: Even though compiled with width and height set to 640 and 480 and embedded in an object of the same size, scroll bars appeared on the HTML page, and the flash object seemed to be “bigger”. We traced the error to a mx:Text object, for which width was set to “100%”. When we removed this width attribute, or set it to “640″, everything was fine. mx:Text‘s width attribute expects numbers of pixels, and does funny things with %. If you want to use percent, use the attribut percentWidth
connecting mturk to qualtrics to scriptingrt
After describing how to get the mturk worder ID into qualtrics, how to get qualtrics data into scriptingRT, and how to use a code number to give mturk workers something to submit, I now put it all together in one template.
This qualtrics survey
- takes the mturk worker id as a value in the url (here 888888) and puts it in an embedded data field called mtwid; the full link is http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG&mtwid=888888
- generates a code number mturkcode
- embeds the flash file using swfobject, and transmits mtwid and mturkcode as flashvars
The scriptingRT file imports mtwid and mturkcode, displays it, and appends it to the data it saves.
I exported the qualtrics survey, maybe that’s useful. See also the examples page for for the mturk.swf and mturk.html files.
Survey Flow should look like this. Note that all of these variable names are case sensitive, if you put mturkcode here, it cannot be MTurkCode later.
HTML to put into a text field in HTML View:
<script type="text/javascript" src="http://scriptingrt.net/swf/swfobject.js"></script>
<script type="text/javascript">
var flashvars = {};
flashvars.mtwid = "${e://Field/mtwid}";
flashvars.mturkcode = "${e://Field/mturkcode}";
var params = {};
params.play = "true";
params.loop = "false";
params.menu = "true";
params.allowfullscreen = "true";
params.allowscriptaccess = "always";
var attributes = {};
swfobject.embedSWF("http://dl.dropbox.com/u/1009114/scriptingrt/mturk.swf", "myAlternativeContent", "800", "600", "9.0.0", false, flashvars, params, attributes);
</script>
</head>
<body>
<div id="myAlternativeContent">
<a href="http://www.adobe.com/go/getflashplayer">
<img src="http://www.adobe.com/images/shared/download_buttons/get_flash_player.gif" alt="Get Adobe Flash player" />
</a>
</div>
<p>Only proceed to the next page after you are told that the data are saved.</p>
new feature in scriptingRT
we learned how to use flashvars to transmit data from HTML (e.g, a qualtrics page) to scriptingRT. See a running example on qualtrics, example files here, and an explanation in the manual.
mistake for timout corrected
ScriptingRT update:
We found a mistake in the manual and some examples: When you want to specify a timeout for items in a testpart, use noresponse=”Time.3000″ (or similar), not an additional response= attribute.
doubletest in the examples section shows the correct syntax.
When this even occurs, “miss” is saved in the data.
How to connect Qualtrics and mturk, Part II
(Updated 15.02., with a description on how to add the confirmation number as a safety net.)
Some time ago, I posted our simple method on how to connect mturk and Qualtrics data. The idea was then to generate a random number in Qualtrics, save that in the dataset as an embedded data field, tell it to the participants at the every end of the study, after which they copy it into a text field in mturk and hit submit.
This method worked reasonably well in many studies, but it has its weaknesses, most importantly that wrong codes end up in the mturk database for probably many reasons. Sometimes, people forget to copy, or don’t see the code, or loose connection to Qualtrics, etc. In short, it would be nice to have a more automated backup connection.
Here I describe how this can be achieved with the workerId. This ID is unique for every MTurk worker, and Amazon makes it available as soon as a worker accepts a HIT (thanks to Winter Mason for suggesting this).
First the technical background:
The workerId is accessible in the source of the frame that presents the HIT. You can try this by accepting a HIT, then (using Firefox) right-click on the frame with the HIT, choose This Frame –> Frame Info, and then look at the location.
Here is an example:
You see the workerId (which is mine, and I masked the end with asterisks) in the middle of the path.
To extract this workerId and submit it to Qualtrics, I wrote a little javascript that has to go into the mturk task.
The html and script:
<p>Go to <a href="http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG"
target="_blank" onClick="addId()">this website and work on the task there</a>
(link opens new page).
<p>At the end, you will get a confirmation number. Copy that number and
paste it into the field below, and then submit the HIT.</p>
<script type="text/javascript">
//from here on leave everything as it is
function addId(){
//setting default values. If the parsing of the workerId fails,
//99999 will be transmitted.
var workerId="99999";
var iFrameURL=document.location.toString();
var temp="";
//parsing and extracting the workerId
if (iFrameURL.indexOf("workerId") > 0){
if (iFrameURL.indexOf("?") > 0){
temp=iFrameURL.split("?")[1];
if (temp.indexOf("&") > 0){
temp=temp.split("&")[2];
if (temp.indexOf("=") > 0){
workerId=temp.split("=")[1];
}
}
}
}
//appending the workerId to the link
document.links[0].href+="&mtwid=" + workerId;
}
</script>
<p> </p>
<p> </p>
<p>confirmation number:</p>
<p><textarea rows="3" cols="80" name="answer"></textarea></p>
The first two paragraphs are HTML and simply add the link. The link contains an onClick event handler that executes the javascript function in the script before the link is followed. If the browser has no javascript activated, the link is followed without the workerId being added.
The javascript function takes the location of the frame, and tries to extract the workerId. If the extraction fails, the workerId keeps the default value of 99999. The workerId is appended to the link, using the syntax that Qualtrics expects.
If it works, the link will call: http://isctecis.us2.qualtrics.com/SE/?SID=SV_bOVwgxx4d2aFvJG&mtwid=99999, with the correct workerId instead of 99999.
How to actually do it:
1. Create your HIT in mturk. Make sure you do that via creating a HIT template, not by “create HIT manually” – in the latter case, the workerID does not seem to be available. When designing the layout, switch to “Edit HTML Source” and enter the code above. Obviously, you want to change the link to the actual link to your Qualtrics survey, but be careful to leave all quotes etc. as they are.
When you preview the HIT, it will look like this:
This is also what workers will see.
In Qualtrics, add an embedded data field and call it mtwid – this is what is appended to the link in the form of … .html?mtwid=999999. It will look like this:
That’s already it. The embedded data field is saved in your data file.
For testing purposes, you can ask Qualtrics to display the content of the embedded data field, like this:
When the link is followed, this will produce this qualtrics page:
Adding a safety net: Giving a confirmation number.
I still recommend that in the end, Qualtrics should give a confirmation number that has to be pasted back into qualtrics. It directs participants back to mturk and make sure they actually submit and don’t loose their HIT because of going over the time limit. Also, if a worker has javascript turned of, the transmission will fail, and the confirmation number is then a backup to match datasets.
In a nutshell, here is how to do this:
In the survey flow, add a web service element that gets a random number from a qualtrics web service, and stores it in an embedded data field that we call mturkcode
It’s important to place this at the beginning of your survey, before the actual question blocks.
The web service URL is http://reporting.qualtrics.com/projects/randomNumGen.php. You can choose any name for the embedded data field you like (instead of mturkcode), but the “random” has to be exactly that. min and max will be the limits of the returned random number.
Then, at the end of the survey, you tell the generated number to your participants. This is done by using the piped text function, which you can easier click on while editing a text field, or directly entering the text as shown below:
or, in plain text: “Thank you for taking part in this study. Your validation code for mTurk is ${e://Field/mturkcode}. Please press on the continue button >>> one more time.”
In MTurk, you then ask for this confirmation number before people submit the HIT.
For more info, see the earlier post on this.
Some additional notes:
- I hope this is not violating any mTurk policies. I read them carefully and don’t think it does. If you think otherwise, please let me know.
- Testing this in the mturk Sandbox does not work, as no workerId is assigned there for a task.
- While creating this, I learned something that might be useful in the future: To store a value in an embedded data field in Qualtrics from Javascript, it’s possible to use a hidden input field: <input type=hidden id=’myField’ name=’ED~myED’ value=’remember_this’> will save “remember_this” in the embedded data field myED, which has to be created in the survey flow.
