slides from a 2012 spsp symposium…

… on using mturk for research

How to repeatedly assign participants to conditions in Qualtrics

In one of our studies using Qualtrics, we came across the problem of randomly assigning participants to different experimental conditions repeatedly.

The point was that we wanted everybody to experience a manipulation depending on condition, then have everybody fill in the same measures, and then present some more materials again depending on condition.

Randomizing the experimental conditions ONCE is easy, but how to do that repeatedly?

Our quite simple solution was to create embedded data fields that store the assignment to condition, and then branch according to that field.

To do this, in the survey flow you set up in the beginning a randomizer

Click survey flow and add a new element: Randomizer

Under this element, add the conditions you want to randomize by pressing add new element here: Embedded data.

On the embedded data element, type “condition” and the label of the condition (e.g. number) on the value field. Do this for every condition.

Go back to the randomizer and choose to “randomly present 1 of the following elements” and tick “evenly present elements“.

At this point, it should look like this (example with four conditions):

Below, add a new element: Branch

In branch, select “If” Embedded Data, type “condition” and select Is Equal To, and type the name/number of the condition

something like this:

Under the branch add a new element, that will be the block containing the desired stimulus for that condition

Do the same for all the experimental conditions.

it should look like this (again, example with four conditions):

In this example, the Blocks Entitativity and ManCheck are done for all participants in the same manner.

After that, you can then repeat the branching and do more condition-specific manipulations or measures.

If you have the problem that there is only very little change between your conditions, for instance a different label, there is alternative: Store the label itself in an embedded data field and insert it with Piped Text.

(written by Pedro Marques and Thomas)

editing scriptingRT in FlashBuilder

In our projects so far, we used the free Notepad++ to edit our .mxml files. It does nice syntax coloring (if you add .mxml as an extension for XML in the style configurator).

But now, in the mailing list the question came up whether you can also use Adobe’s professional editing software FlashBuilder. In fact, Adobe is giving Flash Builder away for free to the education sector.

The answer is yes, you can use FlashBuilder to create scriptingRT experiments.

After download and install, simply create a new project, then save a copy of scriptingrtXXX.swc in the libs folder, and you are ready to edit your .mxml file. To test, use Run, to export the finished script, Export a Release Build from the Project menu. This also builds a html page.

I haven’t tested this much, but I can see two immediate advantages: The syntax coloring works for both XML and ActionScript in the same file, while in Notepad++ the embedded ActionScript is not colored. I imagine that you also get enhanced debugging options this way.

why flex instead of html5 & javascript

the question why we use flex instead of javascript keeps coming up after the PR debacle around flash.

here is an interesting post on the future of flex and its advantages over html5. It’s actually written by a Portuguese flex developer whose office is probably within walking distance from my university iscte, and who even studied here.

and: flex is now called apache flex instead of adobe flex

Slides from the workshop

Winter Mason’s slides from Day 1 on online data collection as crowd sourcing

Thomas’ slides from Day 2 on scriptingRT

playing with the IAT

great “gamification” of the IAT. why are not all tests like that?

Where scriptingRT stands

Some thoughts about the current state of the project and the software after the workshop:

1. We had altogether 29 participants at the workshop (excluding the team), and we had about 20 more applications we could not accept because of space limitations. This great interest tells us that there is really a need for what we are doing, which is very satisfying. Thanks!
2. After I explained how ScriptingRT experiments are programmed in XML, and participants started to try it, somebody remarked: “You did not say it was easy, you said it was free.” Exactly. It became obvious that writing XML needs time getting used to, despite all the experience people have with HTML these days. But, as another memorable quote from the workshop says, “2012 is the year of coding“, and expertise in coding acquired with ScriptingRT might come in handy in many other fields.
3. We have collected data with scriptingRT already, and our empirical tests show that it can replace professional desktop software for paradigms with large effect sizes. Nevertheless, as you can download it today, ScriptingRT is NOT ready to be used for hypothesis-testing data collection. There are a number of bugs that need to be fixed first (see download page), and we have to make sure that data are only accessible to the researchers. We expect that we need another month to solve these issues.
4. I believe that further development of the project hinges upon the development of an active user group. The workshop hopefully created a first critical mass for that group. For that purpose we created the Google group for online discussion. Please sign up!
5. The project is still funded until September 2012. Until then, we can pay our programmer Frank and Carla for support. After that, and until we have found further funding, development is in the hands of volunteer contributors. The source code of the next release will be made available on Github to allow external contributions. Note that we make the source code available under a license that allows its commercial use. This will hopefully increase contributions to it.

Impressions from workshop

This week, we held our first workshop on Online Data Collection and ScriptingRT. Thanks to all who joined for your attention, smart questions, all the expertise that you brought and that made it much more than just lectures, and then for actually sitting down and starting to code in ScriptingRT. In my humble opinion, it was a big success!

I list the participants below so that people interested in the topic can approach them:

Aline Nunes, CIS/ISCTE, Portugal

Ana Louceiro, CIS/ISCTE, Portugal

Beatriz Lloret, CIS/ISCTE‐IUL, Portugal

Carla Murteira, CIS/ISCTE_IUL, Portugal

Caroline Leygue, University of Nottingham, UK

Cláudia Simão, CIS/ISCTE, Portugal

Cristina Godinho, CIS/ISCTE, Portugal

Diniz Lopes, CIS/ISCTE‐IUL, Portugal

Elizabeth Collins, CIS/ISCTE‐IUL, Portugal

Hugo Alves, ISPA, Portugal

Inês Dias, ISPA, Portugal

Joana Costa do Carmo, University of Lisbon, Portugal

Joel Vuolevi, Vrije Universiteit, Netherlands

Joseph Mole, St. Andrews, Scotland

Manuel Oliveira, ISPA, Portugal

Mauro Bianchi, CIS/ISCTE‐IUL, Portugal

Michael van den Oudenalder, Utrecht University, Netherlands

Miriam Rosa, CIS/ISCTE, Portugal

Nuno Costa, CIS/ISCTE, Portugal

Patrícia Arriaga, CIS/ISCTE, Portugal

Paulo Vitória, Beira Interior University/CIS, Portugal

Pedro Marques, CIS/ISCTE, Portugal

Piers Fleming, University of East Anglia, UK

Pilar Aguilar, Universidad Autonoma, Spain

Ricardo Rodrigues, CIS/ISCTE‐IUL, Portugal

Rita Meneses, Portugal

Rodrigo Brito, CIS/Lusófona, Portugal

Rui Costa Lopes, ICS‐UL, Portugal

Rui Gaspar, CIS/ISCTE‐IUL, Portugal

Sónia Gonçalves, Instituto Piaget/CIS, Portugal

Susana Lavado, ICS/FPUL, Portugal

Thomas Schubert, CIS/ISCTE‐IUL, Portugal

Tiago Aguiar, Porto University, Portugal

Winter Mason, Stephens Institute of Technology, USA

qualtrics wiki

very useful: University of Michigan Qualtrics wiki

workshop page updated

The workshop page now has information on venue and schedule.