Artiruno notebook

Created 5 Jul 2021 • Last modified 7 Oct 2022

Basic idea for an Internet study

Collect basic demographic data (after the main part of the study)

Ask subjects about a weighty decision they have to make soon which they're not already sure about and whose outcome they'll be able to assess soon (let's say, 1 month from now).

Have them briefly describe the decision to be made and the options in prose.

Conditions:

  • Artiruno condition: Then they construct the options and do VDA.
  • Comparison: Nothing else. If this study works, then maybe there could be a follow-up to investigate the effect by comparing the Artiruno condition to a condition where subjects construct options in the same style, but don't actually use VDA.

Follow up a month later.

  • Remind them of what they wrote about the decision problem (but not their options or Artiruno's suggestion).
  • Have them briefly describe in prose what decision they made, what the outcome was, and how happy they are with the outcome.
  • Have them rate the outcome on numeric scales.
  • Have them rate the decision-making process on numeric scales.

Pilot

In the pilot, just try having people set up the decision problem, so you can see that people are making reasonable choices of criteria, levels, and decision problems.

Piloting on MTurk was a failure, apparently because of the poor English skills of the subjects I got. Let's try some pre-screened subjects from Reddit instead.

I ran four screened subjects from Reddit and am reasonably happy with the results. The instructions changes these findings suggest are:

  • Mention that you should only list alternatives if they're options you really have, not options you would ideally have.
  • Provide an example of a yes-or-no decision.

Plans for the real study

To start with, aim for 40 subjects, 20 per condition, who return; this means you should probably try to recruit 80 subjects and offer more money for session 2 than session 1. You probably won't be able to get this many from /r/samplesize, but you can check other places where studies are posted, or maybe even hire a service that connects social scientists to respondents.

Set expiration times to a week and a day since your invitation.

  • Session 1 (internally, visits 1 and 2)
    • Warn about performance. ("can be particularly slow on phones and tablets")
    • Allow for session 2 in the consent form. Be clear that the subject is expected to be in both sessions, but the second will be quite short.
    • [Offer how much money?]
    • Ask for a decision description and an expected decision resolution date, as in the pilot.
    • Randomly assign the subject to the VDA condition or the control condition. (If the subject number, among subjects who've gotten this far in this version of the study, is even, counting the first as 1, then use the opposite condition of the previous subject.) Write the study state to disk here, so the subject can't refresh and enter different values depending on his assigned condition.
    • If the subject is in the control condition, skip to the end. Otherwise, continue.
    • Solicit criteria and alternatives, as in the pilot.
    • Conduct VDA.
      • Display a reminder of the subject's chosen criteria and alternatives on this page. Explain that the subject doesn't have to make the choice suggested by Artiruno, but it could be a good idea.
      • Show results as text. (The graph would probably not be very helpful in this situation, with find_best = 1 and non-quantitatively minded users.)
      • Allow the subject to restart the procedure, or to return to the previous page and edit criteria and alternatives; trying to record everything is probably futile by the nature of JavaScript.
      • Don't provide an abort button. (It wouldn't be useful because in the case of find_best = 1 and not showing a graph, Artiruno can provide no useful information before it's done.)
      • Record the questions asked, the subject's choices, the subject's response times, and Artiruno's conclusion. If the subject restarted VDA, use the values from the final round.
    • Ask for any comments.
  • Session 2 (internally, visit 3)
    • Send an invitation 1 month after the subject completed session 1. Say that session 2 assumes the subject made the choice and got to see at least a little of the outcome; if that won't be true for a while, the subject should reply saying when would be a good time to do session 2.
    • [Offer how much money?]
    • Re-display the decision description and expected resolution date.
    • Ask if they've made the choice and gotten to see some of the outcome. (In theory, this should always be "yes", because of the instructions earlier. It's a sanity check.)
    • Have the subject briefly describe in prose:
      • which choice they made
      • what the outcome was
      • how happy they are with the choice and outcome
    • Use 1-to-5 rating scales for the below.
    • How pleased are they with the outcome of their choice?
    • How well-chosen was their choice, given what they knew at the time they made it?
    • How difficult did it feel to make the decision?
    • In the VDA condition:
      • Show the criteria, alternatives, and results text from VDA.
      • How consistent did they feel their choice was with Artiruno's suggestion?
      • How difficult was the procedure (including writing up the criteria and alternatives) to do?
      • How helpful did the procedure feel for making the decision-making process?
    • Debrief
    • Another comments box

Results of running on Prolific

A lot of people who seem to otherwise mostly take well to the task are messing up basic things in the VDA problem setup, particularly, putting criterion levels in the reverse order:

(.sum (getl ratings (sorted (&
  (set (ssi subjects (<= $last_tv 7)))
  (set ratings.index)))))
I value
reversed_criterion 10
nonsense 5
desc_mismatch 2

I should probably take some time to add more checks (e.g., an extra screen where subjects have to confirm that they put criterion levels in the right order) and rerun from scratch rather than have this much compromise of the VDA sample.

Reminders for analysis

  • If subjects refresh the page and redo parts of the task, the timing data you get will only reflect the final attempt.