Artiruno notebook

I'm particularly interested in software, and other means for making decisions in situations that aren't already fully quantitatively specified.

• Available
  • Moshkovich and Mechitov (2018) - https://verbaldecisionanalysis.wordpress.com/ - ORCON_Z (ZAPROS-LM) - still around
• Unavailable
  • Shevchenko, Ustinovichius, and Walasek (2019) - CLARA - no URL provided; the corresponding author didn't reply to email I sent 12 Aug 2020
  • Barbosa, Pinheiro, Silveira, and Filho (2019) - ORCLASSWEB - http://www2.unifor.br/OrclassWeb - dead link
  • A 2018 conference paper - Aranau - no URL provided; corresponding author didn't reply to email I sent on 28 Jun 2020
  • Ashikhmin and Furems (2005) - IDSS UniComBOS - http://iva.isa.ru/DSS - dead link

• Web interface
  • A button to prematurely abort
  • Allow setting allowed_pairs_callback
  • Show a graph of preferences
• For either interactive mode
  • Transcripts of choices and inferences
  • Saving of choices partway through the procedure, and later replaying them
  • Proper error messages instead of assertion failures
  • More validation of the interactive input
• A nicer interactive interface
• Allow the agent to reply "don't know", a la UniComBOS. (UniComBOS takes this to mean that a preference for the given situation should never be inferrable from other preferences, but I think it makes more sense to understand it as totally uninformative; i.e., "skip this question and do what you can with everything else or other questions".)
• More sophisticated alt levels
  • Allow an alternative to have a missing value (None) on a criterion
  • Allow an alt to have multiple values for a criterion
  • Allow an alt to have a fuzzy set of values for a criterion
• Print explanations of the final ranking or choice, a la ZAPROS
• Consistency checks
  • Check whether the agent's preferences are consistent using an all-worst-attributes reference instead of an all-best-attributes reference, a la ZAPROS
  • Check transitively deduced preferences multiple times, a la ZAPROS
• Support group decision-making

Collect basic demographic data (after the main part of the study)

Ask subjects about a weighty decision they have to make soon which they're not already sure about and whose outcome they'll be able to assess soon (let's say, 1 month from now).

Have them briefly describe the decision to be made and the options in prose.

Conditions:

• Artiruno condition: Then they construct the options and do VDA.
• Comparison: Nothing else. If this study works, then maybe there could be a follow-up to investigate the effect by comparing the Artiruno condition to a condition where subjects construct options in the same style, but don't actually use VDA.

Follow up a month later.

• Remind them of what they wrote about the decision problem (but not their options or Artiruno's suggestion).
• Have them briefly describe in prose what decision they made, what the outcome was, and how happy they are with the outcome.
• Have them rate the outcome on numeric scales.
• Have them rate the decision-making process on numeric scales.

In the pilot, just try having people set up the decision problem, so you can see that people are making reasonable choices of criteria, levels, and decision problems.

Piloting on MTurk was a failure, apparently because of the poor English skills of the subjects I got. Let's try some pre-screened subjects from Reddit instead.

I ran four screened subjects from Reddit and am reasonably happy with the results. The instructions changes these findings suggest are:

• Mention that you should only list alternatives if they're options you really have, not options you would ideally have.
• Provide an example of a yes-or-no decision.

To start with, aim for 40 subjects, 20 per condition, who return; this means you should probably try to recruit 80 subjects and offer more money for session 2 than session 1. You probably won't be able to get this many from /r/samplesize, but you can check other places where studies are posted, or maybe even hire a service that connects social scientists to respondents.

Set expiration times to a week and a day since your invitation.

• Session 1 (internally, visits 1 and 2)
  • Warn about performance. ("can be particularly slow on phones and tablets")
  • Allow for session 2 in the consent form. Be clear that the subject is expected to be in both sessions, but the second will be quite short.
  • [Offer how much money?]
  • Ask for a decision description and an expected decision resolution date, as in the pilot.
  • Randomly assign the subject to the VDA condition or the control condition. (If the subject number, among subjects who've gotten this far in this version of the study, is even, counting the first as 1, then use the opposite condition of the previous subject.) Write the study state to disk here, so the subject can't refresh and enter different values depending on his assigned condition.
  • If the subject is in the control condition, skip to the end. Otherwise, continue.
  • Solicit criteria and alternatives, as in the pilot.
  • Conduct VDA.
    • Display a reminder of the subject's chosen criteria and alternatives on this page. Explain that the subject doesn't have to make the choice suggested by Artiruno, but it could be a good idea.
    • Show results as text. (The graph would probably not be very helpful in this situation, with find_best = 1 and non-quantitatively minded users.)
    • Allow the subject to restart the procedure, or to return to the previous page and edit criteria and alternatives; trying to record everything is probably futile by the nature of JavaScript.
    • Don't provide an abort button. (It wouldn't be useful because in the case of find_best = 1 and not showing a graph, Artiruno can provide no useful information before it's done.)
    • Record the questions asked, the subject's choices, the subject's response times, and Artiruno's conclusion. If the subject restarted VDA, use the values from the final round.
  • Ask for any comments.
• Session 2 (internally, visit 3)
  • Send an invitation 1 month after the subject completed session 1. Say that session 2 assumes the subject made the choice and got to see at least a little of the outcome; if that won't be true for a while, the subject should reply saying when would be a good time to do session 2.
  • [Offer how much money?]
  • Re-display the decision description and expected resolution date.
  • Ask if they've made the choice and gotten to see some of the outcome. (In theory, this should always be "yes", because of the instructions earlier. It's a sanity check.)
  • Have the subject briefly describe in prose:
    • which choice they made
    • what the outcome was
    • how happy they are with the choice and outcome
  • Use 1-to-5 rating scales for the below.
  • How pleased are they with the outcome of their choice?
  • How well-chosen was their choice, given what they knew at the time they made it?
  • How difficult did it feel to make the decision?
  • In the VDA condition:
    • Show the criteria, alternatives, and results text from VDA.
    • How consistent did they feel their choice was with Artiruno's suggestion?
    • How difficult was the procedure (including writing up the criteria and alternatives) to do?
    • How helpful did the procedure feel for making the decision-making process?
  • Debrief
  • Another comments box

On Prolific.

In TV 11, I restored the random assignment of conditions, and I slightly increased the pay and the time estimate for the scenario phase. In TV 12, I fixed a typo.

I wanted to run subjects until I had at least 40 in each condition who completed the scenario phase, so I ceased recruiting new subjects once that criterion was satisfied after my last batch. I picked this number because I wanted at least 20 subjects to have completed all phases, and I estimated a 50% follow-up rate.

(comments-dt (ssi subjects (.isin $tv [11 12])))

(wc
  (ss subjects
    (& (.isin $tv [11 12]) (pd.notnull $n_puzzle_attempts)))
  (cbind
    $began
    :res_len (. (- $expected_resolution_date $began) dt days)
    :puz_t (.round $time_puzzle_minutes 1)
    :puz_n $n_puzzle_attempts
    $cond
    :v2 (.round $time_visit2_minutes 1)
    :cy $country
    :edu $education_years))

(.sort-index (.value-counts (wc
  (ss subjects (& (.isin $tv [11 12]) (pd.notnull $time_puzzle_minutes)))
  (cbind $cond :did_v2 (pd.notnull $time_visit2_minutes)))))

Let's send out the first invitations on April 3rd, which is 4 weeks after I ran the last subject in the analytic sample.

I need to ask each subject if they've made the decision and gotten to see some outcome before reinviting them. Let's do it by sending them a message on Prolific.

Send out only one message first. Then you can send them out in waves of 10 or 20 in subject-number order, ideally in the morning so you don't get questions while you're asleep.

• If subjects refresh the page and redo parts of the task, the timing data you get will only reflect the final attempt.

Ashikhmin, I., & Furems, E. (2005). UniComBOS—Intelligent decision support system for multi-criteria comparison and choice. Journal of Multi-Criteria Decision Analysis, 13(2-3), 147–157. doi:10.1002/mcda.380

Barbosa, P. A. M., Pinheiro, P. R., Silveira, F. R. V., & Filho, M. S. (2019). Selection and prioritization of software requirements applying verbal decision analysis. Complexity. doi:10.1155/2019/2306213

Edwards, W. (1977). How to use multiattribute utility measurement for social decisionmaking. IEEE Transactions on Systems, Man, and Cybernetics, 7(5), 326–340. doi:10.1109/TSMC.1977.4309720

Hodgett, R. E., Martin, E. B., Montague, G., & Talford, M. (2014). Handling uncertain decisions in whole process design. Production Planning and Control, 25(12), 1028–1038. doi:10.1080/09537287.2013.798706

Larichev, O. I. (2001). Ranking multicriteria alternatives: The method ZAPROS III. European Journal of Operational Research, 131(3), 550–558. doi:10.1016/S0377-2217(00)00096-5

Moshkovich, H. M., & Mechitov, A. I. (2018). Selection of a faculty member in academia: A case for verbal decision analysis. International Journal of Business and Systems Research, 12(3), 343–363. doi:10.1504/IJBSR.2018.10011350

Moshkovich, H., Mechitov, A., & Olson, D. (2016). Verbal decision analysis. In S. Greco, M. Ehrgott, & J. R. Figueira (Eds.), Multiple criteria decision analysis (2nd ed., pp. 605–636). New York, NY: Springer. ISBN 978-0-387-23081-8. doi:10.1007/978-1-4939-3094-4_15

Shevchenko, G., Ustinovichius, L., & Walasek, D. (2019). The evaluation of the contractor's risk in implementing the investment projects in construction by using the verbal analysis methods. Sustainability, 11(9). doi:10.3390/su11092660

Tamanini, I., & Pinheiro, P. R. (2011). Reducing incomparability in multicriteria decision analysis: An extension of the ZAPROS method. Pesquisa Operacional, 31, 251–270. doi:10.1590/S0101-74382011000200004