Donkey notebook

I want a bare-bones trivial decision-making task where people have a repeated choice between two meaningless and maximally equivalent options (a little like Buridan's ass, which, curiously, never seems to have gotten a psychology experiment in humans). By studying the prediction of decisions in this task, I hope to learn more about how to predict variation in choices that arises from something other than content. Then perhaps I can e.g. break the ceiling of 85% accuracy I observed in Arfer and Luhmann (2015).

It seems like there's a lot of analytic approaches for predicting binary sequences. Let's get at least a little data to start with. Let's run the task on Mechanical Turk for convenience.

I was thinking the task could use the mouse, and I could record mouse movements and use mouse movements before making previous choices as a rich predictor. The trouble is that you can't move the cursor in JavaScript, so I can't force the cursor to a neutral position after each choice. Even if I could, the physical mouse would still be a bit to the side, of course. Asking subjects to perfectly recenter the mouse, and not starting the next trial till they did, would probably drive subjects a little crazy.

Instead, let's use the keyboard, and ask subjects to position their fingers over both of the two allowed keys. "F" and "J" seem like good choices.

Suppose we get 200 choices. For each subject, the first 100 choices will be purely training data, while I'll evaluate predictions on the second 100. I may use online learning methods that effectively treat every trial the same, but I'll be most interested in performance after the method has already had 100 trials to train with. During the task, I'll show the subject how many choices are left to make, but I won't give feedback on which choices they made.

Since the base rates are near 50%, a trivial model will do poorly.

(setv N 200)
(setv n-train (// N 2))
(setv n-test (- N n-train))
(setv d (.copy trials))
(setv ($ d train) (< ($ d trial) n-train))

The first few subjects look like this (O means pressing "f", x means pressing "j"):

(lfor [X x] (.groupby (ss d (.isin $sn [0 1 2])) "sn")
  [(pretty-choice-seq (ss x $train "chose-key1"))])

The quantiles of base rates in the test halves are:

(setx qs-base (.quantile :q [.25 .5 .75] (wcby
  (ss d (bnot $train)) $sn
  (max [(.sum $chose-key1) (.sum (bnot $chose-key1))]))))

When you think about it, it's not surprising that autoregressive-like models, that is, models where the output depends on the previous output, would do badly in this situation. I'm trying to predict trials 100 through 199, so if I mispredict trial 110, that will substantially hurt my ability to predict trial 111, and hence trial 112, and so on. One mistake is enough to throw me off the whole scent.

It's also not surprising that regression doesn't work well, considering the dangers of extrapolation. Each model, in training, only sees trial numbers 0 through 99. It gets no instructions or feedback to control its behavior when the input trial number is 100 or more, which is also the only kind of trial number tried in testing. So, for example, a tree-based model might use a rule like `trial > 97` just to refine its predictions for trials 98 and 99, but that rule will end up applying to all the testing data. I should only expect a hierarchical-style regression model, which is allowed to see all trial numbers for some subjects, to succeed on these later trial numbers.

To make things easier, and also more like Builder, let's try traditional random cross-validation, within subjects. If I do well here, I can use my winning model to inspire the construction of hierarchical models for the original problem.

If only for computational reasons, let's try predicting one test trial at a time.

For the code in this section, tuning parameters have been hand-adjusted, so they may be overfit.

(setv n-folds 10)

(defmacro try-model [#* args]
  (setv args (dict (partition args)))
  (setv
    estimator-expr (.get args ':model)
    fit-fn (.get args ':fit)
    diagnostic-expr (.get args ':diagnostic)
    prob-for-pred (hy.eval (.get args ':prob-for-pred 'False))
    use-rts (.get args ':rt 'False))
  `((fn []

    (setv train-trials (list (range 100)))
    (setv d-choices (.pivot trials
      :index "sn" :columns "trial" :values "chose-key1"))
    (setv X (.to-numpy (geti d-choices : train-trials)))
    (when ~use-rts
      ; Add the standardized reaction times to X.
      (setv rts (.to-numpy (geti
        (.pivot trials :index "sn" :columns "trial" :values "rt")
        : train-trials)))
      (setv X (np.hstack [X (/
        (- rts (np.mean rts :axis 0))
        (* 2 (np.std rts :axis 0)))])))
    ;(setv X (.fit-transform :X X (sklearn.preprocessing.PolynomialFeatures
    ;  :degree 2 :interaction-only T :include-bias F)))

    (setv out (pd.concat :axis 1 (lfor test-trial [100 101 150] (do

      (setv y (.to-numpy (geti d-choices : test-trial)))
      (setv y-prob (np.full y.shape NaN))
      (setv y-pred (np.zeros-like y))
      (setv diagnostic [])

      (for [[fold-i [train test]] (enumerate (.split :X X
          (sklearn.model-selection.KFold n-folds
            :shuffle T :random-state 1234)))]
        (print (.format "Test trial {}, fold {}" test-trial fold-i))
        (np.random.seed (+ 123 fold-i))
        (setv model ~(if fit-fn
          `(~fit-fn (geta X train) (geta y train) fold-i)
          `(.fit ~estimator-expr (geta X train) (geta y train))))
        ~(when diagnostic-expr
          `(.append diagnostic ~diagnostic-expr))
        (setv probs (geta (.predict-proba model X) : -1))
        (setv (geta y-prob test) (geta probs test))
        (setv (geta y-pred test) ~(if prob-for-pred
          '(>= (geta y-prob test) .5)
          '(.predict model (geta X test))))
        (print (.format "Training accuracy: {:.02}"
          (np.mean (= (> (geta probs train) .5) (geta y train))))))

      (pd.Series :name test-trial (dict
        :base-MSE (round (.mean (** (- y (np.mean y)) 2)) 3)
        :MSE (round (.mean (** (- y y-prob) 2)) 3)
        :base-rate (max (.mean y) (- 1 (.mean y)))
        :accuracy (.mean (= y y-pred))))))))

    (setv out.index.name "test_trial")
    out)))

(setx result-between-lr (try-model
  :model (sklearn.linear-model.LogisticRegression
    :penalty "l1" :solver "saga" :max-iter 2,000
    :C (get {100 0.25  101 0.5  150 1e-6} test-trial))
  :diagnostic (+ (list model.intercept_) (list (.flatten model.coef_)))))

(setx result-between-lr-rt (try-model
  :rt True
  :model (sklearn.linear-model.LogisticRegression
    :penalty "l1" :solver "saga" :max-iter 2,000
    :C (get {100 0.25  101 0.75  150 1e-6} test-trial))
  :diagnostic (+ (list model.intercept_) (list (.flatten model.coef_)))))

(setx result-between-nn (try-model
  :model (sklearn.neighbors.KNeighborsClassifier :n-neighbors 3)))

(setx result-between-nn-rt (try-model
  :rt True
  :model (sklearn.neighbors.KNeighborsClassifier :n-neighbors 3)))

(defmacro try-neural-net [#* kwargs]
  (setv kwargs (dict (partition kwargs)))
  `(try-model
    :rt ~(.get kwargs ':rt)
    :fit (fn [X y fold-i]
      (import os) (setv (get os.environ "TF_CPP_MIN_LOG_LEVEL") "3")
      (import
        tensorflow
        tensorflow.keras.models [Sequential]
        tensorflow.keras.layers [Dense Activation]
        tensorflow.keras [regularizers])
      (setv model (Sequential [
        (Dense ~(get kwargs :hidden-nodes) :activation "tanh"
          :kernel-regularizer (regularizers.l1 ~(get kwargs :penalty)))
        (Dense 1 :activation "sigmoid")]))
      (.compile model
        :optimizer "rmsprop"
        :loss "binary_crossentropy")
      (tensorflow.random.set-seed (+ 456 fold-i))
      (.fit model X y
        :verbose False
        :epochs ~(get kwargs :epochs)
        :batch-size 20)
      model)
    :prob-for-pred True))

(setx result-between-neural (try-neural-net
  :hidden-nodes 100 :epochs 100
  :penalty (get {100 .02  101 .01  150 1e6} test-trial)))

(setx result-between-neural-rt (try-neural-net
  :hidden-nodes 200 :epochs 100 :rt True
  :penalty (get {100 .02  101 .02  150 1e6} test-trial)))

(defclass OneColPredictor []
  (defn __init__ [self #** kw]
    (for [[k v] (.items kw)]
      (setattr self k v)))
  (defn predict-proba [self X]
    (np.reshape
      (np.where (geta X : self.column) self.p-true self.p-false)
      (, (get X.shape 0) 1))))

(setx result-between-onecol (try-model
  :fit (fn [X y fold-i]
    (setv column (max (range (get X.shape 1)) :key (λ
      (setv p (np.mean (= (geta X : it) y)))
      (max p (- 1 p)))))
    (OneColPredictor :column column
      :p-true (np.mean (get y (geta X : column)))
      :p-false (np.mean (get y (bnot (geta X : column))))))
  :prob-for-pred True))

Even in this, the easiest form of the problem, with or without reaction times, I'm not getting great accuracy. I also modified the above code to change the training set from the first 100 trials to trial 80 through 100, and that improved performance only a bit.

The sample is not huge, especially when framing the problem hierarchically as predicting one test trail with all 100 training trials. However, considering the gap between the kinds of performance I'm getting (77% accuracy in trial 101; 55% accuracy in trial 150) and the kind I'm looking for (at least 95%), it's not realistic to think that a sample that's twice or ten times this size will suffice. An enormous sample, like over 10,000 subjects, could change things, but that's not something I can readily afford.

I think I made a mistake by not explicitly asking subjects not to use a randomization device. There's no telling how many of them did that. However, I can reasonably assume it was a minority, and that keeping subjects from doing this wouldn't suffice for good accuracy.

Is it true that if two people make the same binary choice on a large number of contiguous trials, then they're highly likely to make the same choice on the trial immediately after?

(require kodhy.macros [geti λ cbind])
(import kodhy.util [rd])

(.apply
  (.groupby (pd.DataFrame retail-order-seqs) [0 1 2 3 4])
  (λ (cbind :n (len it)  :p (rd (.mean (geti it : 5))))))

Maybe not—but maybe 5 choices isn't long enough of a sequence, and my notion of a binary choice here is pretty artificial: it's whether an order in the University of California, Irvine's "Online Retail" dataset included 10 or more distinct items. But most problematic, I guess, is that this idea of prefixes being predictive might work when all the decision problems being considered are the same for all trials in the prefix, as well as the $n$th trial, but not when different subjects are considering different problems—the problem characteristics are too influential on choices for these prefixes to predict the response with high accuracy.

Probably, people's decisions really are noisy. Whether the noise is deterministic (via sensitive dependence on initial conditions) or truly random, people's choices can include variation that is effectively unpredictable from previous choices of the same kind. One can imagine a continuum of predictability to decision problems with no features or constraints, as in this task, to problems where the subject is basically forced to make a certain choice. In between are the typical problems of psychology experiments and real life, where people have some amount of freedom to choose, but the features of the options, such as forseeable consequences, tend to push them towards one option or another.

One could reanalyze the Donkey data with explicit statistical models, or at least probabilistic classification. If people's choices are too inherently variable (due to a random, pseudorandom, or chaotic component) for me to predict individual choices accurately, perhaps I can still characterize their choices accurately as a statistical model. But even then, I feel like I wouldn't get a publishable paper out of this, and there would be no clear way to build further science on my findings. My time is probably better spent elsewhere.

Arfer, K. B., & Luhmann, C. C. (2015). The predictive accuracy of intertemporal-choice models. British Journal of Mathematical and Statistical Psychology, 68(2), 326–341. doi:10.1111/bmsp.12049. Retrieved from http://arfer.net/projects/builder/paper

Hagelbarger, D. W. (1956). SEER, a SEquence Extrapolating Robot. IRE Transactions on Electronic Computers, EC-5(1), 1–7. doi:10.1109/TEC.1956.5219783

Shannon, C. E. (1953). A mind-reading (?) machine. Bell Laboratories.