Belief is value-dependent
Created 10 Apr 2015 • Last modified 8 Dec 2015
Even formal methods with optimality guarantees, particularly statistics, are not enough to completely determine how observations should affect our beliefs. We also need certain assumptions about how to value beliefs, such as what mistakes we'd rather make. This state of affairs is bad for empiricism and science, but not fatal.
So I was neck-deep in proper scoring rules and other statistical arcana today when I read a comment on CrossValidated that argues "ideally, we always distinguish fitting a model from making a decision" and that only for computational reasons would we consider loss functions while choosing or estimating models, and that doing such necessary evils "is where subjectivity in model selection creeps in".
This is, perhaps subtly, wrong. The thing is—and I find myself making points analogous to this one often—statistics, economics, or any other sort of math can't tell you how to form your beliefs, any more easily than they can tell you what to do. For example, in the case of estimating the parameters of the model, there's the method of maximum likelihood, the method of moments, the method of least squares, Bayesian methods, penalized methods, and robust methods. Statistics can tell you about the properties of these methods to help you pick one, but not which one to pick in the first place. You could try using some kind of meta-procedure for choosing a method (e.g. choose the minimax estimator), but then you're stuck with the problem of choosing a meta-procedure (e.g., which loss function should you use?). At some point you need to just need to make an assumption, or declare an axiom, or make a leap of faith, or whatever you want to call it. The same sort of argument goes for the problem of model selection, as well as the problem of making a decision given estimates of the consequences of each option.
I don't think this realization undoes empiricism, that is, the idea that there is a single objective reality and that our beliefs are right to the degree that they approximate this reality. But this realization does show that reasonable people can disagree given the same evidence, so even though we all have the same epistemic destination, we may get there different ways, and we can't always tell in advance which way will be quickest. Perhaps another way to put this is that empiricism alone is not nearly enough epistemology to completely determine one's beliefs.