Artiruno: A free-software tool for multi-criteria decision-making with verbal decision analysis

Most methods for multi-criteria decision analysis (MCDA) require the agent to make numerical judgments. For example, in methods based on multiattribute utility or approximations thereof, such as simple multiattribute rating theory (SMART; Edwards, 1977), agents rate each criterion's importance with integers increasing from 10, and then they quantify the level of each option on each criterion using physical units or an abstract 0-to-100 scale. Or in the Analytic Hierarchy Process (AHP), agents rate each pair of alternatives on each criterion, as well as each pair of criteria, on 1-to-9 scales (Saaty & Vargas, 2012). The numbers are related to each other either with functions chosen by the agent or by standard functions, such as, in the case of SMART, linear preference from the lowest plausible value to the highest plausible value. Numerical judgments and functions to operate on them are a great help to the development of methods, which can then proceed with quantitative operations. The trouble is that the outcome can of course be sensitive to agents' numerical judgments, which can be imprecise or inaccurate, particularly when rather abstract, such as importance ratings.

Outranking methods represent an alternative or supplement to more utility-oriented MCDA methods. In outranking, the goal is to find ordinal relations among options. Ordinal relations, as opposed to cardinal relations, suffice for many real-world decisions (such as hiring from a pool of applicants to fill a limited number of jobs) while potentially demanding less rich quantitative judgments from agents. A drawback is that the method may deem some options incomparable, making the result only a partial order. One example of an outranking method is Visual PROMETHEE (Mareschal, 2013), which allows criteria to be continuous or ordinal, and produces ordinal results about the alternatives. While less numerical, outranking is not necessarily a complete escape from numerical judgments. In the case of Visual PROMETHEE, the criteria must be weighted numerically even if all of them are ordinal.

Verbal decision analysis (VDA; Moshkovich, Mechitov, & Olson, 2016) was introduced by Larichev and Moshkovich (1997) as an approach to outranking that requires no numerical judgments at all. This approach was inspired by research in experimental psychology about what kinds of cognitive operations people can perform most reliably (Larichev, Polyakov, & Nikiforov, 1987). With a VDA method, such as ZAPROS (Larichev & Moshkovich, 1995; Larichev, 2001), the agent defines all criteria ordinally and then makes judgments about the relative appeal of hypothetical alternatives. From these judgments and a few conservative assumptions about the ordinal structure of preferences, VDA can infer an order for the real alternatives. The key idea is to present hypothetical decisions to the agent that are easier to make than the real decision, because fewer criteria are allowed to vary, yet are informative about the comparisons involved in the real decision. VDA is a natural fit for complex real-world decisions in which criteria are difficult to measure and weight, such as a hiring decision that must take into account subjective judgments of candidates' qualities. Further developments of VDA methods and software since the earliest iterations of ZAPROS include UniComBOS (Ashikhmin & Furems, 2005), ORCON_Z (Moshkovich & Mechitov, 2018), and ORCLASSWEB (Barbosa, Pinheiro, Silveira, & Filho, 2019). DEX (Bohanec & Rajkovič, 1990) is an example of a method that, while not construed as a form of VDA, uses VDA-like qualitative criterion scales.

A limitation of the extant VDA methods is that the questions they ask the agent in order to come to their recommendations are insensitive to the specific decision that needs to be made. In ZAPROS, for example, the agent must make a judgment about every pair of criterion levels, whether or not that judgment is necessary for the conclusion. This exhaustiveness is an inefficient use of the agent's attention, possibly reducing accuracy or user acceptability, and makes large numbers of criteria or levels impractical. If the method allows comparing more than two criteria levels at once, in order to alleviate the incomparability issues that outranking methods are prone to, then the agent needs to make even more judgments. Furthermore, this practice of judging criterion levels in isolation, outside the context of fully defined hypothetical alternatives, may undermine VDA's goal of focusing on psychologically realistic judgments.

Another potential problem with VDA, as with many MCDA methods, is a lack of direct empirical investigation of how effective the methods are in improving decisions. The theory is rich, as are examples of real-world use, but one may still ask whether using VDA results in better decisions than not using it—even in a weak sense of decision quality that only measures how satisfied people are with their decisions.

In the present article, I describe a new VDA method that I call "Artiruno". Artiruno's most important innovation in VDA methdology is its tactical selection of questions. The agent can declare that all he requires is to find the n best alternatives (typically the single best), and Artiruno will make inferences mid-interview to ask the questions most relevant to the goal. Even if the agent declares a goal of comparability between all alternatives, Artiruno can still benefit from online inference. Questions whose answers are implied by previous answers are omitted entirely, making it impossible for the agent to provide inconsistent answers. Intelligent selection of questions lowers the decision-making burden on the agent and makes large item spaces feasible: a problem with 10 criteria that each have 10 levels need not require asking billions of questions, or holding in memory a data structure with billions of items. Such selectiveness is especially helpful because Artiruno is multilevel like UniComBOS, in the sense that it can vary two or more criteria at once in its questions it asks of the agent.

To check that intended theoretical guarantees are met and efficiency goals are maintained, Artiruno includes a test suite. Automated tests ensure that simple features of Artiruno (e.g., trivial ordinal inferences) as well as complex features (e.g., inference of preferences defined by a randomly generated additive utility function) are designed and implemented correctly.

To examine how helpful Artiruno is for real-life decision-making, I subject it to an empirical test that MCDA methods (especially VDA) are rarely given. Namely, I compare decision outcomes and user satisfaction when decision-makers—non-experts, in fact—were randomly assigned to use Artiruno or to receive no special help. Randomized controlled trials such as this are generally seen as essential for evaluating medical and psychological interventions because they allow for straightforward causal inferences (Sibbald & Roland, 1998), but they appear underused in research on MCDA and more generally decision aid. The subjects of this empirical study considered decision scenarios they faced in their own lives, leading Artiruno to be tested in a variety of domains.

Like all VDA methods, Artiruno represents a decision scenario using finitely many criteria and items. The criteria, representing dimensions on which an item may be judged, are finite tuples of levels or opaque values. In a decision between job offers, for example, a criterion might be "Salary", with the levels "Low", "Medium", and "High". Levels are ordered such that each level is strictly better than (i.e., preferred to) the one before it, so a job with "Medium" salary is better than a job with "Low" salary, all other things being equal. Essentially, this is an assumption of preferential independence. On the other hand, no order is assumed between criteria. The items, representing imaginable options, are the Cartesian product of all the criteria, while a subset of the items, the alternatives, represent the specific options that the agent can decide between. Thus, each alternative has exactly one value for each criterion, which is a level of that criterion, and the agent's preferences regarding an item are assumed to be completely determined by its values for the criteria. The job-offers scenario might look like this:

• Criteria
  • (A) Salary: "Low", "Medium", "High"
  • (B) Location: "Far from home", "Ideal"
  • (C) Training opportunities: "Low", "High"
  • (D) Promotion opportunities: "Low", "High"
• Alternatives
  • (a) Dewey, Cheatem & Howe
    • Salary: (A₃) "High"
    • Location: (B₂) "Ideal"
    • Training opportunities: (C₁) "Low"
    • Promotion opportunities: (D₁) "Low"
  • (b) MicroCo.
    • Salary: (A₂) "Medium"
    • Location: (B₁) "Far from home"
    • Training opportunities: (C₂) "High"
    • Promotion opportunities: (D₂) "High"

The agent's preferences among the items are represented as a preorder, which is defined as a reflexive and transitive binary relation. The relation a ≤ b for any items a and b means that a is no better than b; that is, a is worse than or equally as good as b. Notice that a preorder allows the agent to be indifferent between two items, denoted a ≈ b, meaning a ≤ b and a ≥ b even though a need not equal b. Preorders also allow a pair of items to be incomparable. However, incomparability represents Artiruno's ignorance of the agent's preference regarding a pair of items. It is assumed that the agent's true preferences are a total preorder, and Artiruno asks the agent questions towards the goal of replacing incomparability with one of the three relations a < b, a > b, or a ≈ b.

To extend the inferences Artiruno can make, one more assumption is included: the rule of segmentwise dominance. Intuitively, the rule of segmentwise dominance says that if the comparison of two items can be broken down into several comparisons on subsets of the criteria ("segments"), all those comparisons agree, and all the segments together cover all the criteria, then we can infer a relation between the original two items.

• In the simple case, the rule of dominance states that for any items a = (a_A, a_B, a_C, …) and b = (b_A, b_B, b_C, …), if a_i ≤ b_i for all i and a_i < b_i for some i, then a < b. We say that b dominates a, or if an item dominates all other items in a set, that it is a dominator of that set.
• The rule of segmentwise dominance generalizes dominance as follows. For each item p and each subset σ of the criteria, let R(σ, p) be the item that has the best value on all criteria except possibly the criteria in σ, for which it has the criterion values of p. In other words, R(σ, p) can only deviate from the best constructible item on σ, and the values for σ are taken from p. Then let a and b be distinct items, and let {α₁, α₂, …, α_m} and {β₁, β₂, …, β_m} be partitions of the criteria (which need not be distinct, and notice that both partitions have the same number of sets, m, although the cardinalities of those sets may vary). Segmentwise dominance applies when R(α_i, a) ≤ R(β_i, b) for all i. If R(α_i, a) < R(β_i, b) for some i, then a < b; otherwise, a ≈ b.

For an example of segmentwise dominance, consider the job-offers scenario laid out above. Let a be Dewey, Cheatem & Howe and b be MicroCo. Representing the criteria as {A, B, C, D} and their levels as {A₁, A₂, A₃, B₁, B₂, C₁, C₂, D₁, D₂}, let α₁ = {A, B, D} and α₂ = {C} while β₁ = {A, C, D} and β₂ = {B}. Suppose we know from previous responses the agent has given that R(α₁, a) = (A₃, B₂, C₂, D₁) < (A₂, B₂, C₂, D₂) = R(β₁, b), while R(α₂, a) = (A₃, B₂, C₁, D₂) < (A₃, B₁, C₂, D₂) = R(β₂, b). Then segmentwise dominance applies, and we conclude that a < b.

Artiruno is intended to be able to find the single best item in a set of alternatives, or more generally the best n. For this purpose, we define the top-n subset of a preordered set as a generalization of the notion of a maximum, distinct from the notion of a maximal element. Given a preordered set X, the top-n subset is the set of all elements x ∈ X such that

• x is comparable to every element of X, and
• there are at most n − 1 distinct elements a ∈ X with a > x.

In the case of a total order, the top-1 subset is simply the singleton of the maximum of S, and the top-n subset has cardinality n so long as n ≤ |S|. For preorders, however, the complication of indifference means that the top-n subset can be of any cardinality {0, 1, …, |S|}.

I distinguish Artiruno from previous VDA software by making it permanently available (on GitHub at https://github.com/Kodiologist/Artiruno, or on the Internet Archive at http://web.archive.org/web/20230924/https://codeload.github.com/Kodiologist/Artiruno/tar.gz/refs/tags/v0.4.2) under a free license, the GNU GPL version 3 or later. Artiruno is free software in the sense of the Free Software Foundation (2008): it may be freely copied, studied, and modified without technical or legal restrictions, contributing to the larger goal of open science (Woelfle, Olliaro, & Todd, 2011). Artiruno is written in Python, a popular programming language in science (Millman & Aivazis, 2011) described as a "lingua franca" for scientific computing (Thomas & Christensen, 2014) and computer-science education (Hunt, 2014). As a concise, readable, and easy-to-learn language, Python is well suited to this role. Other Python libraries for MCDA include pymcdm (Kizielewicz, Shekhovtsov, & Sałabun, 2023) and AHPy.

An additional strength of Artiruno's implementation is its web interface. At http://arfer.net/projects/artiruno/webi, anyone can quickly try out Artiruno (on a preprovided example scenario, or one of their own design) using only a web browser.

This section describes the operation of Artiruno in overview, while referring to particular modules and objects in the code where full details may be found.

One core component of Artiruno is PreorderedSet, a generic class of data structure for representing a set equipped with a preorder. It can contain many types of objects, such as numbers or strings; for VDA, it contains tuples to represent items that the agent may consider. Each pair of elements in the set has one of four corresponding relations: strictly less than (LT), strictly greater than (GT), indifference (EQ), or incomparability (IC). The class supports two important mutating operations. The method add puts a new object in the set, making it initially incomparable to all other objects. The method learn takes two objects already in the set and a relation, and updates the preorder with a modification of Warshall's algorithm of transitive closure (Ingerman, 1962) to reflect this new fact, adding all further relations that can be inferred by transitivity. If the new fact leads to a contradiction, the exception ContradictionError is raised; thus, in the absence of exceptions, the user is assured of a consistent preorder. Finally, PreorderedSet can generate a graph of all relations that can be displayed with Graphviz (Gansner & North, 2000) for easy visualization of complex relational networks (see e.g. Figure 2).

The key function for VDA is vda, which takes lists of criteria and alternatives as arguments. It begins by checking that these are well-formed (e.g., that each criterion has at least one level, with no two levels equal) and initializing the agent's preferences (a PreorderedSet) with the alternatives and the inferences that can be made between them by assumption (e.g., dominance). Another parameter to vda is find_best, which specifies the goal of VDA. If find_best is an integer n, Artiruno will only try to find the top-n subset, whereas if find_best is not provided, Artiruno will try to construct a total preorder among all the alternatives.

In its main loop, vda searches for two items to compare. The set of candidate item pairs to_try is initialized with all pairs of alternatives. Pairs whose relations are already known (e.g., by dominance, or from questions the user was previously asked) are removed, and when find_best is provided, item pairs in which neither item could be in the top-n subset are also removed. The remaining pairs are considered one by one, in a lexicographic order that selects more-preferred criterion values (according to how the user defined the criteria originally) first. Within this loop, sets of criteria are themselves looped over (in the same order the criteria were defined), choosing a criterion (or set of criteria) to vary for the first item of the pair and another criterion (or set of criteria) to vary for the second item. Artificial items that vary from the best possible items on the chosen criteria are constructed in the fashion of R(σ, p), towards the application of segmentwise dominance, as described above. If the relation between these items is unknown, the agent is asked to judge them. The agent may answer that the first item is better, the second is better, or that the two items are equally preferable (Figure 1). Artiruno then updates its PreorderedSet with the response. Given some patterns of responses, it can apply the rule of segmentwise dominance to make further updates.

g/question_interface

The multilevel aspect of Artiruno is realized in how it selects the sets of varying criteria for the questions it asks the agent. Initially, Artiruno varies only two criteria: one for the first item, and one for the second. If all applicable questions have been asked and the goal is still unreached, Artiruno increases the threshold to three, allowing two criteria to vary for one item and one criterion to vary for another. This process continues up to double the number of criteria, or to a limit set by the user.

The return value of vda is the PreorderedSet representing the agent's preferences in its final state. This preorder can be characterized with a graph, or in the case of find_best = 1, a sentence such as "Your choices imply a single best alternative: MicroCo.".

It may be instructive to compare Artiruno to some other VDA methods in detail.

ORCON_Z (Moshkovich & Mechitov, 2018),1 a version of ZAPROS, is implemented as a Microsoft Excel spreadsheet with macros. It limits the number of criteria to 6, and the number of levels to 5 per criterion. It requires the user to express a preference for every pair of criterion levels, rather than skipping questions deducible from previous questions and stopping as soon as the decision-making goal is achievable. The user decides between levels of different criteria in isolation, rather than deciding between complete hypothetical items that differ on the chosen criterion levels as in Artiruno. When the user expresses inconsistent preferences, ORCON_Z points out the inconsistency and allows preferences to be corrected. (Artiruno, by not asking questions whose answers it can deduce, gives the user no opportunity to express inconsistent preferences in the first place.) ORCON_Z is not multilevel: only two criteria are allowed to vary at a time in the questions asked of subjects. ORCON_Z lacks automated tests.

Comparison with UniComBOS (Ashikhmin & Furems, 2005) is more difficult, since neither a runnable copy of the program nor its source code are available.2 Ashikhmin and Furems (2005) describe UniComBOS as a web browser-based Java applet paired with a server-side Java program. Like ORCON_Z, UniComBOS has the user decide between isolated sets of criterion levels rather than items, but is multilevel in a very similar way to Artiruno (which is no accident, since Artiruno's multilevel support was inspired by UniComBOS). User replies to multilevel questions are used to make inferences about items using a principle of "U-dominance" that serves as an equivalent to segmentwise dominance for partially defined items. UniComBOS's strategy for selecting questions and stopping the interview is unclear, but it appears that for each level of the multilevel procedure, the set of questions asked is exhaustive rather than strategic. Like ORCON_Z, UniComBOS notifies the user of inconsistencies and allows them to be corrected; in fact, it deliberately asks logically redundant questions as consistency checks. Unlike both ORCON_Z and Artiruno, UniComBOS allows the user to answer "I don't know" to a question. This is understood as an assertion of incomparability, so if a different preference could later be inferred for the relation in question, UniComBOS would consider that inconsistent. Artiruno, by contrast, treats incomparability as incomplete information, and works on the assumption that the user's true preferences are a total preorder. UniComBOS lacks automated tests.

DEX (Bohanec & Rajkovič, 1990; Bohanec, 2023; https://repo.ijs.si/markobohanec/dexilibrary) is an MCDA method that primarily uses totally ordered qualitative criteria, and requires no numerical judgments; Bohanec draws comparison to VDA without describing DEX as a form of VDA. DEXi, the latest implementation, is free software, but is written in Oxygene, a proprietary dialect of Pascal. In DEX, criteria are arranged in a hierarchy not unlike that of the AHP, where the root element of the hierarchy represents the overall judgment to be made, such as the quality of a car. Each non-leaf criterion is determined by its child criteria according to a function mapping each tuple of the child levels to a parent level. The user can define the entire function by hand, or use dominance or numerical weights to infer some parts of the function from others (manually entered dominance violations are called out, but allowed). So, as with ORCON_Z and UniComBOS, DEX requires full definition of the decision procedure before consideration of items, rather than providing an adaptive interview like Artiruno, and the user generally makes judgments on the basis of partially defined items. But DEX includes extensions for probabilistic or fuzzy criteria, and it has plenty of automated tests. It also has R and Python interfaces in development as of 2022.

Artiruno's automated test suite is divided into two parts: one for PreorderedSet alone and one for VDA.

The PreorderedSet tests check that adding consistent relations makes the necessary inferences by transitivity, while adding inconsistent relations raises an exception. They also check example uses of methods such as extreme, used to get top-n or bottom-n subsets, and get_subset, used to make a new PreorderedSet that contains a subset of the original's items while preserving all relations among them.

The VDA tests check various trivial scenarios (where no questions need to asked of the agent, because the goal is attainable by inferences that follow immediately from assumptions) as well as simple scenarios with manually programmed responses from the agent. They also check that VDA can complete successfully on item spaces that would be too large to hold in memory if fully instantiated. The broadest tests examine whether Artiruno can learn randomly generated preferences, so long as a simulated agent makes decisions in accordance with them. Two classes of randomly generated preferences are tested: lexicographic preferences, in which the criteria have a fixed order such that values on a higher-ranked criterion are always more important than values on all lower-ranked criteria, and additive utility functions, in which each criterion level adds a certain positive amount of utility. The random generation of preferences is seeded, and the total number of questions asked to reach the VDA goal (also randomized) is tracked, so any changes to Artiruno that result in more questions needing to be asked for the same test case can be noticed.

For an example of how Artiruno would operate when used with a larger problem, consider the faculty-selection scenario of Moshkovich and Mechitov (2018). We use four criteria (A, B, C, D) with three levels each, and we set find_best = 2, as if deciding on the two best candidates to consider for the next round of faculty selection. Table 3 of Moshkovich and Mechitov (2018) lists 32 alternatives, but several are equal to each other on all criteria, which is forbidden by Artiruno, since VDA offers no way to distinguish them. Thus, we begin by collapsing identical candidates into single alternatives (e.g., candidates 6 and 10 become the alternative "Candidate 6,10"), leaving 16 alternatives. This problem setup is distributed with Artiruno's source code in the file examples/faculty.json.

Artiruno initializes the set to_try with all (16 choose 2) = 120 pairs of alternatives. This set is immediately reduced to 82 pairs by removing pairs for which the preference relation can be inferred by dominance: for example, candidate 4 (A₃, B₃, C₃, D₂) dominates candidate 22 (A₃, B₃, C₂, D₂), being non-inferior on all criteria and superior on C. We reduce to_try further to 23 pairs by removing pairs containing an alternative known to be worse (by dominance) than 2 or more other alternatives—such an item must not lie in the top-2 subset.

Now we can select the first element of to_try, which is the pair of items a = (A₃, B₂, C₃, D₂) and b = (A₃, B₃, C₂, D₂). There are two criteria on which these items differ: B and C. Thus, we check for preferences regarding R({B}, a) vs. R({B}, b), R({C}, a) vs. R({C}, b), R({B}, a) vs. R({C}, b), and R({C}, a) vs. R({B}, b). The first two relations follow from dominance: they are R({B}, a) < R({B}, b) and R({C}, a) > R({C}, b), respectively. The third does not, so we ask the user to decide between them, that is, to say which of (A₃, B₃, C₂, D₃) and (A₃, B₃, C₃, D₂) is better. Suppose the user chooses the former. Then R({B}, a) > R({C}, b), and since R({C}, a) = (A₃, B₃, C₃, D₃) = R({B}, b), we may conclude a > b by segmentwise dominance.

We continue in this way, filtering out pairs from to_try whose relations are known or that contain items deducible to be outside the top-2 subset, taking the first remaining element of to_try, and attempting to use segmentwise dominance to find their relation, asking the user when necessary. The exact process depends on the user's answers to the questions about hypothetical items. If the user chooses the second option to the subsequent three questions, this suffices for Artiruno to conclude that the top-2 subset is {candidate 4, candidate 27}, with candidate 4 being better. Figure 2 shows the inferred preferences among the alternatives. The user is thus recommended to advance these two candidates to the next round of selection.

g/exex_prefs

Subjects were recruited from Prolific, an England-based web service that allows users to enroll in paid online human-subjects research. The study was implemented as a web application split into three phases: screening, the main or "scenario" phase, and a follow-up. Internally (i.e., in terms of the task code and its database), each phase corresponded to one session except that the scenario phase was split into two sessions, one before condition assignment and one after, to keep subjects from changing their earlier answers after being assigned a condition. In terms of Prolific, each phase was represented as a separate study.