SubQ Snap 1.2: Order-Robust Decisions and Higher Long-Context Accuracy

Last month, we released SubQ Snap, a decision model built specifically to support efficient long-context workloads, and then SubQ Snap 1.1, with gains on enterprise and agentic benchmarks, calibrated probabilities, and better self-knowledge.

Today we are releasing SubQ Snap 1.2, our best decision model yet. It makes better decisions over long documents, and is substantially more consistent when the order of the answer options changes. Like its predecessors, it runs on Subquadratic Sparse Attention, a linear attention algorithm that makes our model more efficient as context length grows. As a result, the compute cost of each token stays constant as the input size grows.

We also release LongDecisionBench, a set of nine open long-context benchmarks for evaluating decision models. We believe complicated decisions often depend on evidence throughout the context length, and many current decision models either do not support inference at long-context lengths or underperform in such settings. We hope that by releasing a more comprehensive set of evaluations for long-context tasks, we can give practitioners a better proxy for how decision models will perform in realistic enterprise applications, such as long-document processing, agentic traces, and other long-horizon tasks where success depends on handling extended context.

The key highlights are:

• Top position on the Artificial Analysis Decision Index. On Artificial Analysis's independent Decision Index v2.1, Snap 1.2 scores 73, tied for first place.
• Invariance to option ordering. Reordering the options for a given question should never change the resulting decision. But transformer-based decision models learn both tokens and their positions, so the same question can get a different answer depending on option order. Snap 1.2 introduces a novel inference-time architectural change that results in nearly perfect, and stable, invariance to option ordering with negligible TTFT overhead. Empirically, we measure negligible variance with respect to option ordering across a variety of standard benchmarks.
• Better decisions over long context. Snap 1.2 includes bespoke post-training for long-context workloads. On a set of 9 benchmarks that we are open-sourcing, we find that Snap 1.2 generally outperforms other decision models on a set of diverse long-context benchmarks.
• Even better performance on HF Decision Index and on Enterprise-16. Snap 1.2 scores 69.05 on the open-source component of the HF Decision Index, outperforming Snap 1.1 (66.43), Jev 1.14 (60.73), and d1-ultra (68.14), a statistically significant gain in a paired comparison. On Enterprise-16, a subset of the Hugging Face Decision Index that covers common enterprise decisions such as tool calling, document retrieval, and intent classification, Snap 1.2 significantly improves over Snap 1.1 on 7 of the 16 subtasks.

Figure 1. Artificial Analysis Decision Index (v2.1, with AA-Symmetry v1.1). Snap 1.2 scores 73 and is tied for first place. Source: Artificial Analysis, as of November 11, 2026.

Option Order Invariance

Permutation sensitivity. Decision models are a function of a state, that is, the input that the model uses to make a decision, a query and a set of answer options. A decision model assigns a probability score to each answer option, given the state and query. Under standard settings, the options form a set rather than a sequence: applying a permutation to the answer options, that is, listing the answer options in a different order, does not change the underlying meaning of the task, and thus the decision model should return the same probabilities to the same options.

However, in practice, decision models, like the language models they are built from, are sensitive to option order. For example, on Typed Decisions, a public set of 600 enterprise workflow decisions, reordering the options changes the top answer on 12–21% of questions for the decision models we tested. This matters as in real-world deployment scenarios, option ordering is often arbitrary and can be subject to change by factors external to the decision model (e.g. by a database query or an upstream LLM). A model that depends on it changes its decisions for reasons unrelated to the state, and its probabilities cannot be taken at face value.

This sensitivity to answer option permutation is hard to remove because a transformer reads every token at a given position, and is thus inherently sequential, not set based. Training on shuffled options reduces this problem but does not guarantee generalization, and averaging over orderings multiplies inference cost by the number of orderings.

Snap 1.2 instead employs a novel architectural change that nearly completely eliminates permutation sensitivity over several tested benchmarks and real-world use cases. Figure 2 demonstrates an example on the aforementioned Typed Decisions dataset, where sensitivity is completely eliminated compared to the non-trivial sensitivity seen in Snap 1.1, Jev 1.14, and d1-ultra.

Figure 2. Option order invariance on Typed Decisions (600 enterprise workflow questions, each asked in every rotation of its options). Snap 1.2's top answer and probabilities do not change with option order. Snap 1.1, Jev 1.14 and d1-ultra change their top answer on 12–21% of questions.

Better Long Context Decisions

Subquadratic's LongDecisionBench. Many applications of decision models require reasoning over long inputs (e.g. legal documents, agent trajectories, etc). Yet, to the best of our knowledge, principled long-context evaluations of decision models are severely limited. Existing long-context benchmarks were designed for generative LLMs, and we find that converting them to the decision setting with simple heuristics introduces a plethora of issues such as information leakage from option choices, and calibration errors.

To address this gap, today we are announcing LongDecisionBench, a suite of 9 long-context benchmarks that can moreover be taxonomized into three groups:

• Two novel benchmarks designed from first principles for the decision model setting (Decision Ledger and Decision Abstain)
• Five leak-tested adaptations of established long-context benchmarks in the LLM setting (BABILong, MRCR, LongBench v2, Oolong-synth and InfiniteBench)
• Revisions of our previously released Decision RULER and Decision LCR.

We overview the motivation and methodology of each benchmark in our companion post, and will provide the suite as open source on GitHub and Hugging Face shortly.

Long-context post-training. Snap 1.2 was trained with a stage that specifically targets important long-horizon skills such as recall, state tracking, and multi-hop reasoning. Training was conducted on a mix of real-world and carefully generated synthetic documents. In the latter setting, data were designed so as to have answers that are known by construction, and the model was supervised over a mix of states over the document. Our evaluation strategy is also carefully designed to stay principled: the post-training data share no source text, vocabulary, or data with the LongDecisionBench suite described above.

Results. Decision Ledger scores a model's probabilities against exact target distributions over synthetic operation logs, with the intent of measuring the full distribution and not only the top answer. Decision Abstain pairs each answerable document with a matched unanswerable one. Both reflect how decision models are used in production: downstream systems act on the full set of probabilities (not only the top answer), and the context need not always contain the information needed to make a semantically meaningful decision. On Decision Ledger, Snap 1.2 leads Snap 1.1 and Jev 1.14 at every context length (Figure 3). On Decision Abstain, Snap 1.2 holds ~98% or above on key lookup through 256K, though the reasoning split remains hard for every model.

Figure 3. Decision Ledger and Decision Abstain, the two benchmarks we designed for decision models. Ledger is shown as 1 − probability error (higher is better).

Next, we turn our attention to the first set of LongDecisionBench evals that are constructed by careful conversion of standard LLM long-document benchmarks. Snap 1.2 leads Snap 1.1 and Jev 1.14 on all three such benchmarks (Figure 4). On Decision LCR v1.1 it scores 68.4, up 5.2 points from Snap 1.1 and 17.9 points ahead of Jev 1.14. LCR counts a question only if every part is right, so each point is a question over a roughly 100K-token document answered correctly end to end. On Decision InfiniteBench, Snap 1.2 reaches 85.9, 3.6 points above Snap 1.1 and 7.7 above Jev 1.14. Finally, on Decision LongBench v2 it scores 66.3, which is 1.0 above Snap 1.1 and 5.8 above Jev 1.14.

Figure 4. Long-document benchmarks: Decision LCR v1.1, Decision LongBench v2 (items up to 256K tokens) and Decision InfiniteBench (three core tasks). d1-ultra's 32K window is too short for these documents.

Finally, Figure 5 below shows the four LongDecisionBench benchmarks that are parameterized by context length: Decision BABILong, Decision-MRCR, Decision Oolong-synth and Decision RULER v1.1. d1-ultra is competitive at short lengths, for example at Decision BABILong at 8K and 32K tokens, but is limited by its comparatively much shorter max token length. Both Snap models cover the full range up to 256K tokens, while Jev 1.14's 262K-token limit excludes its 256K results on Decision-MRCR and Decision RULER. Snap 1.2 outperforms Snap 1.1 at nearly every benchmark and length, with gains of up to 26 points on Decision-MRCR and 19 on Decision BABILong, thus validating our dedicated long-context post-training phase described above. Against Jev 1.14, Snap 1.2 is ahead at 13 of the 21 points where both models are evaluated, including every length of Decision BABILong up to 128K tokens (the two are level at 256K), while on Decision RULER both models are near-ceiling. The exception is Decision-MRCR, where Jev 1.14 leads from 16K to 128K tokens. Decision-MRCR requires recalling a specific reply among many near-identical ones, and closing this gap is a focus of our ongoing work.

Figure 5. Decision BABILong, Decision-MRCR, Decision Oolong-synth and Decision RULER v1.1, from 1K to 256K tokens.

These long-context benchmarks are part of our effort to make decision models useful in data-intensive enterprise workflows, where a system must analyze a large volume of information and decide quickly. This paradigm is new and good ways to evaluate it are still scarce, which is why we built LongDecisionBench.

Continued Focus on Enterprise Applications

Despite Snap's frontier performance as a general decision model, we continue to specifically highlight its application to enterprise settings, where long-context capability may unlock unique applications (e.g. long legal documents, real-world tool calling, etc). Enterprise settings are exactly the type of messy, long-horizon scenarios where we expect a model to be both generally performant and reliable across bespoke and novel applications.

Our enterprise post-training follows principles analogous to our long-context work discussed above. We generate realistic enterprise documents whose correct decisions are known exactly and train toward calibrated soft targets over these data.

Figure 6 highlights the post-training gains of Snap 1.2 versus Snap 1.1 and two competitor decision models on Enterprise-16, a subset of the Hugging Face Decision Index (edition 0.3) that is representative of enterprise applications.

Figure 6. Enterprise-16, 16 public datasets from the Decision Index (edition 0.3). Left: average skill above chance, with Snap 1.2's paired wins, ties and losses against each baseline. Right: Snap 1.2's gain over Snap 1.1 on each dataset.

Figure 6 shows that Snap 1.2 gains broadly over Snap 1.1: all sixteen differences are positive, and the three largest (ACOS, PhishNChips and ToolRet) are on datasets where Snap 1.1 scored below 60. The five datasets where it already scores close to saturation are where the improvement is smallest. Against Jev 1.14, the gap exceeds 19 points on five datasets, including the document-grounded HoVer, RAGTruth and ContractNLI. d1-ultra remains the closest competitor.

Figure 7. Strong generalized performance. Snap 1.2 scores 69.05 on the full publicly available partition of the HF Decision Index v0.3, the highest score among our tested baselines.

Availability and Pricing

Snap 1.2 is available to all users at the same price as Snap 1.1: $0.04 per 1M input tokens, with no cost for output generation.

Users of previous Snap versions will be automatically upgraded to Snap 1.2 free of charge.

To start building, visit console.subq.ai.