Skip to content

decision2

6 TagsUpdated Apache-2.0by vLLM Semantic Router

Decision 2.0 by the vLLM Semantic Router contributors: a fully fine-tuned Qwen3 or Qwen3.5 backbone plus an endpoint head that scores every option at its own last token against the question, in one forward pass per question. Five sizes, 0.6B to 9B.

decisionlong-contextfine-tuned0.60b0.75b1.88b4.21b7.94b
ollaya run decision2 --preset triage "I was charged twice for my subscription this month and want a refund."
NameAccuracyLatency
Most accurateHighest typed-decisions accuracy of the family.
decision2:luxQwen3.5-9B, the largest: 0.707 on typed decisions.0.707364 ms
FasterFastest first; accuracy as measured on the bench.
decision2:kaiQwen3-0.6B-Base, the fastest: 31 ms for five questions on an RTX 5090; 8k-token rows.0.47631 ms
decision2:eosQwen3.5-0.8B: 0.460 on typed decisions.0.46069 ms
decision2:solQwen3.5-2B: 0.554 on typed decisions.0.55489 ms
decision2:noxQwen3.5-4B-Base: 0.680 on typed decisions; 251 ms for five questions on an RTX 5090.0.680251 ms
decision2latestSame as decision2:nox.0.680251 ms

Accuracy: typed-decisions, argmax against the majority label on 400 states (left out for a tag trained on that data). Latency: the median five-question request on an RTX 5090, Ollaya Bench.

Readme

Decision 2.0 is the vLLM Semantic Router contributors' second generation of decision models, released under Apache-2.0 with their code in each model repository. Each is a fully fine-tuned text backbone (Qwen3 for Kai, Qwen3.5 for the others) with an endpoint head: every option is scored at its own last token against the question, in one forward pass per question. It never generates text.

Models

TagBaseParamsContextOllaya Bench score
decision2:kaiQwen3-0.6B-Base0.60B8,19238.0
decision2:eosDecision 1.0 Eos (Qwen3.5-0.8B)0.75B16,38447.0
decision2:solDecision 1.0 Sol (Qwen3.5-2B)1.88B16,38450.0
decision2:noxQwen3.5-4B-Base4.21B16,38460.9
decision2:luxDecision 1.0 Lux (Qwen3.5-9B)7.94B16,38463.2

The 27B Vega uses the same layout but is not in the library.

Usage

ollaya run decision2 --preset triage "My order never arrived and support ignores me. Refund me today or I'm switching to your competitor."

Point any TypeSafe client at http://localhost:11435 and set the model to decision2.

Accuracy

Measured by Ollaya on an RTX 5090, zero-shot as far as the model cards say (they do not list the training sets):

Tagtyped-decisions (400 states)Public benchmark, macro accuracyPooled ECE
decision2:kai0.4760.6080.046
decision2:eos0.4600.6610.093
decision2:sol0.5540.6780.053
decision2:nox0.6800.7460.050
decision2:lux0.7070.7560.056

Speed

Five triage questions in one request, end to end over HTTP (Ollaya Bench), median:

TagRTX 5090CPU (Threadripper 3960X)GPU memory
decision2:kai31 ms1798 ms3701 MB
decision2:eos69 ms3226 ms5909 MB
decision2:sol89 ms4637 ms11027 MB
decision2:nox251 ms12130 ms12387 MB
decision2:lux364 ms18670 ms20349 MB

Every question is a forward pass of its own, so a request costs about one question's time per question.

How it works

  • One row per question. The state, the question and each option in its own segment; the head reads every option's last token against the row's last token. Questions do not see each other.
  • Temperature 1. No package ships a calibration file, so the probabilities are the softmax of the option logits, as upstream computes them. Kai adds the author's fixed offsets to five-level score logits first.
  • Weights. The files download from Hugging Face, pinned to a commit and verified by sha256, byte for byte the authors'. Ollaya hosts only the ONNX graph. The 4B and 9B models keep their BF16 weights in BF16 in memory.
  • Parity. Ollaya's Rust runtime matches the author's own code in fp32 for every tag: identical token rows, the same rejections and the same decision on every test question, on CUDA and on the CPU.

Limits

  • Context. 8,192 tokens per question for Kai, 16,384 for the others, the state included; a longer question is rejected, never truncated.
  • Options. 2 to 255 choice options, 2 to 10 score levels.
  • Not ported. share_context (one forward pass for all of a request's questions) and the GGUF builds.

Badge

Run on Ollaya

For the model card or a README. Day-0 support for a new model

[![Run on Ollaya](https://ollaya.dev/badge/run-on-ollaya.svg)](https://ollaya.dev/library/decision2)