Zero-shot classification on a CPU

For classification on a CPU, use an encoder decision model: laya:multilingual answers five questions in 516 ms and decima:small in 591 ms, with labels you write in each request and no training. The LLM-based decision models are more accurate, but take seconds on a CPU.

Which model

Model	Typed decisions	Public benchmark	Five questions, CPU	Memory
`laya:multilingual`	0.349	0.579	516 ms	1941 MB
`decima:small`	0.414	0.462	591 ms	586 MB
`laya:en`	0.357	0.532	945 ms	3207 MB
`decima:base`	0.483	0.550	1213 ms	1128 MB
`nli:deberta-v3-large`	0.536	0.458	2318 ms	3848 MB
`gliclass:large`	0.477	0.434	2133 ms	2415 MB
`snap:2b-q4_k_m`, an LLM	0.655	0.691	1735 ms	2802 MB

Medians through the HTTP API, five triage questions per request, on Ollaya Bench's CPU, a 24-core Threadripper 3960X; memory is the peak. Typed decisions is 2,000 typed questions; the public benchmark is Bespoke Labs', scored as macro accuracy.

Speed: laya:multilingual and decima:small, both multilingual.
English: laya:en is trained for guardrails and email triage.
Accuracy among encoders: nli:deberta-v3-large scores highest on typed decisions, and takes 2318 ms.
Many labels: the time of nli and gliclass grows with the number of labels (gliclass has the measurements).
Hard questions: the encoders fall well below the LLM-based models. If a second or two per request is fine, snap:2b-q4_k_m is far more accurate; see Jev alternative for the GPU models.

Times depend on your CPU. /api/decide reports eval_duration with every answer, so you can measure on yours.

Classify

Labels go in the request, as the criteria of a choice question:

ollaya pull laya:multilingual
ollaya run laya:multilingual \
  --questions '{"topic":{"type":"choice","instructions":"What is the message about?","criteria":["billing","login","shipping","other"]}}' \
  "Mi pedido lleva dos semanas sin llegar."

The same through the API:

curl http://localhost:11435/v1/systemone -H "Authorization: Bearer local" -d '{
  "model": "laya:multilingual",
  "state": "Mi pedido lleva dos semanas sin llegar.",
  "questions": {"topic": {"type": "choice", "instructions": "What is the message about?", "criteria": ["billing", "login", "shipping", "other"]}}
}'

Each label gets a probability, so you can send the uncertain answers to a person or a larger model. Labels can carry a description too: "criteria": {"shipping": "Late, lost or damaged deliveries", ...}.

To replace an LLM prompt that classifies today, see Replace LLM classifiers, which has a script to measure agreement with your own labels.