Skip to content

nli

3 TagsUpdated MITby Moritz Laurer

Zero-shot classifiers by Moritz Laurer: every option becomes a hypothesis scored for entailment. The most accurate encoder model on typed decisions in our tests.

zero-shot396m435m
ollaya run nli --preset triage "I was charged twice for my subscription this month and want a refund."
NameAccuracyLatency
Most accurateHighest typed-decisions accuracy of the family.
nlilatestSame as nli:deberta-v3-large.0.53618 ms
FasterFastest first; accuracy as measured on the bench.
nli:modernbert-largeModernBERT-large, Apache-2.0. Faster, slightly less accurate (0.515).0.51517 ms
nli:deberta-v3-largeDeBERTa-v3-large, MIT. Most accurate encoder in our tests (0.536 on typed decisions).0.53618 ms

Accuracy: typed-decisions, argmax against the majority label on 400 states (left out for a tag trained on that data). Latency: the median five-question request on an RTX 5090, Ollaya Bench.

Readme

Zero-shot NLI classifiers by Moritz Laurer. Ollaya turns every option of a question into a hypothesis and asks the model whether the state entails it. Choice and score questions take the most-entailed option, and a yes/no question takes the probability that its statement is entailed.

Models

TagBackboneParamsLicenseTyped-decisions accuracy
nli:latest, nli:deberta-v3-largeDeBERTa-v3-large435MMIT0.536
nli:modernbert-largeModernBERT-large396MApache-2.00.515

Accuracy is the argmax against the majority label on all 400 typed-decisions states. For comparison, laya:en scores 0.357 and gliclass 0.477. The labels have low annotator agreement, so compare the numbers to each other rather than reading them as absolutes.

Usage

ollaya run nli --preset triage "Your app crashed during checkout and I was charged twice. Please refund one of the charges."

Point any TypeSafe client at http://localhost:11435 and set the model to nli.

How it works

  • One sequence pair per option. Each option is a premise–hypothesis pair (the state, then the option's hypothesis), so the cost grows with the number of options. A question with 20 options runs 20 sequences.
  • Batching. All sequences in a request share one batched forward pass.
  • Hypothesis templates. They come from each model's recommended phrasing and live in the model's decision layer.
  • Weights. They are the authors' own model.safetensors, downloaded from Hugging Face, pinned to a commit and verified by sha256. Ollaya hosts only the ONNX graph (about 5 MB).
  • Parity. Ollaya's Rust runtime reproduces the Python reference exactly: the same token ids and the same decision on every test question, on CPU and CUDA.

Limits

  • Uncalibrated. These are zero-shot entailment models, not trained decision models. Their probabilities are not calibrated, so fit a CALIBRATION layer on your own data before you rely on thresholds.
  • English. They work best on English text.
  • Training data licenses. The model card notes that part of the DeBERTa model's training data carries non-commercial licenses. The author publishes a -c variant trained on commercially usable data only; Ollaya does not ship it yet.

Badge

Run on Ollaya

For the model card or a README. Day-0 support for a new model

[![Run on Ollaya](https://ollaya.dev/badge/run-on-ollaya.svg)](https://ollaya.dev/library/nli)