Token confidence

Which words was the model unsure about? Any LLM API that returns logprobs tells you, token by token. Paste a response requested with logprobs=True (OpenAI, vLLM, llama.cpp, OpenRouter, Together, Ollama) and get a heatmap with the model's top alternatives on hover. It runs every-other-token from PyPI in your browser with Pyodide; nothing is uploaded.

Loading Python...

What the samples show

Both samples are real, unedited output of the small open model SmolLM2-360M-Instruct (greedy decoding, logprobs from its raw logits). In the first, it names Lance Armstrong as the 1994 winner at 19% probability; the winner was Miguel Indurain, and the low score is the warning. In the second, it puts Kilimanjaro at 4,637 meters; the real height is 5,895 m. The leading "4" scores 52%, just above the default line: move the slider to 55% and the wrong height lights up. A low probability is a signal, not proof: a model can be confidently wrong.

Same thing in your own code

pip install every-other-token llm-token-visualizer

from openai import OpenAI
import every_other_token as eot
import llm_token_visualizer as ltv

resp = OpenAI().chat.completions.create(
    model="gpt-4.1-mini", messages=[{"role": "user", "content": "Who won the 1994 Tour de France?"}],
    logprobs=True, top_logprobs=5)
eot.render(resp)                  # this heatmap, in Jupyter
print(ltv.score(resp).summary())  # low-confidence spans with the alternatives
assert ltv.gate(resp, 0.4)        # fail CI when an answer has a shaky word

llm-token-visualizer has a compiled Rust core, so it runs in your Python, not in this page. The Rust CLI is every-other-token on crates.io. Both are MIT.