Concepts — terms used in Librari Evals

Definitions for every term you'll see in the admin and the docs — schemas, ground truth, test sets, scoring rubrics, and model deployments.

This page defines every term you'll bump into elsewhere in the manual. If you're on a feature page and a phrase doesn't quite click, the definition's here.

If what you actually want is the story — how all of this fits together, in the order you'll use it — head over to Workflow.

The schema model

A schema in Librari Evals describes what you want extracted from a document. It's versioned, composable, and the same schema can be benchmarked against any number of LLMs.

What kind of document you're working with — "MSA", "Invoice", "General Contract". Most other pieces (base prompts, ground truth, test sets) hang off a Document Type, so it's the first thing you set up.

Request Schema Type

A named extraction job within a Document Type — like "MSA Key Terms" or "Invoice Line Items". The Schema Type is the parent of the Fields, Base Prompt Versions, and Request Schema Versions that make up the schema. You can also clone a Schema Type to spin up a draft variant pre-wired with the same fields and field prompts.

Request Schema Version

One frozen snapshot of a schema — its fields, its base prompt, the field prompts attached to each field, and its publication state (Draft, Published, Archived). This is what test runs and the Extract API actually point at. Once a version is Published, it's immutable — you iterate by creating a new version.

Version numbers are auto-assigned per Request Schema Type — your first is v1, then v2, and so on. You don't pick the number. Versions also have a free-form description so you can record why this version exists ("tightened the date prompt", "swapped to the v3 base prompt") — useful when you're scanning the version list weeks later.

Request Schema Field

A single field inside a schema version. Each field has:

a name (the JSON key the extracted value will land under, e.g. effective_date)
a type — one of string, integer, decimal, boolean, date, enum, json, array, object
an optional description the model sees as part of the extraction prompt
a required flag
an attached Field Prompt Version (see below)

Fields can nest up to 10 levels deep via object and array types if you need them to.

Base Prompt Version

The system prompt the model sees before any field-specific prompt. Base prompts are scoped to a Request Schema Type and versioned on their own — so you can swap in an improved base prompt for a new schema version while leaving field prompts alone. Handy when you want to isolate the impact of a single change in a benchmark.

Field Prompt Version

A per-field prompt — guidance specific to extracting one field. These are versioned and reusable across Request Schema Versions: if you've written a solid "extract a date in ISO 8601 format" prompt for v1, you can attach the same field prompt version to v2. Field Prompt Versions are scoped to a Request Schema Field, and they carry their own description for tracking what changed between versions.

Field Prompt Mapping

The connection between a Request Schema Version and the specific Field Prompt Versions it uses. Mappings live in their own collection because the same field can take different prompts in different schema versions, and the mapping is what gets versioned alongside the schema.

Ground truth

To benchmark a model you need a known-correct answer to compare its output against. That's ground truth.

Ground Truth Document

A PDF (or other source document) you've uploaded to your corpus, scoped to a Document Type. On upload the system runs OCR to pull out the text layer; that's what the model sees during scoring runs.

Ground Truth Response

The human-verified correct extraction for a Ground Truth Document × Request Schema Type pair. This is the answer key. Every per-field score in a Test Result is a comparison between what the model said and the corresponding ground truth value.

A single ground truth document can have multiple ground truth responses if you've defined multiple Request Schema Types for the same Document Type.

Test execution

Test Set

A versioned bundle of Ground Truth Documents you use as a benchmark dataset. Test Sets are how you say "evaluate the model against these documents". Like schemas, they have publication states; once published, the document membership is locked so re-runs stay reproducible.

Test Result Set

The full output of one execution: a (Test Set × Request Schema Version × Model Deployment) triple, captured as a single record. Holds the aggregate metrics — average score, total cost, total duration — and links to the per-document Test Result Documents below.

Test Result Document

The result of running one model on one document with one schema. Holds the model's actual extracted JSON, how long it took, what it cost in tokens, and the per-field score breakdown.

Test Result Field Score

The per-field comparison between the model's value and the corresponding Ground Truth Response. Carries a 0.0–1.0 score, the comparison method used, and an explanation if the score isn't perfect.

Scheduled Test Run

A cron-style configuration that re-runs a (Test Set × Request Schema Version × Model Deployment) on a schedule — daily, weekly, whatever. Useful for catching silent regressions when an LLM provider rolls out a model update server-side and forgets to tell anyone.

Scheduled Test Run Result

The Test Result Set produced by a Scheduled Test Run, plus a link back to the schedule that created it.

Scoring & rubrics

Rubric

The scoring approach used to compare a model's extracted value against the corresponding Ground Truth Response. Rubrics are pluggable — exact-match for IDs and dates, fuzzy-match or LLM-graded for free-text fields, custom comparison functions for nested structures.

Field Rubric

A rubric attached to a specific Request Schema Field — so different fields in the same schema can be scored differently. A decimal field might use exact-match while the string field next to it uses an LLM-graded rubric.

LLM-as-a-judge

A rubric where another LLM evaluates whether the extracted value semantically matches the ground truth, returning a score and an explanation. Useful for free-text fields where exact-match would be way too strict.

Models

Maker

The organisation behind an LLM — OpenAI, Anthropic, Google, etc. Makers don't really show up in day-to-day workflows; they're a categorisation layer.

LLM Model

A specific model family — GPT-4o, Claude Sonnet, Gemini Pro. Roughly "the thing the marketing copy talks about."

Model Version

A specific dated release of an LLM Model — e.g. claude-3-5-sonnet-20240620. Different versions of the same model can produce materially different extraction results, so benchmarks always pin to a specific version.

Provider

A cloud platform that hosts LLMs — Azure OpenAI, OpenAI direct, Google Vertex, Anthropic API. The same Model Version can be available on multiple providers, and you pick which one to call.

Model Deployment

A specific (Provider × Model Version) wired up with credentials and ready to use. Test runs target Model Deployments, not raw models — because cost, latency, and sometimes capability differ between, say, Azure-hosted GPT-4 and OpenAI-hosted GPT-4.

This layer is read-only for you — browse what's available via Browse Supported LLM Models. Wiring up new deployments is handled at the system level.

Access & credentials

User

A person with an account in Librari Evals.

Extraction API Key

A bearer token that authorises calls to the public Extract API. You create it in the admin, see it once, then it's masked to the last four characters forever after.

LLM Provider API Key

The encrypted credentials Librari Evals uses to talk to LLM providers on your behalf — your OpenAI key, your Anthropic key, etc. Stored AES-256-GCM-encrypted at rest and only decrypted in-memory at call time. You need at least one before any extraction (manual, scheduled, or via the Extract API) can actually call a provider.

The Extract API

A public, bearer-authenticated HTTP endpoint at /api/extract. You hand it a document and a Published Request Schema Version, and it gives you back structured JSON. This is how production applications integrate with Librari Evals once you've found a schema/model combo you trust.

The Extract API uses the same schema versions and provider credentials your benchmarks use. So what you measured in evaluation is what you actually get in production — there's no separate "production schema" to drift out of sync.

For setup, see Manage Extraction API Keys.

The OCR API

A second public, bearer-authenticated endpoint at /api/ocr. The Extract API does not OCR for you — it extracts. OCR (turning a scanned, image-only PDF into one with a selectable text layer) is its own call. You only need it when a schema requires text and your document is a scan: POST /api/ocr returns the same PDF with a text layer applied, which you then hand to /api/extract. Born-digital PDFs already have a text layer and skip it. Documents uploaded through the admin (Ground Truth Documents) are OCRd automatically and don't use this endpoint.

For the contract, see Apply OCR to a Document.

What this manual deliberately doesn't cover

A few areas live in the system but are out of scope for this manual:

Maker, LLM Model, Model Version, Model Deployment, Provider, LLM Usage Logs — system-level configuration. You'll see most of these surfaced in the read-only Browse Supported LLM Models view.
User management — granular role and access administration.

Workflow — how all these pieces fit together end-to-end.
Build a Request Schema — start defining what to extract.