Workflow — how to use Librari Evals end-to-end

This page is the map. Every box below is a feature page in the manual; this page is just the order you'll typically use them in.

If you don't yet know what a Request Schema Version, Ground Truth Response, or Test Result Set is, start with Concepts. The walkthroughs below assume you know the vocabulary.

The big picture

Librari Evals' workflow is shaped like a funnel. You define the question (a schema), capture the right answer (ground truth), assemble the test (a test set), measure (a benchmark), keep an eye on things over time (scheduled runs), and ship (the Extract API).

       Document Type
            │
            ▼
   Request Schema Version  ◄──── Base Prompt + Field Prompts
            │
            │      ┌──────────────────────┐
            │      │ Ground Truth Docs    │
            │      │ + Ground Truth       │
            │      │   Responses          │
            │      └──────────┬───────────┘
            │                 │
            └────────┬────────┘
                     ▼
                 Test Set
                     │
       ┌─────────────┼──────────────┐
       ▼             ▼              ▼
 Manual Test     Scheduled       Extract API
   Run           Test Runs       (production)
       │             │
       └──────┬──────┘
              ▼
      Test Result Sets
              │
              ▼
       Test Analytics

The same Test Set drives both your "is this new model better?" check AND your recurring "did anything quietly break?" monitoring. The same Request Schema Version drives both your benchmarks AND your production extractions — there's no separate "production schema". What you measured is what you ship.

Path 1 — Build it once

This is the up-front work. Mostly once per Document Type, occasionally re-done as you iterate.

1. 1. Pick a Document Type

   Decide what kind of document you're extracting from. Examples: "MSA", "Invoice", "Insurance Quote". Create the Document Type in the admin — everything else hangs off it.

   Most teams start with one. You can always add more later.

2. 2. Build a Request Schema

   Create a Request Schema Type under your Document Type, then create your first Request Schema Version inside it. Add the fields you want extracted, attach a Base Prompt and per-field prompts, save the version as a Draft, then Publish it when you're confident.

   See Build a Request Schema for the step-by-step.

3. 3. Upload Ground Truth Documents

   Add 5–20 representative documents to your corpus. The system runs OCR on each one. See Upload Ground Truth Documents.

   More is better up to a point — past about 50 documents your benchmark cost grows faster than the marginal accuracy signal you get back. Aim for diversity (different layouts, edge cases, weird values) rather than volume.

4. 4. Define expected responses

   For each ground-truth document, write down what the correct extraction should be. This is the slowest part — you're producing the answer key by hand. See Define Expected Responses for Ground Truth.

Path 2 — Benchmark continuously

Once you have a published schema, ground truth, and a test set, you can answer pretty much every interesting model-quality question that comes up.

1. 5. Assemble a Test Set

   Bundle some of your ground-truth documents into a versioned Test Set. You'll usually have a few — a small fast one for quick iteration, a larger comprehensive one for "is this ready to ship?" checks. See Assemble a Test Set.

2. 6. Run a Manual Test whenever you have a question

   Use Run a Manual Test for ad-hoc benchmarks: "is the new Claude version better than the old one?", "does my v3 schema beat my v2?", "what's the cheapest model that still hits 90% accuracy?"

   A Manual Test is a single (Test Set × Schema Version × Model Deployment) execution. The result lands in your Test Result Sets.

3. 7. Read the Test Analytics

   View Test Analytics cuts your accumulated test results in every direction that matters: by model, by schema version, by field, by document, over time, head-to-head. This is where "evaluation" turns into a business decision.

4. 8. Schedule recurring runs to catch regressions

   LLM providers update models silently — the same gpt-4o you benchmarked last month might behave differently today. Schedule Recurring Tests re-runs your test sets daily or weekly, so quality drops show up as scheduled-run-result diffs instead of as customer complaints.

   Pair scheduled runs with Explore Costs to also catch unexpected spend changes.

Path 3 — Ship to production

When a (Schema Version × Model Deployment) combo passes your bar, expose it to your application.

1. 9. Register your provider credentials

   Manage LLM Provider API Keys — Librari Evals needs your OpenAI / Anthropic / Azure / Vertex credentials wired up before any extraction (manual, scheduled, or API) can actually run.

   These get encrypted at rest with AES-256-GCM and never come back in plaintext after the first reveal.

2. 10. Mint an Extract API key

   Manage Extraction API Keys — create a bearer token your application server will use to call /api/extract. The token is shown once at creation; stash it in your application's secret manager.

3. 11. Call the Extract API

   Your application server posts a document plus a Request Schema Version ID and a Model Deployment ID to /api/extract (multipart) and gets back structured JSON. Because the API uses the same Schema Versions your benchmarks ran against, what you measured is what production gets.

   If your documents are scans and your schema requires text, OCR them first with the separate POST /api/ocr call, then send the flattened PDF to /api/extract. Born-digital PDFs skip that step.

   See Call the Extract API for the full request/response contract, curl + Python + TypeScript examples, the text-layer requirement, and the error table.

4. 12. Iterate without breaking production

   When you want to try a new schema version or a new model in production, you don't edit the live schema. You publish a new Request Schema Version (which auto-increments its number), benchmark it side-by-side with the current production version, and then point your application at the new version's ID once you're happy. The old version stays Published indefinitely. Rollback is just changing the ID your app sends.

Reading order for first-time users

If you're brand new and don't know where to start, this is the path:

1. Concepts — vocabulary
2. Build a Request Schema — make your first schema, even before you have ground truth
3. Upload Ground Truth Documents and Define Expected Responses — capture the answer key
4. Assemble a Test Set — bundle the answers
5. Run a Manual Test — first benchmark
6. View Test Analytics — see what your model is good and bad at
7. Then iterate, schedule, and ship.