Evaluation Functions

An Evaluation function grades another function's output against its schema — scoring confidence and relevance per field, flagging hallucinations, and calling out incomplete values — without producing output of its own.

When to use

The main reason to create an Evaluation function is to wire grading directly into a workflow, so every run is graded automatically with no separate call required. Create one when you want to:

• Grade a node's output every time the workflow runs, as part of the run itself
• Pin a specific evaluation (LLM-judge) version for an environment or tenant, instead of relying on the server default
• Give an evaluation methodology a name, a version history, and an audit trail, the same way you'd version an Extract schema

You can also reference an Evaluation function from a manual POST /v3/eval call instead of wiring it into a workflow — useful for ad-hoc or backfill grading of transformations that already exist. If you're happy with the server's default evaluation version and don't need either of these, you don't need an Evaluation function at all — pass evaluationVersion directly on the manual trigger call, as described below.

Configuration fields

Required fields

Optional fields

Example POST /v3/functions request:

{
  "functionName": "invoice-grader",
  "type": "evaluation",
  "evaluationVersion": "0.2.0"
}

An update (PATCH /v3/functions/:name) that omits evaluationVersion — a display-name-only rename, for example — leaves the current version in place. It doesn't need to be re-sent on every update.

Evaluation versions

A version never changes once released, so results from the same version are always comparable.

Wiring an Evaluation node into a workflow

Add the Evaluation function as a node, then an edge from the node whose output you want graded — Extract, Transform, Analyze, or Join:

{
  "name": "invoice-pipeline",
  "mainNodeName": "extract-invoice",
  "nodes": [
    { "function": { "name": "extract-invoice" } },
    { "function": { "name": "invoice-grader" } }
  ],
  "edges": [
    {
      "sourceNodeName": "extract-invoice",
      "destinationNodeName": "invoice-grader"
    }
  ]
}

From here, every call through extract-invoice automatically queues a grading run of its output using invoice-grader's configured evaluationVersion — no separate POST /v3/eval call needed. The grading run itself is asynchronous: it doesn't block or appear in the extract call's own completion status, since it's a side-effect of the run rather than a step the call waits on. Poll GET /v3/eval/results (below) or GET /v3/calls/:id/trace for the result once it's ready.

Because Evaluation is always a dead end, you can point multiple Evaluation nodes at the same source node — for example, to grade one output with two different evaluation versions — but you can never add a node downstream of an Evaluation node.

Grading a Classify decision

An Evaluation node downstream of a Classify function grades the decision itself — was the chosen label correct, given the full set of labels it was chosen from — rather than a schema-checked output. Classify edges always require a destinationName naming one of the function's classifications, same as any other edge out of a Classify node; the Evaluation edge is no exception, but its destinationName is otherwise irrelevant — the grading run fires regardless of which label the classifier actually chose, since grading isn't itself a branch:

{
  "sourceNodeName": "classify-doc-type",
  "destinationNodeName": "doc-type-grader",
  "destinationName": "invoice"
}

The result carries a single synthetic /choice entry in fieldMetrics in place of a per-schema-field breakdown. From 0.2.0 its suggestedValue is the label that should have been chosen — always one of the function's classifications: the evaluator's suggestion when it judges the choice wrong, otherwise the chosen label itself. It's null when no classification fits the document. Grading a Classify decision is only available by wiring an Evaluation node into a workflow — the manual POST /v3/eval trigger below only accepts Extract/Transform/Analyze/Join transformations. GET /v3/eval/results doesn't resolve classify-graded results either (it's keyed by transformation, which a Classify decision doesn't have); read the result from an evaluation-type subscription webhook or GET /v3/calls/:id/trace instead.

Triggering an evaluation manually

For ad-hoc or backfill grading of transformations that already exist — rather than wiring grading into a workflow — POST /v3/eval queues evaluation jobs for a batch of transformations directly. Transformations must belong to an event of a supported type: extract, transform, analyze, or join.

Pass exactly one of two ways to pick the evaluation version — they're mutually exclusive:

A raw version string, for a one-off or when you don't need a pinned, versioned methodology:

{
  "transformationIDs": ["tr_01HXAB...", "tr_01HXCD..."],
  "evaluationVersion": "0.2.0"
}

A reference to an Evaluation function, to reuse a pinned, versioned methodology:

{
  "transformationIDs": ["tr_01HXAB...", "tr_01HXCD..."],
  "evaluationFunctionVersion": { "name": "invoice-grader" }
}

Omit both and the server's default evaluation version (0.2.0) is used. evaluationFunctionVersion accepts the same { id | name, versionNum? } shape as any other function reference; leave versionNum off to always use the function's current version.

The call returns immediately:

{ "queued": 2, "skipped": 0 }

skipped counts transformations that already have a pending or completed evaluation. errors (when present) maps a transformation ID to why it couldn't be queued — not found, or an unsupported event type.

Reading results

Evaluations run asynchronously. Poll GET /v3/eval/results with eventIDs (preferred) or transformationIDs — up to 100 per request, comma-separated:

curl -G "https://api.bem.ai/v3/eval/results" \
  -H "x-api-key: $BEM_API_KEY" \
  -d eventIDs=evt_01HXAB...,evt_01HXCD...

Each requested ID lands in exactly one of three buckets:

{
  "results": {
    "evt_01HXAB...": {
      "fieldMetrics": {
        "/invoice/number": {
          "confidenceScore": 0.97,
          "reasoning": "Matches canonical invoice number in the source document.",
          "hallucination": false,
          "relevanceScore": 1.0,
          "suggestedValue": "INV-4521"
        },
        "/invoice/total": {
          "confidenceScore": 0.08,
          "reasoning": "Extracted 73.00 but the document states the total as 137.00.",
          "hallucination": true,
          "relevanceScore": 0.95,
          "suggestedValue": 137.0
        },
        "/invoice/poNumber": {
          "confidenceScore": 0.05,
          "reasoning": "PO-5512 is printed in the header but was not extracted.",
          "hallucination": true,
          "relevanceScore": 0.9,
          "suggestedValue": "PO-5512"
        }
      },
      "overallConfidence": 0.37,
      "runtime": 3.42,
      "hasHallucinations": true,
      "evaluationVersion": "0.2.0",
      "incomplete": {
        "/invoice/poNumber": "PO-5512 is printed in the header but was not extracted."
      },
      "createdAt": "2026-04-23T18:05:00Z"
    }
  },
  "pending": [
    { "eventID": "evt_01HXCD...", "createdAt": "2026-04-23T18:04:55Z" }
  ]
}

• results — completed evaluations, keyed by event KSUID. fieldMetrics gives per-field confidenceScore, relevanceScore, hallucination, the evaluator's reasoning, and (from 0.2.0) a suggestedValue, keyed by JSON pointer. This endpoint doesn't resolve Classify-graded evaluations (see Grading a Classify decision above) — only Extract/Transform/Analyze/Join.
• pending — still running.
• failed — terminal failure or not found, with an errorMessage.

suggestedValue is the value the field should have. It's the evaluator's suggestion when it gives one — for a wrong, missing or messy value — and otherwise the extracted value itself. It's always a single value (never a list or object) that validates against the field's own schema — a number for a number field, one of the allowed values for an enum, an ISO date for a date field — so it can be applied to the output as-is. null means the field should be empty, the evaluator couldn't determine a value, or the field is an array or object. A suggestion that can't be made to fit the schema is also returned as null rather than a wrong-typed value. Results from 0.1.0-gemini don't include it.

incomplete (when present) maps a JSON pointer to an explanation for every field with a missing value: a truncated array, a null or empty value that should have had one, or a required field missing outright. A value that's present but wrong — like /invoice/total above — isn't missing, so it's not listed here; it shows up as hallucination: true in fieldMetrics. A null for information the document genuinely doesn't contain is a correct extraction and isn't listed either. Absent when nothing is missing.

ungraded (when present) lists the JSON pointers of fields the evaluator was asked about but never answered, even after a retry — usually because its response for that batch of fields couldn't be used. Those fields have no entry in fieldMetrics and count as 0 in overallConfidence, so a result with ungraded scores lower for reasons unrelated to extraction quality. Re-trigger the evaluation to grade them. Absent when every field was graded.

Related