Evaluation Functions

An Evaluation function grades another function's output against its schema — scoring confidence and relevance per field, flagging hallucinations, and calling out incomplete values — without producing output of its own.

When to use

Create an Evaluation function to wire grading directly into a workflow, so every run grades a node's output automatically with no separate call required.

To grade transformations that already exist, you don't need an Evaluation function: call POST /v3/eval, as described below.

Configuration fields

Required fields

Optional fields

Example POST /v3/functions request:

{
  "functionName": "invoice-grader",
  "type": "evaluation",
  "evaluationVersion": "0.2.0"
}

An update (PATCH /v3/functions/:name) doesn't need evaluationVersion.

Wiring an Evaluation node into a workflow

Add the Evaluation function as a node, then an edge from the node whose output you want graded — Extract, Transform, Analyze, or Join:

{
  "name": "invoice-pipeline",
  "mainNodeName": "extract-invoice",
  "nodes": [
    { "function": { "name": "extract-invoice" } },
    { "function": { "name": "invoice-grader" } }
  ],
  "edges": [
    {
      "sourceNodeName": "extract-invoice",
      "destinationNodeName": "invoice-grader"
    }
  ]
}

From here, every call through extract-invoice automatically queues a grading run of its output — no separate POST /v3/eval call needed. The grading run itself is asynchronous: it doesn't block or appear in the extract call's own completion status, since it's a side-effect of the run rather than a step the call waits on. Poll GET /v3/eval/results (below) or GET /v3/calls/:id/trace for the result once it's ready.

Because Evaluation is always a dead end, you can never add a node downstream of an Evaluation node.

Grading a Classify decision

An Evaluation node downstream of a Classify function grades the decision itself — was the chosen label correct, given the full set of labels it was chosen from — rather than a schema-checked output. Classify edges always require a destinationName naming one of the function's classifications, same as any other edge out of a Classify node; the Evaluation edge is no exception, but its destinationName is otherwise irrelevant — the grading run fires regardless of which label the classifier actually chose, since grading isn't itself a branch:

{
  "sourceNodeName": "classify-doc-type",
  "destinationNodeName": "doc-type-grader",
  "destinationName": "invoice"
}

The result carries a single synthetic /choice entry in fieldMetrics in place of a per-schema-field breakdown. Its suggestedValue is the label that should have been chosen — always one of the function's classifications: the evaluator's suggestion when it judges the choice wrong, otherwise the chosen label itself. It's null when no classification fits the document. Grading a Classify decision is only available by wiring an Evaluation node into a workflow — the manual POST /v3/eval trigger below only accepts Extract/Transform/Analyze/Join transformations. GET /v3/eval/results doesn't resolve classify-graded results either (it's keyed by transformation, which a Classify decision doesn't have); read the result from an evaluation-type subscription webhook or GET /v3/calls/:id/trace instead.

Triggering an evaluation manually

For ad-hoc or backfill grading of transformations that already exist — rather than wiring grading into a workflow — POST /v3/eval queues evaluation jobs for a batch of transformations directly. Transformations must belong to an event of a supported type: extract, transform, analyze, or join.

{
  "transformationIDs": ["tr_01HXAB...", "tr_01HXCD..."]
}

The call returns immediately:

{ "queued": 2, "skipped": 0 }

skipped counts transformations that already have a pending or completed evaluation. errors (when present) maps a transformation ID to why it couldn't be queued — not found, or an unsupported event type.

Reading results

Evaluations run asynchronously. Poll GET /v3/eval/results with eventIDs (preferred) or transformationIDs — up to 100 per request, comma-separated:

curl -G "https://api.bem.ai/v3/eval/results" \
  -H "x-api-key: $BEM_API_KEY" \
  -d eventIDs=evt_01HXAB...,evt_01HXCD...

Each requested ID lands in exactly one of three buckets:

{
  "results": {
    "evt_01HXAB...": {
      "fieldMetrics": {
        "/invoice/number": {
          "confidenceScore": 0.97,
          "reasoning": "Matches canonical invoice number in the source document.",
          "hallucination": false,
          "relevanceScore": 1.0,
          "suggestedValue": "INV-4521"
        },
        "/invoice/total": {
          "confidenceScore": 0.08,
          "reasoning": "Extracted 73.00 but the document states the total as 137.00.",
          "hallucination": true,
          "relevanceScore": 0.95,
          "suggestedValue": 137.0
        },
        "/invoice/poNumber": {
          "confidenceScore": 0.05,
          "reasoning": "PO-5512 is printed in the header but was not extracted.",
          "hallucination": true,
          "relevanceScore": 0.9,
          "suggestedValue": "PO-5512"
        }
      },
      "overallConfidence": 0.37,
      "runtime": 3.42,
      "hasHallucinations": true,
      "incomplete": {
        "/invoice/poNumber": "PO-5512 is printed in the header but was not extracted."
      },
      "createdAt": "2026-04-23T18:05:00Z"
    }
  },
  "pending": [
    { "eventID": "evt_01HXCD...", "createdAt": "2026-04-23T18:04:55Z" }
  ]
}

• results — completed evaluations, keyed by event KSUID. fieldMetrics gives per-field confidenceScore, relevanceScore, hallucination, the evaluator's reasoning, and a suggestedValue, keyed by JSON pointer. This endpoint doesn't resolve Classify-graded evaluations (see Grading a Classify decision above) — only Extract/Transform/Analyze/Join.
• pending — still running.
• failed — terminal failure or not found, with an errorMessage.

suggestedValue is the value the field should have. It's the evaluator's suggestion when it gives one — for a wrong, missing or messy value — and otherwise the extracted value itself. It's always a single value (never a list or object) that validates against the field's own schema — a number for a number field, one of the allowed values for an enum, an ISO date for a date field — so it can be applied to the output as-is. null means the field should be empty, the evaluator couldn't determine a value, or the field is an array or object. A suggestion that can't be made to fit the schema is also returned as null rather than a wrong-typed value.

incomplete (when present) maps a JSON pointer to an explanation for every field with a missing value: a truncated array, a null or empty value that should have had one, or a required field missing outright. A value that's present but wrong — like /invoice/total above — isn't missing, so it's not listed here; it shows up as hallucination: true in fieldMetrics. A null for information the document genuinely doesn't contain is a correct extraction and isn't listed either. Absent when nothing is missing.

ungraded (when present) lists the JSON pointers of fields the evaluator was asked about but never answered, even after a retry — usually because its response for that batch of fields couldn't be used. Those fields have no entry in fieldMetrics and count as 0 in overallConfidence, so a result with ungraded scores lower for reasons unrelated to extraction quality. Re-trigger the evaluation to grade them. Absent when every field was graded.

Related