Evaluation Functions

An Evaluation function grades another function's output against its schema — scoring confidence and relevance per field, flagging hallucinations, and calling out incomplete values — without producing output of its own.

When to use

The main reason to create an Evaluation function is to wire grading directly into a workflow, so every run is graded automatically with no separate call required. Create one when you want to:

• Grade a node's output every time the workflow runs, as part of the run itself
• Pin a specific evaluation (LLM-judge) version for an environment or tenant, instead of relying on the server default
• Give an evaluation methodology a name, a version history, and an audit trail, the same way you'd version an Extract schema

You can also reference an Evaluation function from a manual POST /v3/eval call instead of wiring it into a workflow — useful for ad-hoc or backfill grading of transformations that already exist. If you're happy with the server's default evaluation version and don't need either of these, you don't need an Evaluation function at all — pass evaluationVersion directly on the manual trigger call, as described below.

Configuration fields

Required fields

Optional fields

Example POST /v3/functions request:

{
  "functionName": "invoice-grader",
  "type": "evaluation",
  "evaluationVersion": "0.1.0-gemini"
}

An update (PATCH /v3/functions/:name) that omits evaluationVersion — a display-name-only rename, for example — leaves the current version in place. It doesn't need to be re-sent on every update.

Wiring an Evaluation node into a workflow

Add the Evaluation function as a node, then an edge from the node whose output you want graded — Extract, Transform, Analyze, or Join:

{
  "name": "invoice-pipeline",
  "mainNodeName": "extract-invoice",
  "nodes": [
    { "function": { "name": "extract-invoice" } },
    { "function": { "name": "invoice-grader" } }
  ],
  "edges": [
    {
      "sourceNodeName": "extract-invoice",
      "destinationNodeName": "invoice-grader"
    }
  ]
}

From here, every call through extract-invoice automatically queues a grading run of its output using invoice-grader's configured evaluationVersion — no separate POST /v3/eval call needed. The grading run itself is asynchronous: it doesn't block or appear in the extract call's own completion status, since it's a side-effect of the run rather than a step the call waits on. Poll GET /v3/eval/results (below) or GET /v3/calls/:id/trace for the result once it's ready.

Because Evaluation is always a dead end, you can point multiple Evaluation nodes at the same source node — for example, to grade one output with two different evaluation versions — but you can never add a node downstream of an Evaluation node.

Grading a Classify decision

An Evaluation node downstream of a Classify function grades the decision itself — was the chosen label correct, given the full set of labels it was chosen from — rather than a schema-checked output. Classify edges always require a destinationName naming one of the function's classifications, same as any other edge out of a Classify node; the Evaluation edge is no exception, but its destinationName is otherwise irrelevant — the grading run fires regardless of which label the classifier actually chose, since grading isn't itself a branch:

{
  "sourceNodeName": "classify-doc-type",
  "destinationNodeName": "doc-type-grader",
  "destinationName": "invoice"
}

The result carries a single synthetic /choice entry in fieldMetrics in place of a per-schema-field breakdown. Grading a Classify decision is only available by wiring an Evaluation node into a workflow — the manual POST /v3/eval trigger below only accepts Extract/Transform/Analyze/Join transformations. GET /v3/eval/results doesn't resolve classify-graded results either (it's keyed by transformation, which a Classify decision doesn't have); read the result from an evaluation-type subscription webhook or GET /v3/calls/:id/trace instead.

Triggering an evaluation manually

For ad-hoc or backfill grading of transformations that already exist — rather than wiring grading into a workflow — POST /v3/eval queues evaluation jobs for a batch of transformations directly. Transformations must belong to an event of a supported type: extract, transform, analyze, or join.

Pass exactly one of two ways to pick the evaluation version — they're mutually exclusive:

A raw version string, for a one-off or when you don't need a pinned, versioned methodology:

{
  "transformationIDs": ["tr_01HXAB...", "tr_01HXCD..."],
  "evaluationVersion": "0.1.0-gemini"
}

A reference to an Evaluation function, to reuse a pinned, versioned methodology:

{
  "transformationIDs": ["tr_01HXAB...", "tr_01HXCD..."],
  "evaluationFunctionVersion": { "name": "invoice-grader" }
}

Omit both and the server's default evaluation version is used. evaluationFunctionVersion accepts the same { id | name, versionNum? } shape as any other function reference; leave versionNum off to always use the function's current version.

The call returns immediately:

{ "queued": 2, "skipped": 0 }

skipped counts transformations that already have a pending or completed evaluation. errors (when present) maps a transformation ID to why it couldn't be queued — not found, or an unsupported event type.

Reading results

Evaluations run asynchronously. Poll GET /v3/eval/results with eventIDs (preferred) or transformationIDs — up to 100 per request, comma-separated:

curl -G "https://api.bem.ai/v3/eval/results" \
  -H "x-api-key: $BEM_API_KEY" \
  -d eventIDs=evt_01HXAB...,evt_01HXCD...

Each requested ID lands in exactly one of three buckets:

{
  "results": {
    "evt_01HXAB...": {
      "fieldMetrics": {
        "/invoice/number": {
          "confidenceScore": 0.97,
          "reasoning": "Matches canonical invoice number in the source document.",
          "hallucination": false,
          "relevanceScore": 1.0
        }
      },
      "overallConfidence": 0.97,
      "runtime": 3.42,
      "hasHallucinations": false,
      "evaluationVersion": "0.1.0-gemini",
      "createdAt": "2026-04-23T18:05:00Z"
    }
  },
  "pending": [
    { "eventID": "evt_01HXCD...", "createdAt": "2026-04-23T18:04:55Z" }
  ]
}

• results — completed evaluations, keyed by event KSUID. fieldMetrics gives per-field confidenceScore, relevanceScore, hallucination, and the evaluator's reasoning, keyed by JSON pointer. This endpoint doesn't resolve Classify-graded evaluations (see Grading a Classify decision above) — only Extract/Transform/Analyze/Join.
• pending — still running.
• failed — terminal failure or not found, with an errorMessage.

incomplete (when present) maps a JSON pointer to an explanation for every field the evaluator flagged as short of the mark: a truncated array, a wrong scalar, a null that should have had a value, or a required field missing outright. Absent when nothing was flagged.

Related