Comparing Functions on a Dataset

Model comparison grades several functions (or versions) against the same dataset of known-correct answers, so you can see which one extracts most accurately — and how each differs from a baseline. It reuses the eval-score grading engine per entry and reports precision / recall / F1, latency, and lift against the baseline entry. Grading runs are internal and not billed.

Everything is API-key authenticated (x-api-key; account + environment come from the key) and asynchronous: you POST to create a resource that returns immediately with an id and pending status, then poll a GET until it is complete.

select functions ──▶ dataset (gds_) ──▶ comparison (cmp_) ──▶ metrics + lift ──▶ mislabel check

1. Build a dataset

A dataset holds your ground truth: input files paired with the expected JSON. The most direct way is to build one from the corrected transformations of functions already in your account.

POST /v3/datasets/from-function-outputs

{
  "name": "invoices golden",
  "query": {
    "functionNames": ["invoices", "invoices-experiment"],
    "isLabelled": true
  }
}

• isLabelled: true keeps only reviewed/corrected rows — the ground truth a comparison scores against.

The response contains the new dataset: { "dataset": { "id": "gds_…", … } }.

Choosing which outputs become rows

query selects the rows, from coarse to fine. Combine as needed:

• By function — functionNames (or functionIDs) pulls every corrected transformation of those functions, across all versions.
• By (function, version) pair — functionVersions pins exact versions. Pairs union rather than cross-product: [{ "functionName": "invoices", "versionNum": 2 }, { "functionName": "receipts", "versionNum": 3 }] yields exactly invoices@v2 + receipts@v3. Omit versionNum to take all versions of a function.
• Individually — transformationIDs, eventIDs, or referenceIDs hand-pick specific rows.

{
  "query": {
    "functionVersions": [
      { "functionName": "invoices", "versionNum": 2 },
      { "functionName": "invoices", "versionNum": 3 }
    ],
    "isLabelled": true
  }
}

2. (Optional) Score a single function first

Before comparing, you can grade one function against the dataset with the eval-score endpoint — handy for a quick sanity check.

POST /v3/eval/score

{
  "functionName": "invoices",
  "functionVersionNum": 3,
  "datasetID": "gds_…"
}

Returns 202 { "scoreRunID": "evalrun_…", "status": "pending" }. Poll GET /v3/eval/score/{scoreRunID} for the aggregate score and per-row, per-field diffs.

The run itself only extracts; the comparison against your expected values is redone on every read, so the numbers always reflect your dataset as it stands now.

3. Create a comparison

POST /v3/model-comparisons

{
  "name": "invoices v3 vs v2 vs experiment",
  "datasetID": "gds_…",
  "entries": [
    { "functionName": "invoices", "functionVersionNum": 2, "label": "v2" },
    { "functionName": "invoices", "functionVersionNum": 3, "label": "v3" },
    { "functionName": "invoices-experiment", "label": "candidate" }
  ]
}

• Each entry is a (functionName, functionVersionNum) pair — one function version, i.e. one configuration to grade. Omit functionVersionNum to use the function's current version.
• Provide 2–3 distinct entries (the same (function, version) twice is rejected). They may span different functions as well as different versions. The dataset is capped at 1000 rows.
• The first entry is the baseline; every other entry's lift is measured against it.
• Columns (input / corrected / schema) resolve by role, so no column names are needed. Because the dataset carries a per-row schema, every entry is scored against that ground-truth schema — so the comparison stays fair as functions and schemas evolve.

Returns 202 { "comparisonID": "cmp_…", "status": "pending" }.

4. Read the results

GET /v3/model-comparisons/{comparisonID}?matchMode=normalized&arrayMatching=any-order

matchMode (strict | normalized | fuzzy, anything else a 400) controls how leaf values are judged equal, and arrayMatching (any-order | position | sequence, anything else a 400) how array elements pair. Grading is recomputed on read, so you can change these without re-running the functions.

{
  "comparisonID": "cmp_…",
  "status": "complete",
  "entries": [
    {
      "label": "v2",
      "functionName": "invoices",
      "functionVersionNum": 2,
      "scoreRunID": "evalrun_…",
      "status": "complete",
      "isBaseline": true,
      "coverage": { "completed": 50, "total": 50 },
      "metrics": {
        "aggregateMetrics": {
          "precision": 0.88,
          "recall": 0.85,
          "f1Score": 0.86,
          "accuracy": 0.91,
          "tp": 120,
          "fp": 16,
          "fn": 21,
          "tn": 300
        },
        "fieldMetrics": []
      },
      "latencyPercentiles": {
        "latencyP50": 2.1,
        "latencyP90": 3.4,
        "latencyP95": 3.9
      }
    },
    {
      "label": "v3",
      "isBaseline": false,
      "metrics": { "aggregateMetrics": { "f1Score": 0.91, "…": "…" } },
      "lift": {
        "f1Score": {
          "baselineValue": 0.86,
          "comparisonValue": 0.91,
          "difference": 0.05,
          "liftPercent": 5.8
        },
        "precision": { "…": "…" },
        "recall": { "…": "…" },
        "accuracy": { "…": "…" }
      }
    }
  ]
}

Each entry carries its metric bundle (aggregate + per-field accuracy, latency, and a dataset baseline when present). The baseline entry has no lift; every other entry's lift gives the difference and percent change against it. GET /v3/model-comparisons lists comparisons newest-first, paginated with limit (default 50, max 100) and a startingAfter comparison-ID cursor.

5. Surface likely-mislabeled data

Because a comparison grades several independent functions against the same labels, it is a strong label-error detector: when several entries agree with each other on a value but disagree with the label, the label is the likely error — not the models.

There is no dedicated endpoint; it is a read over the comparison itself. Fetch it with ?includeRowResults=true (and your chosen matchMode):

GET /v3/model-comparisons/{comparisonID}?includeRowResults=true&matchMode=normalized

Each entry then carries rowResults: [{ rowKey, fields: [{ path, category, expected, actual }] }], where category is match / mismatch / missing / extra — produced by the same read-time scoring settings as the metrics, so the two can never disagree; a surface-form difference you've told the comparison to treat as equal via matchMode won't show up as a false mislabel. Group by (rowKey, path): where the entries whose category is mismatch converge on the same actual, flag that row for review.

Reference

Comparison status (rolled up from the entries' score runs): pending → running → complete when all entries succeed, partial if some entries errored but at least one succeeded (the successful results are still usable), error if all failed, or cancelled if you cancel it.

Cancel a running comparison with POST /v3/model-comparisons/{comparisonID}/cancel — it stops the still-running entries; already-finished entries keep their results.

Scoring is a read-time decision. Creating a run or a comparison only dispatches the extractions; nothing about how the output is judged is fixed at that point. Change the settings on the GET and the numbers change, with no re-run:

There are deliberately no tunable thresholds and no way to exclude fields: matchMode=normalized already compares numbers numerically and dates by calendar value rather than by spelling, and fuzzy adds a fixed similarity pass over free text. A percentage band, a similarity cutoff, or a list of fields to skip would each do the same thing — quietly stop counting something that was wrong.

Each score run's fieldResults carry the evidence behind every verdict — delta for numeric pairs, similarity (Levenshtein ratio) for string pairs. They tell you how close a wrong value was, which is worth knowing and never makes it right.

Pair array records

arrayMatching decides which returned array element is scored against which expected one:

sequence aligns two arrays with a weighted longest common subsequence. A record the model dropped or invented costs that record alone, and the records behind it keep their pairing; a record that moved still costs, because the pairing cannot be reordered. Two records that trade places cost one missing and one extra rather than two mismatches. Expected A, B returned as B, A matches A, counts the expected B missing, and counts the returned B extra. Under sequence, matchMode decides both which records align and whether their paired values match.

A five-row table that lost its third row, no two rows sharing a leaf value, scores 0.44 F1 under position, because rows four and five each land against their neighbour, and 0.89 under sequence, where the one dropped row is the whole cost.

sequence also reads a null differently. any-order and position find a path that is null on both sides and score it a match; sequence counts only non-null scalars, so a field neither owed nor returned never inflates the numerator. Measured over 2,280 corrected production documents that is 2,742 of 371,932 field results — 0.74%, every one of them a match the other two settings report and sequence does not.

Under sequence, a leaf is one non-null scalar — a value that is null or absent on both sides is never compared and costs nothing — and the score is a micro-averaged F1 over leaves, so a wrong value inside a 79-row table costs one leaf while a dropped table costs every leaf beneath it. Returning a string where the table was costs those same misses plus an extra, because a container facing a non-container scores as two unaligned subtrees rather than one wrong value. A mismatch counts against precision and recall both: omitting a value costs recall alone, inventing one costs both.

orderMatching is a 400 on this endpoint; send arrayMatching=position to pair elements by index.