Comparing Functions on a Dataset

Model comparison grades several functions (or versions) against the same dataset of known-correct answers, so you can see which one extracts most accurately — and how each differs from a baseline. It reuses the eval-score grading engine per entry and reports precision / recall / F1, latency, and lift against the baseline entry. Grading runs are internal and not billed.

Everything is API-key authenticated (x-api-key; account + environment come from the key) and asynchronous: you POST to create a resource that returns immediately with an id and pending status, then poll a GET until it is complete.

select functions ──▶ dataset (gds_) ──▶ comparison (cmp_) ──▶ metrics + lift ──▶ mislabel check

1. Build a dataset

A dataset holds your ground truth: input files paired with the expected JSON. The most direct way is to build one from the corrected transformations of functions already in your account.

POST /v3/datasets/from-function-outputs

{
  "name": "invoices golden",
  "query": {
    "functionNames": ["invoices", "invoices-experiment"],
    "isLabelled": true
  }
}

• isLabelled: true keeps only reviewed/corrected rows — the ground truth a comparison scores against.

The response contains the new dataset: { "dataset": { "id": "gds_…", … } }.

Choosing which outputs become rows

query selects the rows, from coarse to fine. Combine as needed:

• By function — functionNames (or functionIDs) pulls every corrected transformation of those functions, across all versions.
• By (function, version) pair — functionVersions pins exact versions. Pairs union rather than cross-product: [{ "functionName": "invoices", "versionNum": 2 }, { "functionName": "receipts", "versionNum": 3 }] yields exactly invoices@v2 + receipts@v3. Omit versionNum to take all versions of a function.
• Individually — transformationIDs, eventIDs, or referenceIDs hand-pick specific rows.

{
  "query": {
    "functionVersions": [
      { "functionName": "invoices", "versionNum": 2 },
      { "functionName": "invoices", "versionNum": 3 }
    ],
    "isLabelled": true
  }
}

2. (Optional) Score a single function first

Before comparing, you can grade one function against the dataset with the eval-score endpoint — handy for a quick sanity check.

POST /v3/eval/score

{
  "functionName": "invoices",
  "functionVersionNum": 3,
  "datasetID": "gds_…",
  "matchConfig": { "numericTolerance": 0.01, "stringMatch": "fuzzy" }
}

Returns 202 { "scoreRunID": "evalrun_…", "status": "pending" }. Poll GET /v3/eval/score/{scoreRunID} for the aggregate score and per-row, per-field diffs.

3. Create a comparison

POST /v3/model-comparisons

{
  "name": "invoices v3 vs v2 vs experiment",
  "datasetID": "gds_…",
  "entries": [
    { "functionName": "invoices", "functionVersionNum": 2, "label": "v2" },
    { "functionName": "invoices", "functionVersionNum": 3, "label": "v3" },
    { "functionName": "invoices-experiment", "label": "candidate" }
  ]
}

• Each entry is a (functionName, functionVersionNum) pair — one function version, i.e. one configuration to grade. Omit functionVersionNum to use the function's current version.
• Provide 2–3 distinct entries (the same (function, version) twice is rejected). They may span different functions as well as different versions. The dataset is capped at 1000 rows.
• The first entry is the baseline; every other entry's lift is measured against it.
• Columns (input / corrected / schema) resolve by role, so no column names are needed. Because the dataset carries a per-row schema, every entry is scored against that ground-truth schema — so the comparison stays fair as functions and schemas evolve.

Returns 202 { "comparisonID": "cmp_…", "status": "pending" }.

4. Read the results

GET /v3/model-comparisons/{comparisonID}?matchMode=normalized&orderMatching=false

matchMode (strict | normalized | semantic) controls how leaf values are judged equal, and orderMatching whether array elements are compared in order. Grading is recomputed on read, so you can change these without re-running the functions.

{
  "comparisonID": "cmp_…",
  "status": "complete",
  "entries": [
    {
      "label": "v2", "functionName": "invoices", "functionVersionNum": 2,
      "scoreRunID": "evalrun_…", "status": "complete", "isBaseline": true,
      "coverage": { "completed": 50, "total": 50 },
      "metrics": {
        "aggregateMetrics": { "precision": 0.88, "recall": 0.85, "f1Score": 0.86, "accuracy": 0.91, "tp": 120, "fp": 16, "fn": 21, "tn": 300 },
        "fieldMetrics": []
      },
      "latencyPercentiles": { "latencyP50": 2.1, "latencyP90": 3.4, "latencyP95": 3.9 }
    },
    {
      "label": "v3", "isBaseline": false,
      "metrics": { "aggregateMetrics": { "f1Score": 0.91, "…": "…" } },
      "lift": {
        "f1Score":   { "baselineValue": 0.86, "comparisonValue": 0.91, "difference": 0.05, "liftPercent": 5.8 },
        "precision": { "…": "…" }, "recall": { "…": "…" }, "accuracy": { "…": "…" }
      }
    }
  ]
}

Each entry carries its metric bundle (aggregate + per-field accuracy, latency, and a dataset baseline when present). The baseline entry has no lift; every other entry's lift gives the difference and percent change against it. GET /v3/model-comparisons lists comparisons newest-first, paginated with limit (default 50, max 100) and a startingAfter comparison-ID cursor.

5. Surface likely-mislabeled data

Because a comparison grades several independent functions against the same labels, it is a strong label-error detector: when several entries agree with each other on a value but disagree with the label, the label is the likely error — not the models.

There is no dedicated endpoint; it is a read over the comparison itself. Fetch it with ?includeRowResults=true (and your chosen matchMode):

GET /v3/model-comparisons/{comparisonID}?includeRowResults=true&matchMode=normalized

Each entry then carries rowResults: [{ rowKey, fields: [{ path, category, expected, actual }] }], where category is match / mismatch / missing / extra — re-matched under the same matchMode as the metrics, so a surface-form difference you've told the comparison to treat as equal won't show up as a false mislabel. Group by (rowKey, path): where the entries whose category is mismatch converge on the same actual, flag that row for review.

Reference

Comparison status (rolled up from the entries' score runs): pending → running → complete when all entries succeed, partial if some entries errored but at least one succeeded (the successful results are still usable), error if all failed, or cancelled if you cancel it.

Cancel a running comparison with POST /v3/model-comparisons/{comparisonID}/cancel — it stops the still-running entries; already-finished entries keep their results.

matchConfig (applies per grading run): numericTolerance, stringMatch (exact | fuzzy), fuzzyThreshold, arrayMatch, ignorePaths.