Compare Metrics Between Function Versions

Compares metrics between two function versions to show lift or regression

This endpoint retrieves metrics for two function versions and calculates:

• Absolute differences for each metric
• Percentage lift/regression for each metric
• Field-level changes that contribute to overall differences

Use Cases

• Version Comparison: Compare metrics between any two versions of a function
• Regression Detection: Identify if a new version has regressed in performance
• Improvement Tracking: Track lift percentages across function iterations

Metrics Compared

• Accuracy: Overall correctness of extractions
• Precision: True positives / (True positives + False positives)
• Recall: True positives / (True positives + False negatives)
• F1 Score: Harmonic mean of precision and recall
• Confusion Matrix: TP, FP, TN, FN counts
• Precision-Recall AUC: Area under the PR curve

curl -X POST "https://api.bem.ai/v2/functions/compare" \  -H "Content-Type: application/json" \  -d '{    "functionName": "invoice-transformer",    "baselineVersionNum": 2,    "comparisonVersionNum": 3,    "isRegression": true  }'

{
  "functionName": "invoice-transformer",
  "baselineVersionNum": 3,
  "comparisonVersionNum": 4,
  "baselineTransformationCount": 100,
  "comparisonTransformationCount": 80,
  "aggregateComparison": {
    "f1Score": {
      "baselineValue": 0.92,
      "comparisonValue": 0.78,
      "difference": -0.14,
      "liftPercent": -15.22
    }
  }
}

{
  "message": "baseline and compare versions cannot be the same"
}

{
  "message": "No transformations found for function 'invoice-transformer' version 5"
}

{
  "message": "string",
  "code": 0,
  "details": {}
}

See also

• System overview