Functions
Compare Metrics Between Function Versions
Compares metrics between two function versions to show lift or regression
This endpoint retrieves metrics for two function versions and calculates:
- Absolute differences for each metric
- Percentage lift/regression for each metric
- Field-level changes that contribute to overall differences
Use Cases
- Version Comparison: Compare metrics between any two versions of a function
- Regression Detection: Identify if a new version has regressed in performance
- Improvement Tracking: Track lift percentages across function iterations
Metrics Compared
- Accuracy: Overall correctness of extractions
- Precision: True positives / (True positives + False positives)
- Recall: True positives / (True positives + False negatives)
- F1 Score: Harmonic mean of precision and recall
- Confusion Matrix: TP, FP, TN, FN counts
- Precision-Recall AUC: Area under the PR curve
Authorization
API Key x-api-key<token>
Authenticate using API Key in request header
In: header
Request Body
application/json
functionName*string
Name of the function to compare versions for
baselineVersionNum?integer
Baseline version number for comparison
If not provided, defaults to the previous version (current - 1).
Range
1 <= valuecomparisonVersionNum?integer
Comparison version number
If not provided, defaults to the current version.
Range
1 <= valueisRegression?boolean
Whether to compare regression test data only
If true, only compares transformations marked as regression tests.
Default
falseResponse Body
application/json
application/json
application/json
application/json
curl -X POST "https://api.bem.ai/v2/functions/compare" \ -H "Content-Type: application/json" \ -d '{ "functionName": "invoice-transformer", "baselineVersionNum": 2, "comparisonVersionNum": 3 }'{
"functionName": "invoice-transformer",
"baselineVersionNum": 2,
"comparisonVersionNum": 3,
"baselineTransformationCount": 100,
"comparisonTransformationCount": 150,
"aggregateComparison": {
"accuracy": {
"baselineValue": 0.85,
"comparisonValue": 0.92,
"difference": 0.07,
"liftPercent": 8.24
},
"precision": {
"baselineValue": 0.88,
"comparisonValue": 0.94,
"difference": 0.06,
"liftPercent": 6.82
},
"recall": {
"baselineValue": 0.82,
"comparisonValue": 0.9,
"difference": 0.08,
"liftPercent": 9.76
},
"f1Score": {
"baselineValue": 0.85,
"comparisonValue": 0.92,
"difference": 0.07,
"liftPercent": 8.24
}
},
"fieldMetricsChanges": []
}{
"error": "functionName is required",
"code": "validation_error"
}{
"error": "Function 'invoice-transformer' not found",
"code": "not_found"
}{
"message": "invalid request body: missing required field 'transformationIDs'",
"code": 400,
"details": {
"field": "transformationIDs",
"reason": "required field missing"
}
}