Run Function Regression Testing
Initiates regression testing between function versions using historical transformation data
This endpoint creates function calls to test historical data with the latest function version, allowing you to measure performance improvements or regressions against ground truth data.
How It Works
- Selects Historical Data: Retrieves transformations with user corrections (ground truth)
- Creates Function Calls: Generates new function calls using the latest function version
- Returns Function Call IDs: Provides immediate response with function call tracking
- Async Processing: Function calls execute in background using standard infrastructure
Checking Results
Use standard function call endpoints to monitor progress and retrieve results:
GET /v2/functions/calls/{functionCallId}- Check individual function call status
Best Practices
- Start with smaller sample sizes (10-50) for initial testing
- Use
onlyCorrectedData: trueto ensure quality ground truth comparisons - Monitor function call completion using webhooks or polling
- Compare results manually using transformation endpoints
Data Requirements
- Function must have historical transformations with
correctedJSON(user corrections) - Baseline version must exist and have associated transformation data
- Function must be currently active and callable
Authenticate using API Key in request header
In: header
Name of the function to test for regressions
Must be an existing function with historical transformation data containing user corrections. The function must be currently active and callable.
^[a-zA-Z0-9_-]+$Function version number to use as baseline for comparison
- Defaults to
currentVersionNum - 1(previous version) - Must be a valid, existing version number for the function
- Used to retrieve historical transformation data for comparison
- Cannot be the same as
comparisonVersionNum
uint1 <= valueFunction version number to test against the baseline
- Defaults to current version number (latest version)
- Must be a valid, existing version number for the function
- This version will be used to create new function calls for testing
- Cannot be the same as
baselineVersionNum
uint1 <= valueNumber of historical samples to test
- Defaults to 50 samples
- Minimum: 1, Maximum: 1000
- Only transformations with
correctedJSON(user corrections) are eligible - Actual sample size may be smaller if insufficient corrected data exists
- Larger samples provide more statistical confidence but take longer to process
501 <= value <= 1000Whether to only test transformations with user corrections
- Defaults to
true(recommended) - When
true: Only uses transformations withcorrectedJSONas ground truth - When
false: May include transformations without corrections (less reliable) - Corrected data provides the most accurate regression testing results
trueResponse Body
curl -X POST "https://api.bem.ai/v2/functions/regression" \ -H "Content-Type: application/json" \ -d '{ "functionName": "invoice-transformer", "sampleSize": 50 }'Function calls created and processing in background
{
"functionName": "invoice-transformer",
"result": {
"functionName": "invoice-transformer",
"totalSamples": 50,
"functionCalls": [
{
"originalReferenceId": "invoice-123",
"functionCallId": "fc_2N6gH8ZKCmvb6BnFcGqhKJ98VzP"
},
{
"originalReferenceId": "invoice-456",
"functionCallId": "fc_3M7gH9ZLDnwc7CoGdHriLK99WzQ"
},
{
"originalReferenceId": "invoice-789",
"functionCallId": "fc_4P8hI0ZMEoxy8DpHeSsjMLA0XzR"
}
]
}
}{
"error": "sampleSize cannot exceed 1000",
"code": "validation_error"
}{
"error": "Function 'invoice-transformer' not found",
"code": "not_found"
}{
"error": "Internal server error occurred",
"code": "internal_error"
}