“Our AI bill is growing faster than usage.”
You know what providers charge, but not which workflows, customers, retries or architectural decisions drive the increase.
Fixed-scope production engineering
AI Systems Audit
Find where your AI system is unnecessarily expensive, slow, unreliable or difficult to operate.
One real production workflow · Quantified evidence · Prioritized remediation
- FIXED SCOPE
- VENDOR NEUTRAL
- SENIOR ENGINEER LED
- NO LONG-TERM CONTRACT
01 / Why teams call
The AI feature shipped. The difficult questions came afterwards.
“The agent usually works. Occasionally it really doesn't.”
Model calls, tools, queues, retries and timeouts interact in ways that are difficult to reproduce from normal application logs.
“A new model looks better. We don't know if it's actually better.”
Quality, cost and latency change simultaneously. Without a representative baseline, migration becomes guesswork.
“One request cost ten times more than another. Why?”
The execution crossed multiple models and tools, but nobody can reconstruct where the time and money went.
“It worked in beta. Production traffic changed the behaviour.”
Rate limits, concurrency, queueing and resource pressure appear once the workload becomes real.
“We know our total AI bill. We don't know which customers are profitable.”
Provider invoices do not automatically provide product-level unit economics.
02 / Fit
Designed for teams with a real AI workflow in production.
Good fit
- An AI system already in production — LLM, RAG or agent workflow
- Meaningful usage, so behaviour can be measured
- Economics matter: model spend is visible on the P&L
- Reliability matters: failures reach customers
- Scaling, or a model migration, is ahead of you
- Some trace or log evidence exists — or its absence is the problem
Probably not a fit
- AI is still only an idea
- No working workflow to measure yet
- General AI education or team training
- Broad transformation strategy
- Looking for a low-cost outsourced development team
If you are building rather than operating, AI Systems Engineering is the better starting point.
03 / Inputs
What we typically need
Missing telemetry does not automatically block an audit. Sometimes the absence of usable telemetry is itself a finding.
- CODERelevant modules or repositories.
- TRACESSuccessful, slow and failed runs.
- USAGERequest and model volume.
- COSTProvider billing or usage exports.
- CONFIGURATIONModels, prompts and routing.
- EVALUATION DATAWhere available.
04 / Method
From production traffic to remediation priorities.
We sample real traffic, decompose the workflow into its model, tool and retry steps, and analyse the same sample through four lenses. Where a hypothesis can be tested cheaply, we run a controlled experiment before recommending anything.
01Economics & model efficiencyCost per workflow, model fit, tokens, caching, routing02Reliability & failure behaviourRetries, timeouts, backoff, idempotency, rate limits03Observability & diagnosisTrace completeness, cost attribution, versions04Evaluation & release safetyDatasets, thresholds, canaries, rollback
ANALYSIS PIPELINE
PRODUCTION TRAFFICread-only
TRACE SAMPLEok · slow · failed
WORKFLOW DECOMPOSITIONmodels · tools · retries
ECONOMICSRELIABILITYOBSERVAB.EVALUATION
CONTROLLED EXPERIMENTSreplay · candidates
FINDINGSseverity · confidence
REMEDIATION PRIORITIESquick wins → structural
05 / What we assess
Four views of the same production system.
Every audit covers economics, reliability, observability and evaluation. The same traces feed all four, so findings stay consistent with each other.
01 · ECONOMICS & MODEL EFFICIENCY
Are you paying for capability the workload does not need?
We assess
- cost per workflow
- model selection
- token use
- context size
- duplicate calls
- caching
- routing
- failed-call cost
- customer economics
- provider alternatives
MODEL COMPARISON · CLASSIFICATION STEPQUALITY THRESHOLD ≥ 98.5%
MODEL€ / 1K REQP50AGREEMENT
current · gpt-y
baseline€ / 1K REQ18.00P501.8sAGREEMENT99.0%
baseline€ / 1K REQ18.00P501.8sAGREEMENT99.0%
candidate A
● passes threshold€ / 1K REQ3.00P500.7sAGREEMENT98.8%
● passes threshold€ / 1K REQ3.00P500.7sAGREEMENT98.8%
candidate B
✕ below threshold€ / 1K REQ1.20P500.5sAGREEMENT94.1%
✕ below threshold€ / 1K REQ1.20P500.5sAGREEMENT94.1%
cheapest model that still clears quality−83% cost / step
02 · RELIABILITY & FAILURE BEHAVIOUR
What actually happens when part of the workflow fails?
We assess
- retries
- retry amplification
- timeouts
- backoff
- provider failure
- tool failure
- idempotency
- concurrency
- queueing
- rate limits
- recovery
RETRY BRANCHING · ONE FAILED TOOL CALLNESTED, NO JITTER
user request×1
└ workflow retry×2
└ sdk retry×3
└ tool attempt×2
DOWNSTREAM CALLS
12 expected 1
WORST-CASE LATENCY
48s
COST OF FAILURE
€0.52
03 · OBSERVABILITY & DIAGNOSIS
Can your engineers reconstruct one bad request end to end?
Why did this request take 43 seconds and cost €2.18?
We assess whether one trace carries
- trace IDs
- model calls
- tool calls
- retries
- prompt / model version
- tokens
- cost
- latency
- deployment version
- customer
- outcome
req 01J9…K4QF43.1s · €2.18
classifier 0.4s
retrieval 2.3s
reasoner loop 1 7.8s
tool: crm 8.1s
reasoner loop 2 7.6s
reasoner loop 3 8.2s
reasoner loop 4 7.6s
respond 1.1s
agent loop: 4 iterations after tool failure · 72% of cost in loops 2–4▲ no loop budget
04 · EVALUATION & RELEASE SAFETY
How do you know a change actually improves the product?
We assess
- datasets
- regression tests
- candidate models
- prompt versioning
- model versioning
- quality thresholds
- cost thresholds
- latency thresholds
- canaries
- rollback
RELEASE GATE · CANDIDATE vs CURRENTREPLAY SET 1,200 CASES
GATETHRESHOLDCURRENTCANDIDATE
qualityTHRESHOLD≥ 98.5%CURRENT99.0%CANDIDATE98.8% ✓
cost / reqTHRESHOLD≤ €0.010CURRENT€0.018CANDIDATE€0.003 ✓
p95 latencyTHRESHOLD≤ 2.5sCURRENT3.1sCANDIDATE1.4s ✓
regressionsTHRESHOLD0 criticalCURRENT—CANDIDATE3 ✕
■ HOLD · 3 regressions in refund_intentrollback: prompt v13 + current model
06 / What a finding looks like
One finding, fully evidenced.
ILLUSTRATIVE OUTPUT — NOT A CLIENT RESULT FINDING 3 OF 11 · WORKFLOW #A83F
High-cost model used for deterministic classification
EVIDENCE
38% of inference calls perform a bounded classification task (12 labels, deterministic output schema) on the same model used for open-ended reasoning. Output agreement with a smaller candidate model was measured on 4,200 replayed production requests.
| METRIC | CURRENT | CANDIDATE |
|---|---|---|
| Quality / agreement | 99.0% | 98.8% |
| Cost / request | €0.018 | €0.003 |
| p50 latency | 1.8s | 0.7s |
POTENTIAL IMPACT
~€4,700/ month
≈ 83% of current classification spend
ENGINEERING EFFORT
LOWCONFIDENCE
HIGHREMEDIATION PRIORITY
Route bounded classification to the candidate model behind a quality gate; keep the current model as fallback below 98.5% agreement.
Subject to production validation
07 / Deliverables
Enough evidence to know what deserves engineering time.
No 70-page strategy deck. No recommendations without evidence.
- 01
Current-state baseline
Volume, calls, models, spend, latency, failures and retries — measured, not estimated.
- 02
Severity-ranked findings
Evidence, impact, recommendation, benefit, effort and confidence for each.
- 03
Controlled comparisons
Representative workloads tested against appropriate alternatives — models, prompts, routing or retry policy.
- 04
Prioritized remediation plan
Quick wins separated from structural changes, with the order we would do them in.
- 05
Technical walkthrough
Senior engineering discussion with the team owning the system.
08 / Pricing
Fixed price
One production workflow. Fixed scope agreed before work starts. No hourly billing, no retainer, no obligation to continue.
Discuss your AI system INCLUDED
- Audit of one production workflow
- Severity-ranked, evidenced findings
- Applicable controlled experiments
- Remediation roadmap
- Technical walkthrough with your team
NOT INCLUDED
- Unlimited implementation
- Production support
- On-call
- Open-ended consulting
Implementation is available separately through AI Systems Engineering.
09 / How it works
Small engagement. Clear boundary. Concrete output.
- 0120–30 MIN
Technical fit
A short conversation about one workflow and the question behind it. We say clearly if an audit would not produce useful evidence.
- 02WRITTEN
Scope
One workflow, the evidence we will need, access boundaries and a fixed price — agreed before work starts.
- 03ASYNC
Analyse
Code, traces, cost, failure behaviour and controlled experiments. Most of this is asynchronous; your team is not blocked.
- 04WALKTHROUGH
Findings
Quantified report, prioritized remediation and a technical discussion with the engineers who own the system.
Your system stays your system.
ACCESS & HANDLING PRINCIPLES- READ ONLYWhere practical.
- LEAST PRIVILEGEMinimum required access.
- NO PRODUCTION CHANGESDuring the audit.
- NO TRAININGOn customer code or data.
- NDA-FRIENDLYClear handling and deletion expectations.
FAQ
Questions engineers ask before an audit.
No. The audit examines one production workflow you already run: its code, traces, cost data and failure behaviour. The output is quantified findings and a remediation plan for engineers, not a strategy deck or a roadmap workshop.
Usually read-only access to relevant code, traces, usage and billing data is enough. We work with least privilege, make no production changes during the audit, and can operate on exported traces or a staging replica where that is preferable.
No. The audit is vendor-neutral. Candidate models and providers are chosen for the workload under test, and comparisons are run against your quality, cost and latency thresholds rather than a preferred vendor.
Yes, as a separate engagement through AI Systems Engineering. The audit is deliberately scoped so your own team can act on it without us; implementation is optional, never a condition.
Typically two to three weeks from scope agreement to walkthrough, depending on how quickly inputs are available. Most of the work is asynchronous and does not block your team.
Then you receive a measured baseline and evidence that the system is in good shape, which is useful in itself. In practice, findings usually appear around retries, cost attribution and evaluation gaps even in well-built systems.
No. Cost is one of four views. Reliability, observability and evaluation carry equal weight, and many findings are about failure behaviour or release safety rather than spend.
Have an AI workflow in production worth pressure-testing?
Bring one real system and the problem you're trying to understand.
Discuss your AI system hello@steerscale.com
20–30 minutes · Technical conversation · No sales presentation