> Markdown alternate of https://steerscale.com/services/ai-systems-audit Fixed-scope production engineering # AI Systems Audit Find where your AI system is unnecessarily expensive, slow, unreliable or difficult to operate. One real production workflow · Quantified evidence · Prioritized remediation [Discuss your AI system](https://steerscale.com/contact) Pricing and scope - FIXED SCOPE - VENDOR NEUTRAL - SENIOR ENGINEER LED - NO LONG-TERM CONTRACT AI SYSTEMS AUDIT Fixed price One production workflow. Fixed scope agreed before work starts. Price depends on workflow complexity and the evidence available. DURATION 2–3 weeks, mostly async COVERS Economics · Reliability · Observability · Evaluation 01 / Why teams call ## The AI feature shipped. The difficult questions came afterwards. COST ### “Our AI bill is growing faster than usage.” You know what providers charge, but not which workflows, customers, retries or architectural decisions drive the increase. RELIABILITY ### “The agent usually works. Occasionally it really doesn't.” Model calls, tools, queues, retries and timeouts interact in ways that are difficult to reproduce from normal application logs. EVALUATION ### “A new model looks better. We don't know if it's actually better.” Quality, cost and latency change simultaneously. Without a representative baseline, migration becomes guesswork. OBSERVABILITY ### “One request cost ten times more than another. Why?” The execution crossed multiple models and tools, but nobody can reconstruct where the time and money went. SCALING ### “It worked in beta. Production traffic changed the behaviour.” Rate limits, concurrency, queueing and resource pressure appear once the workload becomes real. MARGIN ### “We know our total AI bill. We don't know which customers are profitable.” Provider invoices do not automatically provide product-level unit economics. 02 / Fit ## Designed for teams with a real AI workflow in production. ### Good fit - An AI system already in production — LLM, RAG or agent workflow - Meaningful usage, so behaviour can be measured - Economics matter: model spend is visible on the P&L - Reliability matters: failures reach customers - Scaling, or a model migration, is ahead of you - Some trace or log evidence exists — or its absence is the problem ### Probably not a fit - AI is still only an idea - No working workflow to measure yet - General AI education or team training - Broad transformation strategy - Looking for a low-cost outsourced development team If you are building rather than operating, [AI Systems Engineering](https://steerscale.com/services/ai-systems-engineering) is the better starting point. 03 / Inputs ## What we typically need > Missing telemetry does not automatically block an audit. Sometimes the absence of usable telemetry is itself a finding. - CODERelevant modules or repositories. - TRACESSuccessful, slow and failed runs. - USAGERequest and model volume. - COSTProvider billing or usage exports. - CONFIGURATIONModels, prompts and routing. - EVALUATION DATAWhere available. 04 / Method ## From production traffic to remediation priorities. We sample real traffic, decompose the workflow into its model, tool and retry steps, and analyse the same sample through four lenses. Where a hypothesis can be tested cheaply, we run a controlled experiment before recommending anything. 01Economics & model efficiencyCost per workflow, model fit, tokens, caching, routing02Reliability & failure behaviourRetries, timeouts, backoff, idempotency, rate limits03Observability & diagnosisTrace completeness, cost attribution, versions04Evaluation & release safetyDatasets, thresholds, canaries, rollback \[Figure: Audit analysis flow: production traffic, trace sample, workflow decomposition, parallel economics, reliability, observability and evaluation analysis, controlled experiments, findings, remediation priorities.\] 05 / What we assess ## Four views of the same production system. Every audit covers economics, reliability, observability and evaluation. The same traces feed all four, so findings stay consistent with each other. 01 · ECONOMICS & MODEL EFFICIENCY ### Are you paying for capability the workload does not need? We assess - cost per workflow - model selection - token use - context size - duplicate calls - caching - routing - failed-call cost - customer economics - provider alternatives \[Figure: Illustrative model comparison for a classification step: current model €18.00 per 1k requests at 99.0% agreement; candidate A €3.00 at 98.8%, above the 98.5% threshold; candidate B €1.20 at 94.1%, below threshold.\] 02 · RELIABILITY & FAILURE BEHAVIOUR ### What actually happens when part of the workflow fails? We assess - retries - retry amplification - timeouts - backoff - provider failure - tool failure - idempotency - concurrency - queueing - rate limits - recovery \[Figure: Illustrative retry amplification: one user request becomes two workflow retries, each with three SDK retries and two tool attempts, producing twelve calls instead of one.\] 03 · OBSERVABILITY & DIAGNOSIS ### Can your engineers reconstruct one bad request end to end? > Why did this request take 43 seconds and cost €2.18? We assess whether one trace carries - trace IDs - model calls - tool calls - retries - prompt / model version - tokens - cost - latency - deployment version - customer - outcome \[Figure: Illustrative distributed trace of a 43-second request costing €2.18: an agent loop ran the reasoning model four times for 31 seconds, an external tool took 8 seconds, retrieval 2.3 seconds.\] 04 · EVALUATION & RELEASE SAFETY ### How do you know a change actually improves the product? We assess - datasets - regression tests - candidate models - prompt versioning - model versioning - quality thresholds - cost thresholds - latency thresholds - canaries - rollback \[Figure: Illustrative release gate comparing candidate and current model: quality, cost and p95 latency thresholds pass, but three regressions in the refund-intent set hold the release.\] ### Production architecture comes with the audit. Model calls sit inside ordinary distributed systems. Where relevant, we inspect the parts around them. - QUEUES - STATE - CONCURRENCY - CACHING - WORKERS - PROVIDER COUPLING - RATE LIMITS - DEPLOYMENT 06 / What a finding looks like ## One finding, fully evidenced. ILLUSTRATIVE OUTPUT — NOT A CLIENT RESULT FINDING 3 OF 11 · WORKFLOW #A83F AI-007 HIGH MODEL EFFICIENCY ### High-cost model used for deterministic classification EVIDENCE 38% of inference calls perform a bounded classification task (12 labels, deterministic output schema) on the same model used for open-ended reasoning. Output agreement with a smaller candidate model was measured on 4,200 replayed production requests. CONTROLLED COMPARISON · 4,200 REPLAYED REQUESTS METRIC CURRENT CANDIDATE Quality / agreement 99.0% 98.8% Cost / request €0.018 €0.003 p50 latency 1.8s 0.7s POTENTIAL IMPACT ~€4,700/ month ≈ 83% of current classification spend ENGINEERING EFFORT LOW CONFIDENCE HIGH REMEDIATION PRIORITY \[Figure: Priority 1 of 5\] Route bounded classification to the candidate model behind a quality gate; keep the current model as fallback below 98.5% agreement. Subject to production validation 07 / Deliverables ## Enough evidence to know what deserves engineering time. No 70-page strategy deck. No recommendations without evidence. 1. 01 ### Current-state baseline Volume, calls, models, spend, latency, failures and retries — measured, not estimated. 2. 02 ### Severity-ranked findings Evidence, impact, recommendation, benefit, effort and confidence for each. 3. 03 ### Controlled comparisons Representative workloads tested against appropriate alternatives — models, prompts, routing or retry policy. 4. 04 ### Prioritized remediation plan Quick wins separated from structural changes, with the order we would do them in. 5. 05 ### Technical walkthrough Senior engineering discussion with the team owning the system. 08 / Pricing ## Fixed price One production workflow. Fixed scope agreed before work starts. No hourly billing, no retainer, no obligation to continue. [Discuss your AI system](https://steerscale.com/contact) INCLUDED - Audit of one production workflow - Severity-ranked, evidenced findings - Applicable controlled experiments - Remediation roadmap - Technical walkthrough with your team NOT INCLUDED - Unlimited implementation - Production support - On-call - Open-ended consulting Implementation is available separately through [AI Systems Engineering](https://steerscale.com/services/ai-systems-engineering). 09 / How it works ## Small engagement. Clear boundary. Concrete output. 1. 0120–30 MIN ### Technical fit A short conversation about one workflow and the question behind it. We say clearly if an audit would not produce useful evidence. 2. 02WRITTEN ### Scope One workflow, the evidence we will need, access boundaries and a fixed price — agreed before work starts. 3. 03ASYNC ### Analyse Code, traces, cost, failure behaviour and controlled experiments. Most of this is asynchronous; your team is not blocked. 4. 04WALKTHROUGH ### Findings Quantified report, prioritized remediation and a technical discussion with the engineers who own the system. ### Your system stays your system. ACCESS & HANDLING PRINCIPLES - READ ONLYWhere practical. - LEAST PRIVILEGEMinimum required access. - NO PRODUCTION CHANGESDuring the audit. - NO TRAININGOn customer code or data. - NDA-FRIENDLYClear handling and deletion expectations. FAQ ## Questions engineers ask before an audit. **Is this an AI strategy workshop?** No. The audit examines one production workflow you already run: its code, traces, cost data and failure behaviour. The output is quantified findings and a remediation plan for engineers, not a strategy deck or a roadmap workshop. **Do you need production access?** Usually read-only access to relevant code, traces, usage and billing data is enough. We work with least privilege, make no production changes during the audit, and can operate on exported traces or a staging replica where that is preferable. **Are you tied to one model provider?** No. The audit is vendor-neutral. Candidate models and providers are chosen for the workload under test, and comparisons are run against your quality, cost and latency thresholds rather than a preferred vendor. **Can you implement the recommendations?** Yes, as a separate engagement through AI Systems Engineering. The audit is deliberately scoped so your own team can act on it without us; implementation is optional, never a condition. **How long does an audit take?** Typically two to three weeks from scope agreement to walkthrough, depending on how quickly inputs are available. Most of the work is asynchronous and does not block your team. **What if nothing significant is wrong?** Then you receive a measured baseline and evidence that the system is in good shape, which is useful in itself. In practice, findings usually appear around retries, cost attribution and evaluation gaps even in well-built systems. **Is this only about reducing model cost?** No. Cost is one of four views. Reliability, observability and evaluation carry equal weight, and many findings are about failure behaviour or release safety rather than spend. ## Have an AI workflow in production worth pressure-testing? Bring one real system and the problem you're trying to understand. [Discuss your AI system](https://steerscale.com/contact) [hello@steerscale.com](mailto:hello@steerscale.com) 20–30 minutes · Technical conversation · No sales presentation