Fine-tuning changes how a model speaks. Retrieval changes what it knows. Most teams reaching for the first one actually needed the second, and find that out after the training bill.
What Changes at Scale
Behaviour that is fine at ten requests a day becomes a problem at ten thousand. Rate limits, cost per call and queue depth all move from theoretical to urgent.
The version of the system that survives scale usually looks more boring than the version that impressed everyone in the demo.
Where the Real Constraint Sits
Before changing anything, map the process as it actually runs, not as the documentation claims. The gap between those two is where most of the wasted effort hides.
A constraint has a signature: work piles up in front of it and starves behind it. If you cannot point at a queue, you have not found it yet.
- Time in queue. How long does a request sit before anyone touches it?
- Rework rate. How often does a step have to be done twice because of bad input?
- Escalation frequency. How often does the process need a human override to finish?
What to Measure Before You Change Anything
Baseline first. Without a measurement taken before the change, any improvement claim afterwards is a story rather than a result.
Two weeks of honest observation beats a quarter of instrumented guesswork, provided the two weeks are representative rather than cherry-picked.
| Metric | Why it matters |
|---|---|
| Cycle time end to end | The number the customer actually experiences |
| Touches per request | Proxy for how much human attention each unit consumes |
| Exception rate | Tells you how much of the process resists automation |
The Part That Should Stay Human
Judgment under ambiguity stays human. If a step requires weighing two things that cannot both be satisfied, it is a decision and not a task.
Automating a decision does not remove the decision. It moves it earlier, into whoever wrote the rules, and hides it from the people living with the consequences.
- Anything where being wrong is expensive and hard to reverse
- Anything a customer would be upset to learn was handled by a machine
- Anything whose correct answer changes with context the system cannot see
Building for the Failure Case
Production automation is mostly error handling. The happy path is the small part, and it is the part that gets all the attention during the build.
Every external call can fail, time out, or return something unexpected. A workflow that assumes otherwise will work until precisely the moment it matters.
| Failure | Minimum acceptable handling |
|---|---|
| API timeout | Retry with backoff, then alert a human |
| Unexpected payload shape | Fail loudly, never silently continue |
| Credential expiry | Alert before expiry, not after the break |
Conclusion
The tooling is rarely the hard part. Scope it narrowly, measure the baseline first, name an owner, and build for the failure case rather than the demo.
Frequently Asked Questions
How long before this pays for itself?
Most well-scoped automations pay back within a few months, but only when the baseline was measured beforehand. Without a baseline you cannot honestly answer this question at all.
Do we need a dedicated person to run this?
Not dedicated, but named. The failure mode is shared ownership, where everyone assumes someone else is watching.
What if the underlying model or API changes?
Assume it will. Anything that depends on an exact output format should validate that format explicitly rather than trusting it.