Nobody in this story exists, though the setting will feel familiar to anyone near a paperwork-heavy service. A small online services desk receives hundreds of scanned uploads a day: photo pages, covering letters, supporting forms, and a long tail of oddly rotated phone pictures. To save time, they train a model that sorts each upload into a category before a human reviewer opens it. It works beautifully in the demo. What follows is the next six months, with the lessons of MLOps pencilled in the margins, taken from a published reference on the discipline.

The launch that felt finished

On launch day the data lead pushed the model behind a small web service, the support lead cheered, and the team went for lunch. In their heads the project was over. In MLOps terms, it had barely begun. The Atlas frames the discipline as two jobs, shipping a trained model and then keeping it right, and observes that one cloud provider counts deployment as merely one of seven lifecycle processes. The other six were about to introduce themselves.

The first to knock was reproducibility. Three weeks in, the data lead retrained with a few hundred fresh scans and the results got slightly worse. Nobody could say why, because nobody had recorded which scans, which settings or which library versions went into the original. The team lost two days rebuilding the old result by trial and error, and the support lead asked a fair question: “Can we do this on purpose next time?”

Chapter one: the logbook

The team started a simple logbook, automated so nobody had to remember it. From then on every training run saved its commit, a fingerprint of the precise scan set, its settings and its software environment. The Atlas likes to say a model is compiled from code plus data, and the logbook made that literal. They also decided what “same result” meant: because training has random elements, a rerun was fine if its metrics stayed inside an agreed band.

It sounds like paperwork, and for a month it felt like it. Then a reviewer reported that invoices were being mistaken for supporting letters, and the logbook let the team see in minutes that the problem began with a dataset refresh, not a code change.

Chapter two: the slice nobody tested

The next episode was the one the team now tells new hires. The data lead built an improved model, and its overall accuracy was clearly higher. Pleased, she proposed to switch over that afternoon. The support lead asked a question that turned out to matter: “Higher for whom?”

They did something they had not done before. They froze a test set, split by time so that it came strictly after the training data, and compared the old and new models not only overall but on slices: low-light photos, rotated pages, and scans from phones. The new model was better on average and clearly worse on rotated phone scans, which happened to be what many clients upload. The Atlas reports a lab with the same shape: a challenger up 0.009 overall, down 0.029 on one slice, and refused by a policy written in advance. The desk adopted a gate of its own: minimum improvement, maximum slice loss, a response-time ceiling, and a check that the input format was unchanged. The weak slice was fixed with extra training examples, and the second attempt passed.

Chapter three: shadow mode

Even with a gate, the team did not want to surprise clients. They ran the new model in shadow for a fortnight: it classified every live upload, but only the old model’s answers reached the reviewers. A small number of disagreements were examined by hand. Then it handled a slice of traffic as a canary. The Atlas adds a last stage beyond this, an online comparison against something the business cares about, since a better score on paper can fail to pay off. For the desk, the measure was reviewer time per file, and it fell, which was the point of the whole project.

Chapter four: the quiet failure

The scariest episode involved no error message at all. After a form redesign on the upload page, the scans began arriving at a different resolution. The classifier’s confidence crept down, but the service stayed green. A reviewer noticed that the “unknown” pile was growing.

Because the team had by now added layered monitoring, the cause surfaced quickly. The service layer showed nothing; the data layer showed a distribution shift in image size beginning on the exact day of the redesign. The Atlas separates this from the other two lookalikes. If the training and serving pipelines had disagreed, that would be skew; if an upstream change had broken something, a defect; and only genuine drift in the world justified retraining. Here it was an upstream change, so the fix was at the source: the form was adjusted to preserve the resolution, and no retraining occurred. The team’s rule since then is to diagnose before retraining.

Interlude: deciding when to retrain

By the fourth month the team faced a question every group meets: how often should the model be refreshed? Nobody wanted a rigid calendar or a fully automatic loop. The Atlas lists five possible triggers, from someone asking to a calendar, new data, a fall in measured performance or a large shift in inputs. The desk chose a blend: a monthly review, plus an alert-driven review when the data layer showed a shift.

To settle the cadence, the data lead ran a staleness test. She trained a model on scans up to a chosen date and then measured its accuracy on each later week. The decay was gentle for six weeks and then steeper, so monthly it was. The team also decided against automatic promotion. The Atlas warns that a nightly retrain-and-promote job can feed on its own outputs and cause outages, and the desk’s reviewers, whose corrections would have flowed straight back into training, were exactly the sort of loop it had in mind.

Chapter five: the rollback they practised

Months later a new version misbehaved on a certain type of letter. The support lead, remembering the quarterly drill, asked the data lead to flip the alias to the previous version. It took a few minutes. Retraining the old model from scratch, they now knew, would have taken far longer and might not have recreated its earlier behaviour. Behind the alias sat a simple fallback too: if the service was unavailable, uploads simply went to the manual queue.

The team also remembered a smaller case from the Atlas: a mobile app store’s table that had gone six months without a refresh. Updating it alone raised the install rate by roughly two percent. Their own version of that lesson was a lookup list of document types that nobody had updated since launch. Refreshing it improved accuracy without touching the model at all.

What the team would tell you to do

  • Keep a logbook for every training run, automatically.
  • Freeze a time-split test set and test slices, not just averages.
  • Release in shadow, then canary, then measure what the business cares about.
  • Monitor service, data and prediction quality separately.
  • Diagnose before retraining, and roll back by switching, not rebuilding.
  • Keep a list of everything that reads the model’s output.

That last item is the one the desk learned in month five, when the finance team’s weekly report turned out to depend on the classifier’s labels and broke during a version change. The Atlas calls such hidden readers undeclared consumers and says they make retirement impossible.

Reading list

The story is made up, yet each piece traces back to the Ops Atlas page for MLOps, which sets out the seven core practices and the maturity levels, and ends with a modest starter recipe of three parts: a repeatable pipeline, versions that can be traced back to their inputs, and an exit ramp. For readers of this site who deal with document-driven services, two examples of documentation-heavy processes appear in the guide to requirements for an employment-based residence visa in Sri Lanka and the matching guide for the student category. They are informational reading only; always check the official source for current rules.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted