{"id":983,"date":"2026-11-17T10:25:00","date_gmt":"2026-11-17T10:25:00","guid":{"rendered":"http:\/\/desinri.com\/blog\/?p=983"},"modified":"2026-11-20T20:25:00","modified_gmt":"2026-11-20T20:25:00","slug":"mlops-diagnosis-desk","status":"publish","type":"post","link":"https:\/\/desinri.com\/blog\/mlops-diagnosis-desk\/","title":{"rendered":"Why Did Our Estimates Go Wrong? An MLOps Diagnosis Guide"},"content":{"rendered":"<p>A small travel-services firm builds a model that estimates how long a document-checking step will take, so that its advisers can tell clients roughly when to expect the next update. For months the estimates are good. Then complaints begin: clients are told &#8220;about a week&#8221; and wait three. Nothing has crashed, no error appears in any log, and the data team insists the model passed every test before launch. This is the classic shape of an MLOps problem, and this article works through it as a diagnosis: symptom, likely causes, checks, and the matching fix.<\/p>\n<p>The firm is fictional, and nothing here concerns immigration rules or anyone&#8217;s application; the point is the method. The technical ideas are drawn from a reference entry on MLOps, the discipline of putting trained models to work and then keeping their answers right.<\/p>\n<h2>The symptom: a confident answer that no longer matches reality<\/h2>\n<p>The first temptation is to say &#8220;the model is stale, retrain it&#8221;. It is a natural reaction, but the Atlas urges diagnosis before treatment, because three quite different faults look identical on a dashboard. Writing down the candidates is the first step:<\/p>\n<ol>\n<li>The code that prepares inputs during training and during live use has come apart (skew).<\/li>\n<li>Something upstream changed or broke (a data defect).<\/li>\n<li>The world itself moved (drift).<\/li>\n<\/ol>\n<p>Every one of them has its own owner and its own cure, and a wrong guess can eat weeks.<\/p>\n<h2>Check one: do the two pipelines agree?<\/h2>\n<p><strong>Problem.<\/strong> The training job computes &#8220;days since submission&#8221; from one date field, while the live service uses another, because a developer refactored the intake form.<\/p>\n<p><strong>Test.<\/strong> Take a sample of recent live records, run each through the code used for training and again through the code used for live answers, and compare the resulting features. In the setup the Atlas describes as a reference, the acceptable disagreement is exactly nil. A single figure above zero means two code paths are telling the model different stories.<\/p>\n<p><strong>Fix.<\/strong> Repair the pipelines so that one definition feeds both. Do not retrain around the discrepancy, because the model would then learn to compensate for a bug. The test also needs a permanent home: schedule it to run whenever the input code changes.<\/p>\n<h2>Check two: did something upstream change?<\/h2>\n<p><strong>Problem.<\/strong> The intake form was redesigned, and a field that used to be mandatory is now optional, so many records arrive with blanks. Or a partner system started sending dates in a different format.<\/p>\n<p><strong>Test.<\/strong> Look at the data layer of monitoring: schema violations, missing-value rates, and the freshness of the statistics jobs that watch them. In the Atlas&#8217;s reference setup, schema violations that last 15 minutes trigger a page, and a statistics job silent for more than 26 hours is flagged too, a hint that monitoring needs a freshness alarm of its own. Otherwise the checks can stop and nobody notices.<\/p>\n<p><strong>Fix.<\/strong> Correct the problem where it originates, by agreeing a data contract with whoever owns the form or feed. Retraining on defective data would cement the defect.<\/p>\n<h2>Check three: has the world actually changed?<\/h2>\n<p><strong>Problem.<\/strong> Perhaps nothing is broken. The office now handles more complex cases, or seasonal volumes have changed the mix of work.<\/p>\n<p><strong>Test.<\/strong> Compare the current input distribution with a reference period of the same kind. Be careful here: comparing a peak season against an ordinary month and calling the gap drift features among the anti-patterns the entry catalogues. Pick a sensible window, and calibrate any distance measure on your own history. The entry notes that a popular stability index has rule-of-thumb bands that lack academic support, and that its value depends on binning and sample sizes. A second measure can act as a cross-check.<\/p>\n<p><strong>Fix.<\/strong> This is the case where retraining belongs, and the details depend on what moved. If the mix of inputs changed and fresh outcomes are available, retrain. If only the frequency of outcomes shifted, recalibrate thresholds. If the relationship between inputs and outcomes changed, the answer is new labels, new features or a redesign.<\/p>\n<h2>How the story would unfold week by week<\/h2>\n<p>In week one, complaints arrive and the instinct is to blame the model. The team resists and instead opens a short incident note listing the three candidate causes. In week two, the replay check shows a mismatch in exactly one feature, the date field that changed in the intake refactor. That explains a good part of the error, but not all of it, since estimates were still worse for one category of file than another. In week three, a look at the data layer reveals that a second field, one the partner system fills in, had been arriving blank for about a fortnight. The team corrects both defects at their sources, adds the two checks to the release pipeline, and only then looks at whether any genuine drift remains.<\/p>\n<p>The final surprise is how little retraining was needed. The accuracy recovered almost entirely from pipeline and source repairs. That outcome is common enough that the Atlas treats &#8220;retrain first&#8221; as a mistake: a refreshed model on defective inputs would have taken weeks to build and learned the wrong lesson.<\/p>\n<h2>The complication: the truth arrives late<\/h2>\n<p>In this firm&#8217;s case, the real processing time is known only when a file closes, which can be weeks later. The Atlas describes the same difficulty for a churn model, where the outcome takes about 30 days, and recommends proxies in the meantime: the spread of predictions, and how the model&#8217;s predicted rate compares with a base rate that has matured. Those proxies would have shown the model&#8217;s estimates piling up around &#8220;one week&#8221; while closing times stretched, long before the complaints.<\/p>\n<h2>Containing the damage while you investigate<\/h2>\n<p>While the team diagnoses, the advisers still need something to say to clients. The right immediate action is a rollback or fallback. If a previous model version is registered with a clear alias, point traffic back to it. If not, switch to the plain non-learning fallback, perhaps a published range by case type, and tell advisers to quote ranges instead of a point estimate. The Atlas is firm that going back means flipping the alias or the traffic split, and that retraining is no way to reverse a release; it also wants the switch practised roughly every quarter, so the first attempt is not made mid-incident.<\/p>\n<h2>Preventing a repeat<\/h2>\n<p>After the fix, the firm adds the missing layers. A written release gate now compares any new version with the current one on a frozen, time-split test set: overall error, error for each case type, response time and input schema. A consumer list records every tool that reads the estimates, such as the client-update emails and the weekly workload report, so nobody is surprised by a change. And a short retraining policy names the triggers: manual request, schedule, new data, measured degradation, or significant distribution change.<\/p>\n<p>Teams that make decisions on behalf of travellers will recognise the logic. Check eligibility and inputs first, then process, then communicate; our own articles on <a href=\"http:\/\/desinri.com\/blog\/understanding-permanent-residency-eligibility-before-you-apply-abroad\/\">understanding eligibility ahead of an application abroad<\/a> and <a href=\"http:\/\/desinri.com\/blog\/planning-better-indonesia-trips-with-flexible-vehicle-rental-options-motoshare\/\">planning Indonesia trips with flexible vehicle options<\/a> reflect the same sequence of verifying the basics before committing to a plan.<\/p>\n<h2>A one-page diagnosis card<\/h2>\n<ul>\n<li><strong>Replay shows a mismatch:<\/strong> fix the pipelines, then add the replay to the build.<\/li>\n<li><strong>Data layer shows broken inputs:<\/strong> fix at the source and add a data contract.<\/li>\n<li><strong>Distributions have genuinely moved:<\/strong> pick the remedy by type of change, then retrain with a stated policy.<\/li>\n<li><strong>Not sure yet:<\/strong> fall back or roll back first, investigate second.<\/li>\n<\/ul>\n<p>For the broader scope of the practice, including what belongs to neighbouring disciplines, read <a href=\"https:\/\/www.xopsschool.com\/mlops\/\">the Ops Atlas entry on MLOps<\/a>. Its starter advice suits a wall poster: build one repeatable pipeline, keep a registry that records lineage, and have a way back, rather than buying an impressive platform that never gets used.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A fictional services firm sees its time estimates drift from reality. This problem-and-fix walkthrough shows how MLOps separates skew, bad data and real drift.<\/p>\n","protected":false},"author":8,"featured_media":984,"comment_status":"open","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[146],"tags":[],"class_list":["post-983","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-visa-immigration"],"_links":{"self":[{"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/posts\/983","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/comments?post=983"}],"version-history":[{"count":1,"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/posts\/983\/revisions"}],"predecessor-version":[{"id":1109,"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/posts\/983\/revisions\/1109"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/media\/984"}],"wp:attachment":[{"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/media?parent=983"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/categories?post=983"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/desinri.com\/blog\/wp-json\/wp\/v2\/tags?post=983"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}