This is a made-up story, a composite of the kind of evening that online service teams know well, offered to illustrate IncidentOps, the discipline of responding in an organised way to unplanned service disruptions. Imagine a small team that runs the web portal for an immigration-information service. Applicants use it to upload scanned documents, book appointment slots and check progress. No real organisation is portrayed, and the story offers no guidance on visas or immigration law. It is a story about how a team behaves when its own system breaks on a deadline-heavy day.
The evening it began
On a Thursday, an hour before the portal’s daily cut-off for submitting documents, support staff noticed a pattern of messages. Applicants were uploading files and receiving a confirmation screen, yet the status on their dashboard stayed empty. Two support agents mentioned it in the general chat. A developer glanced at it, thought it might be a slow queue, and went back to her task. Forty minutes passed.
The team had no written rule about when to call something an incident. That is the first lesson of the story. Public playbooks used by large operators suggest simple triggers, such as customers being able to see the problem, another team being needed, or an hour going by without a fix. The portal team met the first and, eventually, the third. Had a rule existed, the call would have happened in the first ten minutes.
The scramble
When the head of operations finally typed that this was an incident, nearly everyone joined a single video call. Eleven people were present. Three of them began restarting different services. Someone rolled back a configuration change without telling anyone, and someone else deployed a patch on top of it. Updates to applicants consisted of one hurried banner on the website that said only “we are aware of a problem”.
The Ops Atlas describes this pattern as the opposite of a command structure. In its account, a coordinator should decide and organise but not fix. The group that changes production should be small and known to everyone. A communicator should post on a schedule. None of those things existed, so every person did a bit of everything and nobody saw the whole picture.
The cost of that hour was not only technical. Support staff were fielding anxious messages with nothing to say. Developers were interrupted every few minutes by colleagues asking for news. Managers hovered at desks. Each of those distractions slowed the actual repair, and each could have been avoided by a single, regularly updated message that everyone could read instead of asking. The story shows how quickly a lack of structure multiplies small delays into a long evening.
The turn
About an hour into the call, an engineer from another team who happened to be listening said something sensible: “Can we agree who is in charge?” The head of operations offered the role to the person with the calmest voice on the call, a mid-level product analyst who had never touched the upload service. She accepted on one condition, that she would not try to fix anything. She asked for the names of two engineers who would be the only people changing the system, and she asked everyone else to leave the call and watch a read-only updates channel.
The change in atmosphere was immediate. The two engineers compared notes and discovered that the half-forgotten configuration rollback and the new patch were fighting each other. Undoing the patch restored uploads within twenty minutes. Nobody had located the true underlying defect yet, which turned out not to matter. The Atlas makes the same point in its guidance on mitigation: generic moves like rollbacks need only a fault location, not a verified root cause.
The communication problem
With the technical side stabilising, the analyst turned to the other half of the job. She appointed a colleague to write the customer messages and gave her a template. The first proper post named the problem in ordinary language: uploads were being accepted but not showing in dashboards, and no submissions had been lost. It promised another update within twenty minutes. A common rhythm in public playbooks is a first note within about five minutes, a scoping follow-up soon after, and further messages at least every twenty minutes for the opening two hours. The portal team was late, but it caught up, and applicants who had been anxious about their deadline wrote back with thanks.
This is also where a status page matters. The team had been using the portal’s own front page for notices, which was awkward when the front page itself was unreliable. Afterwards they moved their notices to a hosted page on separate infrastructure. A rule of thumb in the Atlas is that the channel that tells people about an outage should never share fate with the thing that is broken.
The handover
Four hours in, the analyst needed sleep. Rather than quietly drifting away, she named a successor and said the sentence aloud. The new commander replied clearly that she accepted the role. The Atlas calls out this small ritual for good reason. Without it, two people can believe they are in charge or, worse, nobody does. Staff who joined the next shift read the timeline that a volunteer scribe had kept, which saved a long recap.
The review
Before the call closed, the head of operations named the person who would run the review and set a date within a few working days. Because that facilitator had not been part of the scramble, people felt comfortable describing what they had done. The review did not stop at “operator confusion”. It asked what let two well-meaning changes collide, and the answers were about conditions: no change freeze during an incident, no single record of who was touching what, and no written criteria for declaring.
The group settled on four follow-ups, every one assigned to a single person and stamped with a deadline. None were heroic. One was to write the declaration criteria. Another was to create role cards. A third was to practise a handover. The last was to try a quarterly tabletop. Months later the actions were finished, which is the quiet measure of a healthy review.
What readers can take from the story
Teams in the immigration-information space deal with anxious, deadline-driven users, so downtime feels personal. If your own work involves guiding people through eligibility questions, our explainer on how to think about permanent residency eligibility before applying overseas shows the kind of content that people may be reading when your systems go down, and a companion piece on making sense of skilled migration to Australia shows how much trust rides on your pages being reachable. Neither article is a substitute for professional guidance, and neither is this one.
The wider lesson is that a calm, rehearsed routine beats improvisation. The sources and reasoning behind each practice in the story are collected in the Ops Atlas page on IncidentOps. Start with something small: a written threshold for declaring, a named commander and a status page that stays up when everything else is down.