The story below is invented. Its team, its portal and its numbers are illustrative, assembled so that the lessons of ObservabilityOps can be seen in motion. The setting is a small online service that publishes plain-language reference pages about residence and travel documents for many countries, a bit like the pages on this very site. The team has six people, a handful of services and a monitoring setup nobody has ever audited. What follows is a year in their working life, told month by month, with the practical decisions highlighted as they go.

January: the quiet outage

On a Monday morning a support colleague forwards a message from a reader who says the fee table on a country page has not changed since the autumn, although the official figures were revised. The engineering lead opens the dashboard: everything is green. She checks the log of page updates and finds that the publishing job has failed every night for nine days. The alert for failed jobs existed, but it depended on the same telemetry pipeline that had silently stopped delivering data the previous week.

That evening she reads the Ops Atlas page on ObservabilityOps and recognises the problem. The Atlas separates two jobs: using observability data to run services, and operating the observability system itself. Her team had been doing the first and ignoring the second. She decides that someone has to own it, and volunteers herself for the first month.

February: ownership, inventory and a heartbeat

She follows the opening moves in the Atlas’s thirty-day plan. First, a written owner: her name, with a deputy. Second, an inventory: a spreadsheet listing every service that emits data, every place the data lands and who pays. It reveals three forgotten test environments still shipping logs. Third, a heartbeat: a test alert that fires every minute and an external probe, hosted outside their own infrastructure, that sends an email when the heartbeat stops. Within a week the probe catches a certificate problem on the collector, which would previously have gone unnoticed for days.

The team also fixes the three worst offenders in their metric labels. One of them was a page identifier, thousands of values, attached to a counter. Dropping it cuts the number of stored series sharply and shortens dashboard load times as a bonus.

March: the label that came back

A new feature records how readers move between country pages. A developer, reasonably, adds the country code and the visa category as labels. That is fine. Another developer adds the reader’s session as well. Within a day the series count doubles. Because the lead had added a pre-release estimate in February, the build fails with a message showing the projected series count against the team’s allowance. The developer moves the session into a log field, and the feature ships on time.

The Atlas rule of thumb the lead printed for the wall says to keep the variety of any single metric label under about ten values and to investigate when it passes a hundred. The team adds that sentence to its contributor guide, along with an example of a bad label and a good one.

April and May: standards that people actually use

With the basics steady, the lead writes a two-page instrumentation standard. It covers naming, which log fields must appear at the top level (trace identifier, span identifier and severity), and how services identify themselves. A quick query shows that a fifth of the data arrives with an unknown service name. She treats that share as a quality metric, aiming for near zero, and by the end of May the figure has dropped to a trivial remainder.

She also introduces limits at the gateway and a redaction rule. Readers of an information service rarely submit personal data, but contact forms do, and she wants no such details drifting into logs. As a related safeguard she adds an allow-list for the context-carrying baggage that travels between services and strips it before requests leave the building, in line with the Atlas warning that baggage can leak to outside APIs.

Colleagues who handle content sometimes ask whether the information on the site is current. The site’s own reference page on a student residence category in Sri Lanka is an example of the type of page whose freshness matters most. For those pages the team starts a data-freshness canary: a synthetic update that moves through the publishing path every hour, timed from the moment it starts until it shows up on the live page.

June: the span-metrics surprise

Encouraged, the team turns on metrics derived from trace spans. A week later, the series count climbs again, and the lead is puzzled. The explanation is in the Atlas: a per-instance identifier on the generated series multiplies the count by the number of collector replicas, and it changes on every restart, which creates churn. The Atlas is explicit that neither stripping labels during the scrape nor loosening the sample limit cures it. What works is a sum that leaves the instance label out when aggregating. They apply it, restart a collector on purpose and watch the series count hold steady.

July to September: alert reviews and the first unit cost

Summer is quieter, which is when the team conducts its first alert review. Of the pages sent over the past quarter, fewer than half led to any action. They delete several, rewrite others to describe reader-visible symptoms and keep the heartbeat. They also compute a monthly unit cost for the first time, dividing the telemetry bill by the number of page requests served. The number is not impressive, but it exists, and the finance partner appreciates a single figure to track.

For test material they reuse the employment residence category in Sri Lanka, since its fee and processing sections change often enough to exercise the canary.

October to December: sampling and a calmer pager

By autumn, one service, the search function, generates most of the trace volume. The team decides to apply tail sampling there only. They keep every trace with an error, every slow request and a small fraction of the rest, routing all spans of each trace to the same instance. They pilot it for a week, ask colleagues whether any trace they needed is missing, and then adopt it. Trace storage costs fall, and unit cost trends down while coverage holds, which the Atlas describes as the top rung of its maturity ladder.

The conversation that changed the budget

In November the finance partner asked a pointed question: why does the telemetry bill grow faster than readership? The lead came prepared. She showed the unit-cost trend, the effect of the label fixes and the series-count estimate that now guards every release. She also admitted what she could not yet prove, such as how much value the long-term archive provided. The two agreed on a simple experiment: shorten the hot tier, move older data to cheaper storage, and check in three months whether anyone had missed it.

That candour mattered. Nobody had to pretend that observability was either free or priceless. It was a service with a measurable cost, a named owner and a record of improvements, which matches the fourth rung on the Atlas ladder, the governed product.

What the team learned

  • The first outage was not a tooling failure; it was an ownership failure.
  • Cheap early steps such as a heartbeat and an inventory pay off immediately.
  • Limits that fail builds are kinder than limits that fail production.
  • One unit-cost figure makes budget conversations shorter.
  • The owner never took over everyone’s alerts. Each service team kept deciding what should wake its own people.

Practical takeaways for a team of your size

A team of six does not need a dedicated platform group. In the Atlas’s decision guide, a named on-call owner is justified for small organisations when they host their own back ends or work in a regulated field; otherwise a documented standard and a rotating champion may be enough. As the number of teams grows and cost or noise become real concerns, the balance shifts towards a dedicated function.

If you recognise your own January in this story, begin with the cheapest items: write down an owner, make the inventory and install a heartbeat with an outside probe. Everything else in the year above grew from those three small steps.

0 0 votes
Article Rating
Subscribe
Notify of
guest
0 Comments
Oldest
Newest Most Voted