The watcher

Most systems are observable in principle and unread in practice. The data is being collected. Somebody could go and look. But looking means picking a time range, remembering the query syntax, joining two things that were never meant to be joined, and knowing in advance roughly what you are looking for — which is a small project, and small projects do not happen casually.

So the looking does not happen, and the first sign of trouble arrives from somewhere else: a customer, a bill, a colleague asking whether something is down. Not because the information was missing. Because reading it cost more than the question seemed to be worth at the time.

The watcher removes the assembly. The system stays legible while it runs, so asking is cheap enough to do on a hunch — and a hunch you can check in ten seconds is worth having.

It reads a window of real operational logs and answers questions about them: which deployments happened, what failed, how long requests took, what the sequence was. Every figure it states comes from a tool call you can watch it make.

This particular window has a trap in it, and it is the reason this dataset was chosen. The deployment that failed was also the fastest — its requests returned in about a tenth of a second, because the handler never loaded and there was no work to do. The healthy deployment that replaced it served a request that took over eight seconds, because eight seconds is what the work actually costs.

A latency chart of that window ranks the outage as the best-performing version ever shipped. Ask the agent which deployment had the best response times and watch what it does with the question. Getting that right is most of what separates a system you can read from a dashboard you can misread.

The data is a snapshot of CloudWatch logs from the Lambda function serving this page. It is our own operational history, and it was not curated: the first deployment in the window is a packaging mistake we made shipping the demonstration next door — an ES module sent to a runtime expecting CommonJS. It is in here because that is what logs are for.

The agent was built with these tools:

• list_deployments

  Every deployment in the log window, oldest first, with when it started, whether it came up healthy, how many requests it served, and how fast those requests were. Use this first when asked what happened or which version broke. Health and timing are returned together on purpose: one deployment in this window failed to start, and it was the fastest.

• find_errors

  The failures in the window, with when each happened, which deployment it came from, its error type and its message. Defaults to real errors only. Pass severity "warning" for the non-fatal notices or "all" for both — warnings here fired inside healthy invocations, so counting them as failures would misreport a working deployment as broken.

• invocation_stats

  Counts and timings, for the whole window or for one deployment: how many requests ran, how many failed, cold starts, fastest, slowest, median, and memory used against memory allocated. Pass a deployment id from list_deployments to scope it. Returns null for a deployment that does not exist, which is different from one that ran and did nothing.

• timeline

  Everything that happened in order on one clock — deployments, requests and failures interleaved. Use this when the question is about sequence: what broke, when it was fixed, what happened next. This is the tool that shows the recovery rather than describing it.

• render_chart

  Draw a small chart of figures ALREADY RETURNED by another tool in this same turn, when a picture makes the answer clearer than a list — comparing several values, or showing a sequence. Call your data tools first; this tool reads no data of its own. Every value is checked against what those tools returned and the chart is refused if a figure cannot be found among them, so use returned figures exactly and never round, scale or estimate them. Needs at least two points. Use "bar" to compare labelled things and "line" only when the points are in a genuine order, such as time. The chart supplements your written answer; it never replaces it.

Every deployment, request and failure is parsed out of the raw event text rather than summarised by hand, and the raw text is published beside it with a checksum, so the parse and its source can be checked against each other. A written summary of a log is a claim nobody can verify, and it goes stale the moment anything is redeployed.

The shape transfers wherever a system is generating a record of itself that nobody reads. Application logs, job runs, queue depth, sync failures, integration retries, deployment history — anything where the evidence exists and consulting it is a task rather than a question.

The work is the same three pieces:

• — a reader that turns raw events into things worth asking about
• — an agent constrained to speak only from it, including about what it cannot see
• — enough provenance that any answer can be traced back to the underlying record

The honest test of whether this fits you is not technical. It is whether the last few times something went wrong, you found out by looking — or by being told. If it was mostly being told, the instrumentation is probably fine and the reading is what is missing.

• — Spend across cloud, software subscriptions and payment providers stays readable as it accrues, so a line that quietly doubled is something you notice the week it happens rather than at the year-end review.
• — The state of the systems the business runs on is legible at a glance, so the first sign of a problem is not a customer reporting it.
• — Who has access to what, and who approved it, is a standing view rather than something reconstructed each time an audit asks for it.
• — When you want to know whether the change you made last month actually worked, the before and after is already there — nobody has to go and assemble it first, which is the reason that question usually goes unasked.