Practical Monitoring Effective Strategies for the Real World

1. 1Write down what 'working' means for the service before you choose any check or tool. Define it from the user's perspective first, then list the components that must function to deliver it. Source: monitoring is observing the behavior and outputs of a system and its components over time, so you must know that behavior.
2. 2Inventory the layers you must understand. The passages list server hardware (out-of-band controller, RAID controller, disks), the operating system, each running service, the network infrastructure, the applications, and the interactions between them. Mark which layers you can explain and which you cannot.
3. 3Close the gaps by working with the people who own each layer, not by assigning monitoring to one specialist. 'How can you build monitoring for a thing you don't understand?' Pair the person building the check with the person who runs the component.
4. 4Map each service's dependencies. For the Django example, the app depends on a database. List every dependency the same way, then decide which of them must be healthy for the service to count as working.
5. 5Build a health check that exercises the real path. The /health endpoint connects with the app's own database configuration and runs a simple query (SELECT 1 FROM table_name, then fetchone()). It returns HTTP 200 on success and 503 with an error on failure. Reusing the app's configuration avoids the endpoint and the app using different credentials.
6. 6Add context to the endpoint: deployed version, build information, and dependency status. This helps debugging, because you can see what is running in the environment. Deepen the check over time until the app is aware of its own health.
7. 7Learn the domain-specific measures for each component type. Voice and video come down to latency, jitter, and packet loss, plus a consistent codec across the network (checkable via SNMP) and QoS policies. Capture configuration changes, for example with a RANCID-style tool, so you can tell whether a change explains a behavior shift.
8. 8Choose tools after the above, not before. Pick tools that match the needs you identified, and prototype them to verify their assumptions fit your team. Expect a set of general and specialized tools rather than one.

A Django app depends on a SQL database. A shallow check would only confirm that the web process responds. Someone who understands the system knows the app is not working if it cannot reach its database, so the health() function opens a cursor using the app's existing database connection, runs SELECT 1 FROM table_name, and calls fetchone(). On success it returns JSON with HTTP 200. On any exception it returns JSON with the error and HTTP 503. Because it reuses the app's configuration, it cannot pass with credentials the app does not use. A load balancer or service discovery tool can poll the route to pull unhealthy instances out of rotation. The same endpoint can report the deployed version so you can see what is running. The check is only as good as your understanding: it tests one dependency, and anything the app needs beyond the database goes unchecked until you add it.

• If you cannot explain what a component does and how it fails, then do not write its alerts yet; learn it first or involve the person who owns it.
• If a health check uses different credentials or configuration than the app, then change it to reuse the app's own configuration so it tests what the app actually uses.
• If a dependency's failure means users cannot be served, then include it in the health check; if it does not, then report its status without failing the check.
• If a component behaves in a domain-specific way (voice and video: latency, jitter, packet loss, codec, QoS), then monitor those specific measures instead of generic CPU or uptime.
• If the proposal is 'we need better monitoring' or a new tool, then first state the capability and mission you need, and choose tools afterward.

• Monitoring-as-a-job: one person builds monitoring for systems they do not understand. The tell is checks that pass while users are affected, and nobody else on the team can explain the alerts.
• Tool obsession: the team swaps tools every six months expecting a silver bullet. The tell is a repeating cycle of 'we need better monitoring' and an evaluation, with the underlying problem unchanged.
• Copying a famous company's tools and procedures without understanding why they work. The tell is that the tool's assumptions clash with your team's norms and workflow.
• Shallow health checks that confirm the process is up but not that dependencies work. The tell is a green /health response during a database outage.
• Assuming permanent things stay permanent, such as the codec or device configuration. The tell is a performance problem traced to a change nobody knew about, which version-controlled config tracking would have surfaced.

• ▸Service name and user-facing purpose: one sentence on what the user gets when it works.
• ▸Definition of 'working': observable conditions, from the user's perspective, that must be true.
• ▸Components and layers: list hardware, OS, services, network, and application pieces involved.
• ▸Dependencies: each dependency and whether its failure makes the service not work.
• ▸Known behaviors and domain-specific measures: the metrics that matter for this type of component (for example latency, jitter, packet loss).
• ▸Health check design: what the check exercises, using which configuration, and which status codes it returns.
• ▸Context exposed: version, build info, and dependency status shown by the endpoint.
• ▸Change tracking: how configuration changes are recorded and who is notified.
• ▸Knowledge gaps and owners: what you cannot yet explain and who will teach you.

1. 1Pull a month of alert history. List every alert with three columns: what fired, what action the responder took, and what user or business impact followed. The passages prescribe exactly this review.
2. 2Ask of each alert, 'does this require someone to act?' If the answer is no, delete it. The 'it'll clear itself up in a few minutes' alert is the first candidate.
3. 3For alerts that stay, choose a fix from the passage's list: modify the threshold, redesign the underlying check to be more accurate, or build automation that makes the alert obsolete.
4. 4Replace static thresholds where they miss the real problem. Use percent change or derivative ('disk usage has grown by 50% overnight'). If your metrics infrastructure allows (e.g., Graphite), try moving averages, confidence bands, or standard deviation.
5. 5Add user-impact signals. Track core business metrics such as searches performed, logins, comments posted, or ads purchased. They stay stable over time, so a drop shows that real users are affected even though they don't say what is broken.
6. 6Run a daily on-call tuning pass. Whoever is on-call compiles every alert that fired the previous day and asks how each alert's signal can be improved or whether it can be deleted. Repeat every shift.
7. 7Separate noise from firefighting. If the alerts are legitimate but there are dozens a day, the problem is system resilience. Have on-call work on resiliency between fires, or schedule that work in the next sprint planning from the on-call notes. Try both and keep whichever works better.
8. 8Use maintenance periods when you do planned work, so expected changes don't trigger alerts.

Take the disk-usage case from the passages. A static threshold on disk usage tells you nothing until the disk is nearly full. A percent-change or derivative check says 'disk usage has grown by 50% overnight,' which is the condition you want to know about. Say you swap the check in and review a month of history. Alerts that cleared on their own within minutes with no action are deleted. Alerts with a real action but a poor threshold get retuned. Each day the on-call engineer lists yesterday's alerts and asks whether each signal can be improved or deleted. In parallel, you put a business-metric dashboard on a screen, in the Yelp-style set of searches performed, signups, and ads purchased. If searches drop, you know users are affected before any backend alert explains why. Fewer pages arrive, and the ones that do are actionable, so people trust them again.

• If an alert does not require a human to act, delete it or convert it to a non-paging metric.
• If you recognize an alert as 'it'll clear itself up,' then fix the check, change the threshold, or automate the remedy. Do not keep paging for it.
• If a static threshold misses the failure you actually care about (a fast-growing disk), then switch to percent change or a statistical approach.
• If alerts are legitimate but numerous, then the problem is resilience, not signal. Allocate time to fix underlying systems.
• If you need to know whether a backend problem hurts users, then check a core business metric rather than inferring from infrastructure alerts.

• Living in the 'treacherous middle ground' where some alerts are actionable and some are not. The tell is that people still respond but say 'probably nothing.'
• Alert fatigue. The tell is slowing response times, ignored alerts, disrupted sleep, and staff burnout after repeated adrenaline-rush pages.
• Checkbox monitoring, where monitoring exists only to satisfy a requirement or compliance rule. The tell is monitoring described as ineffective, noisy, and untrusted.
• Tuning alerts when the real issue is unfixed systems. The tell is dozens of legitimate alerts a day and no time set aside for resiliency work.
• Relying only on infrastructure thresholds with no user-impact metrics. The tell is that you can't say what a backend slowdown does to users without investigating.

• ▸Alert name: the alert as it appears in the pager or ticket.
• ▸Times fired (last 30 days): count from the alert history.
• ▸Action taken each time: what the responder actually did, or 'none'.
• ▸User or business impact: the impact observed, e.g., a drop in a core business metric.
• ▸Self-clearing? Yes or no. Note whether it cleared within minutes without action.
• ▸Requires human action? Yes or no. If no, mark for deletion.
• ▸Fix chosen: delete, modify threshold, redesign check (e.g., percent change), or automate.
• ▸Resilience follow-up: the underlying system issue to schedule in the on-call shift or sprint planning.
• ▸Reviewed by / date: the on-call person and the daily review date.

1. 1Pull a month of alert history, as the source advises. List every alert that fired, what action someone took, and what the impact was. Trust is lost in the gap between 'fired' and 'someone did something', so measure that gap first.
2. 2Run the one question on every alert: does it require someone to act? If not, delete it. Deleting is the first move because every non-actionable alert teaches the team to ignore the next one.
3. 3Find the 'I've seen this before, it'll clear itself in a few minutes' alerts. Ask the team which alerts they would skip without a second thought. These are the middle-ground alerts (some actionable, some not), and they do the most damage to trust.
4. 4For alerts that stay, tune the threshold or redesign the check. If a static threshold misses real problems, replace it with a rate-of-change or percent-change alert, or with a seasonal comparison against the same time one week, one day, or one hour earlier.
5. 5Check whether the alert watches what 'working' means for the service. Ask the service owner what working means. For a webapp, start with an HTTP GET /, record the response code, expect a 200 OK, and check for specific text on the page. Alerts tied to real outcomes earn belief faster than CPU or memory alerts that never explain an outage.
6. 6Count alert frequency per person. Ten adrenaline-triggering alerts a week for months produces alert fatigue, slower response, and disrupted sleep. Treat the weekly count as a health metric for trust, not a staffing problem.
7. 7Spread ownership. Have network engineers and software engineers define and tune the alerts for what they know best. Trust drops when one person builds monitoring for systems they don't understand.
8. 8Re-audit on a cadence. Repeat the month-of-history review so noise doesn't build back up, and delete or tune again each time.

Take the source's disk-usage case. The team has a static alert at 'free space under 10%'. A disk jumps from 11% used to 80% used overnight and the alert stays silent, because free space (about 20%) never crosses the threshold. Meanwhile the same team gets other disk or load alerts that clear on their own within minutes, and people have learned to say 'I've seen that one, it'll clear itself.' Run the month-of-history audit and, for each alert, write down the action taken. The self-clearing alerts have no action, so delete them or raise their thresholds. Replace the disk threshold with a percent-change alert that says 'disk usage has grown by X% overnight', which catches the one event that mattered. For request volume, use seasonality: if the web servers normally handle about 100 requests/sec on a weekday, then half or double that number is worth investigating, and you compare against the same time last week. The team now sees fewer alerts, and each one maps to a real action. That is what rebuilds trust.

• If an alert fired in the last month and nobody took an action, delete it or retune it. Do not leave it 'just in case'.
• If the team says 'it'll clear itself', treat the alert as broken. Change the threshold, add duration or rate logic, or remove it.
• If a static threshold would miss a fast-moving failure (the disk going from 11% to 80% used), use percent change or a derivative. If your metrics system supports it, consider moving averages, confidence bands, or standard deviation.
• If the workload has seasonality, compare current values to the same period a week, day, or hour ago instead of using a fixed number.
• If services still go down without anyone knowing why even though CPU, load, and memory are recorded, you have checkbox monitoring. Go back to defining what 'working' means with the service owner.

• Checkbox monitoring: monitoring exists to satisfy a manager or compliance requirement. The tell is that the service goes down with no explanation and alerts are mostly false alarms.
• Living in the treacherous middle: a mix of actionable and non-actionable alerts that isn't bad enough to distrust outright. The tell is that people skim alerts and sleep and response time suffer.
• Monitoring-as-a-job: one person builds monitoring for systems they don't understand and sets up only the simplest checks. The tell is that other engineers don't own or tune any alerts.
• Static thresholds that miss fast changes: the team gets silence during a real incident and concludes the monitoring is useless. The tell is incidents discovered by users before any alert fires.
• Infrequent checks with no history: checking every five minutes or less often and not storing historical metrics. The tell is that nobody can compare an alert to past behavior, so judgment replaces evidence.

• ▸Alert name: the exact name of the alert as it fires.
• ▸Times fired in last 30 days: count from alert history.
• ▸Action taken each time: write what a person actually did, or 'none'.
• ▸Impact: what would have happened if nobody responded.
• ▸Self-clears? (Y/N): did people say 'it'll clear itself'?
• ▸Requires human action? (Y/N): the keep-or-delete test.
• ▸Decision: delete, retune threshold, redesign (percent change or seasonal comparison), or keep.
• ▸Owner: the engineer who knows the system best, not a monitoring specialist.
• ▸Review date: when this alert is audited again.

1. 1Define an alert before you tune one: an alert evokes urgency and requires action from the person receiving it. Anything that fails this test becomes a log entry, a chat-room message, or an auto-generated ticket.
2. 2Route by response required, using three categories. Immediate action goes to the pager (SMS, PagerDuty, or similar). Awareness without immediate action goes to an internal chat room or a small review webapp. Historical or diagnostic information goes to a log file. Stop sending alerts to email, which overwhelms everyone with noise and does not wake anyone anyway.
3. 3Log every alert so you can report on it. Pull a month of alert history and record, for each alert, what it was, what action was taken, and what the impact was. This shows which parts of the service cause the most trouble.
4. 4Run the disposition questions against each alert: Does it require someone to act? Can it be deleted? Can its threshold be modified? Can the underlying check be redesigned to be more accurate? Can automation make it obsolete?
5. 5Make tuning a daily on-call duty. The person on-call compiles every alert that fired the previous day and asks of each how its signal can be improved or whether it can be deleted. Repeat every day you are on-call.
6. 6Replace low-level alerts with alerts on what 'working' means. Stop paging on OS metrics such as CPU and memory unless you have a specific reason. A high-level check such as HTTP GET / (response code, expected text, latency) is a better leading indicator. Keep the OS metrics for diagnostics.
7. 7Automate the known fix. If the most common response is a documented series of steps, put it in a script and let the monitoring system run it instead of notifying a human. Send an alert only if the auto-healing attempt fails.
8. 8Protect the humans structurally. Use maintenance periods for planned work, but silence narrowly, not with blanket silences. Avoid a standing backup on-call on small teams, since it doubles the on-call load. Keep escalation paths for issues beyond the on-call person's knowledge.

The author managed a few dozen small embedded sensor devices on roofs, in trees, and beside dirt roads. Their SNMP engine regularly went unresponsive while the system kept running, two to three times a week. The fix was to SSH in and restart the sensor. The team needed to know immediately, because the SNMP data was the reason the devices existed, but being woken up for it was tiresome. The author built a check that grabbed any OID and watched for a timeout, while also confirming the device answered ICMP. If SNMP failed but the device was still on the network, a script logged into the sensor and restarted it. A human was alerted only if that attempt did not resolve the problem. The same few-times-a-week page became an automated action, which is the 'automation that makes the alert obsolete' answer to alert fatigue.

• If an alert does not require someone to act, then delete it or demote it to chat, ticket, or log.
• If the usual response to an alert is a known, documented series of steps, then script it and page a human only when the script fails.
• If a metric such as CPU or memory is high but the service still does what it is supposed to, then do not page on it. Alert on what 'working' means instead.
• If you expect planned work to trigger an alert, then put that specific alert in a maintenance period. Do not blanket-silence, because you may miss an unknown dependency breaking.
• If the team is small (the source's example is four people on one-week rotations), then do not add a backup on-call. It puts everyone on-call two weeks a month and speeds burnout. Provide escalation paths instead.

• Sending alerts to email: the inbox fills with noise and people stop trusting the alerts. The tell is alerts that get ignored because 'it'll clear itself up in a few minutes.'
• Sitting in the middle ground between high-signal and low-signal monitoring, with lots of alerts, some actionable and some not. The tell is a response like 'I've seen that alert before' and slowing response times.
• Checkbox monitoring built only to say you have it. The tell is that you consistently ignore alerts because they are false alarms more often than not.
• Treating alert volume as a signal problem when it is a firefighting problem: the alerts are legitimate, but there are dozens a day. The tell is that tuning does not reduce the pages, and irritability, sleep deprivation, and anxiety appear.
• Making everyone always available by adding a backup on top of the primary on-call. The tell is people doing on-call duties (near a computer, sober) for two of every four weeks, plus quicker burnout.

• ▸Alert name: the exact name of the alert as it fires.
• ▸Fire count (past month): how many times it fired, from your alert log.
• ▸Action taken: what the responder actually did each time, or 'none'.
• ▸Impact: what was affected, or 'self-cleared'.
• ▸Required action? (Y/N): does it need a human to act urgently?
• ▸Routing: pager, chat room, or log file, per the three categories.
• ▸Disposition: delete, modify threshold, redesign the check, or automate the fix.
• ▸Auto-heal script (if any): the documented steps to codify and the condition for escalating to a human.
• ▸Owner and review date: who completes the change and when you re-check the count.

1. 1Start from the mission, not the tool. Write down what you need to know and do (e.g., detect a drop in user activity, find a slow query) before evaluating any product. Bejtlich's point: tool-driven teams become captive to the features and limitations of their tools, while mission-driven teams keep looking if requirements aren't met.
2. 2Instrument the business layer first. List your app's core user actions and count each one. Yelp: searches performed, active users, ads purchased, review responses placed. Reddit: users on site, logins, comments posted, threads submitted, votes cast, private messages sent, Gold purchased, ads purchased. These are leading indicators: if search is broken or slow, searches drop.
3. 3Put those business metrics on a dashboard everyone can see, as if on a TV in the office. Let people learn what normal looks like at a glance. These metrics say whether things are going south, not what is wrong, so treat them as the entry signal, not the diagnosis.
4. 4Tie each business metric to the backend services that feed it. When a backend slows, open the dashboard and read the user impact directly instead of wondering about it.
5. 5Add application-layer visibility for diagnosis. For web servers, track requests per second (the one golden metric for throughput and traffic level) plus HTTP response codes from the request log. Watch for a rising count of non-200 responses, especially 5xx. Add request time by changing the NGINX or Apache log format, since it is not included by default.
6. 6Cover database servers with diagnosis in mind. Track connections (threads in MySQL), queries per second as the direct measure of how busy the server is, slow queries from the slow-query log (execution time, count, exact query), replication delay against the configured normal, and IOPS.
7. 7Collect infrastructure metrics with a push-based tool such as collectd, Telegraf, or Diamond rather than SNMP. SNMP is covered under interface metrics, but the passage recommends push tools for easier configuration and more capabilities.
8. 8Test the whole stack by walking one failure down the layers: business metric drops, then app symptom (5xx, slow requests), then infrastructure cause (slow queries, replica lag). If any hop has no signal, that is your gap.

Take the Reddit-style core metrics: logins, comments posted, threads submitted, votes cast. Suppose votes cast falls well below its usual stable level. The business dashboard flags it, but it does not say why, which matches the passage: these metrics signal overall health, not cause. You check web server req/sec and the HTTP response codes in the NGINX log and see a rising number of non-200 responses, such as 500 or 503. Next you check the database: queries per second, the slow query log with execution time and exact query, and replication delay. A slow query would show up as a slow user experience, and an out-of-sync replica is something you want to learn about now rather than days later. Because business, application, and database signals line up, you move from 'something is wrong' to a specific layer in minutes. You can also quantify user impact by reading the drop in votes or comments instead of guessing.

• If you catch yourself saying 'we need to buy X' before defining what you must detect, then stop and write the mission and requirements first; keep looking if a tool doesn't meet them, and build one only when it solves a particular problem effectively.
• If a metric tells you health but not cause (searches performed, logins), then treat it as a leading indicator on the shared dashboard and pair it with application and infrastructure metrics for diagnosis.
• If you need to know how busy a database is, then use queries per second rather than connection count; connections only indicate overall traffic levels.
• If web server request time isn't in your access logs, then update the log format to include it before relying on it.
• If you're about to roll out SNMP for new infrastructure monitoring, then choose a push-based tool like collectd, Telegraf, or Diamond instead.

• Tool obsession: the team's monitoring goals are phrased as product purchases and capability gaps are defined by what the tool can't do. Tell: requirements lists read like feature lists.
• Infrastructure-only monitoring: CPU and disk graphs look healthy while users are failing, because no business metrics exist. Tell: the question 'what is the user impact of this backend slowdown?' cannot be answered from a dashboard.
• Business metrics with no diagnostic path: the TV shows searches dropping but nobody can say which layer failed. Tell: long investigations that start from raw logs every time.
• Misreading database load via connection count. Tell: connections look flat while users report slowness, and nobody has checked qps or the slow query log.
• Silent replication drift: replicas fall out of sync and nothing alerts. Tell: you learn about it days later, from data inconsistencies instead of monitoring.

• ▸Mission statement: one sentence on what you must detect and respond to, written before naming any tool.
• ▸Business metrics: list core user actions counted (e.g., logins, comments, searches) and where they are displayed.
• ▸Normal range per business metric: the stable level you expect, so deviation is visible at a glance.
• ▸Web/app signals: req/sec, non-200 response rate by code group, request time (confirm log format includes it).
• ▸Database signals: connections/threads, qps, slow query log source, replication delay vs. configured normal, IOPS.
• ▸Infrastructure collection method: push-based tool in use (collectd, Telegraf, Diamond) or reason for exception.
• ▸Business-to-backend mapping: for each business metric, the backend services it depends on.
• ▸Gap found in failure walk-through: the layer where a simulated failure produced no signal.
• ▸Owner and fix date: who closes each gap and by when.

1. 1Set the target as a number of nines, not an adjective. Write the availability target per service (99% is two nines, 99.99% is four) and the period it is measured over (SLAs are typically month by month). Track it even if no contract requires it: you cannot improve a reliability number you do not measure.
2. 2Find the ceiling before you promise anything. List every dependency under the service (cloud region, network, database, load balancer). Your service can be no more available than the least available component beneath it. Example: AWS EC2 offers 99.95% for a single region, about four hours of downtime a year, so a single-region AWS service cannot honestly promise more.
3. 3Measure from the user's side first. Put the first checks as close to the user as possible, so downtime is counted when the user experiences it, not when one server looks unhappy. Then add component-level monitoring (web server, database, load balancer backends) to explain why the user number moved.
4. 4Compute availability with a = uptime / total time, where total time is up plus down, over the SLA period. Keep a downtime log with start, end and minutes, so the numerator is auditable rather than reconstructed from memory.
5. 5Match your sampling interval to the precision you claim. To detect a two-minute outage you must collect at minute-long intervals. To claim accuracy down to one second you need sub-second collection. If your checks are slower, report availability only as finely as the data supports; the source says reporting better than 99% accurately is hard for this reason.
6. 6Turn every outage into a fix, not just an alert. Run a postmortem and name the underlying cause (for example, a failing disk that dropped database IOPS, a root volume that ran out of space, a replica silently out of sync). Log the remediation with an owner and date. Do not call the work done when the alert fires; call it done when the cause is removed.
7. 7Prune alerts and graphs as you fix. Monitoring exists to ask questions, so not every collected metric needs an alert. Keep alerts for conditions a human must act on, and keep the rest as graphs you consult during diagnosis.
8. 8Price the next nine before chasing it. Each additional nine costs significantly more, and many customers cannot tell 99% from 99.9%. Raise the target only when users or contracts would notice the difference.

The source's own calculation: an app runs a complete month, 43,800 minutes, and has 93 minutes of downtime. Uptime is 43,800 − 93 = 43,707 minutes, so a = 43,707 / 43,800 ≈ 0.9979, about 99.8% (the source states it as 99.7%). That is between two and three nines. Next, check the data. Per the Nyquist-Shannon point, a check that runs every five minutes cannot reliably resolve a two-minute outage, so the 93 minutes may be undercounted or misattributed. Then check the promise. If this app runs in one AWS region, it sits on a 99.95% dependency (about four hours a year), so a four-nines target is not defensible without changing the architecture. Finally, look at the cause. If the postmortem traces the 93 minutes to a failing disk that cut database IOPS, the reliability work is replacing that disk and alerting on IOPS and replication delay. Buying a new monitoring tool would not address it.

• If a service depends on a component with a published SLA lower than your target, then lower the target or add redundancy. Do not promise above the dependency's number.
• If your check interval is longer than the shortest outage you want to count, then shorten the interval or stop reporting at that precision.
• If the same incident type recurs, then the next action is a fix with an owner and date. Do not add more alerts or evaluate a new tool.
• If customers cannot perceive the difference between your current nines and the next, then do not fund the next nine. Spend on the cause of your worst recent outage instead.
• If a metric has no action attached when it crosses a threshold, then keep it as a graph for questions and do not page anyone on it.

• Tool-swap cycle: every outage triggers a new monitoring evaluation, and six months later the cycle repeats. The tell: reliability numbers are flat while the tool list grows.
• Promising an SLA above your dependencies, such as 99.99% on a single-region EC2 deployment. The tell: the SLA exceeds the lowest SLA in the stack.
• Reporting availability to more precision than the data allows, for example with coarse polling. The tell: short outages are missing from the log and the percentage looks better than users report.
• Treating alerts as the product of monitoring, so the team wakes people for every metric and fixes nothing underneath. The tell: many alerts per week with the same root causes in successive postmortems.
• Chasing 100% or nine nines. The tell: a target with no cost estimate, and no customer who asked for it.

• ▸Service and period: name the service and the measurement window (e.g., one month, in minutes).
• ▸Target nines: the availability percentage you commit to, and whether a penalty clause exists.
• ▸Dependency ceiling: list each dependency with its published or measured availability, and mark the lowest.
• ▸Check source and interval: where the check runs (user side or internal) and how often. Confirm the interval is shorter than the shortest outage you intend to count.
• ▸Total minutes / downtime minutes: enter both, then compute a = (total − downtime) / total.
• ▸Outage log: for each outage, record start, end, minutes, user impact.
• ▸Underlying cause: the postmortem finding for each outage (the failing component or condition, not the symptom).
• ▸Fix and owner: the remediation that removes the cause, who owns it, and the due date.
• ▸Next-nine decision: record the estimated cost of the next nine, whether users would notice it, and a fund / don't-fund verdict.

1. 1Write down the questions the business asks before choosing any metric. The passages give two starting ones: "Is the site up?" and "Are users impacted?" Add the questions your executives actually ask. Their questions differ from those of software and infrastructure engineers, so interview them rather than guess.
2. 2Define each KPI as a metric that measures how the company is doing along lines the company has deemed important to its health. Take the list from the business. Do not derive it from what your tools already collect.
3. 3Instrument at the point where users touch the app first. Start with HTTP response codes, especially 5xx, then request time (latency). These do not say what is wrong. They say that something is wrong and that users are affected, which is the business-impact signal.
4. 4Put an impact test on every deeper metric. For each web-node, worker-node, or database metric, ask: "How will this metric show me the user impact?" If you cannot answer, graph it for diagnosis and do not treat it as a business signal. A database CPU spike with no user impact is not necessarily a problem.
5. 5Track availability even when no contract requires it. Knowing your own reliability is what lets you improve it and report it in terms the business understands.
6. 6Put a dollar figure on the monitoring itself. Use fully loaded FTE cost, engineer count, build weeks, and monthly maintenance hours. Add opportunity cost: time spent on a monitoring platform is time not spent on revenue-generating work. Compare the total with the SaaS price. This is how you show the business what monitoring costs and returns.
7. 7Keep tools that correlate. Several tools are fine if each adds different information. Consolidate only where tools duplicate each other or cannot correlate their data, because that is where visibility into business health breaks.
8. 8Do not give every graphed metric an alert. Alert on what signals user or business impact, and leave the rest as graphs for investigation.

The source's build-versus-buy comparison shows how monitoring is expressed in business terms. Inputs: an FTE costs $150,000 (salary, benefits, overhead), 3 engineers work for 4 weeks to build a decent solution (reliable, scalable, documented, with training), and maintenance takes 20 hours a month. The result is about $35,000 in raw engineering time to build, plus roughly $18,750 a year to maintain. That is about $53,750 in year one, before opportunity cost. Opportunity cost is three engineers spending a month on something that generates no revenue. The source says to judge it by role: network engineers at a company that does not manage networks for customers are probably less tied to revenue than software engineers at a SaaS company. A good SaaS monitoring product costs most companies $6,000 to $9,000 a year. Presented this way, an executive can compare the two options and see what the money buys. The $35,000 and $18,750 figures are the source's rough estimates, so replace them with your own numbers.

• If a metric spikes but user-facing 5xx rates and latency are normal, then treat it as a diagnostic signal rather than a business-impact problem, and do not page on it.
• If you cannot say how a proposed metric would show user impact, then do not put it on the business-facing view.
• If three tools report the same database information, then consolidate. If they report different information, then keep them.
• If tools cannot correlate their data, then fix that first, because fragmentation hurts business visibility only in that case.
• If you must justify a monitoring spend, then compare fully loaded build cost, maintenance cost, and opportunity cost with the SaaS price. Use your own judgment on opportunity cost, since it varies by role.

• Starting from infrastructure internals. The tell: your first dashboards show node CPU and worker counts, and nobody can say whether users are affected.
• Hunting for a single-pane-of-glass tool to deliver business visibility. The tell: tool evaluations repeat every six months with the same complaints.
• Measuring what engineers care about instead of the company's KPIs. The tell: executives never open the dashboards, or ask questions the monitoring cannot answer.
• Costing only the build. The tell: the in-house estimate leaves out maintenance hours, documentation, training, and opportunity cost, so it looks cheaper than the $6,000 to $9,000 a year SaaS option.
• Alerting on every graphed metric. The tell: pages fire for conditions with no user impact, which hides the signals that matter to the business.

• ▸Executive question: write one question the business asks (for example, "Are users impacted?").
• ▸Company KPI: name the metric the company has deemed important to its health that answers this question.
• ▸User-facing signal: list the HTTP 5xx rate, latency, or other measure taken where users interact with the app.
• ▸Impact link: for each deeper metric, write how it would show user impact. Mark any you cannot answer as diagnostic only.
• ▸Availability tracking: record where availability is tracked and who sees it.
• ▸Alert decision: write yes or no for whether this metric warrants an alert.
• ▸Cost inputs: record FTE cost, engineer count, build weeks, and monthly maintenance hours.
• ▸Opportunity cost note: state which revenue-generating work the engineers would give up, using your best judgment.
• ▸SaaS comparison: record the annual SaaS price and the resulting build-versus-buy difference.

1. 1Name the five facets of your current monitoring: data collection, data storage, visualization, analytics and reporting, alerting. Even a monolith like Nagios covers all five inside one tool, so map each facet to the product that handles it today.
2. 2Mark the facets where one tool does the job badly or locks you in. Those are the first candidates to split out. Leave the facets that work alone.
3. 3Pick a specialized tool for one facet at a time. The passages name Graphite, Sensu, logstash, and collectd as examples. Commercial services with extensive APIs (Librato, Loggly, Pingdom) also qualify, because the API is what lets you couple them loosely.
4. 4Choose a collection model per target, not globally. Use pull where you are stuck with it (SNMP for network gear, a polled /health endpoint for apps) and push or metrics-based collection elsewhere. The author says the push/pull decision isn't that important: use what works and move on.
5. 5Couple the tools through their interfaces (APIs, structured formats such as JSON). Emit structured logs and return structured data from health endpoints so any downstream tool can parse them. Avoid unstructured formats.
6. 6Replace one component at a time when it stops serving you. The payoff of composability is that you swap a single tool instead of the whole platform. Record each swap and the reason for it.
7. 7Accept the added architectural complexity as a deliberate trade, and check that data from different facets can still be correlated (for example, a log entry turned into a metric, or a metric alongside a health check).
8. 8Schedule a review every two to three years. The author expects you to rearchitect monitoring on that cadence as needs and the industry change, and composability makes that affordable.

The passages trace how composable monitoring started. In 2011, the #monitoringsucks hashtag on Twitter drew complaints about monolithic tools, chiefly Nagios. The conversation became #monitoringlove and the founding of the Monitorama conference in Boston. One of the biggest points raised was that the field needed new, more specialized tools. Graphite, Sensu, logstash, and collectd followed, and tying them together produced a more flexible, less painful stack. The same logic applies to your own stack. Suppose your log analysis tool can no longer keep up. Under a monolith you would replace the entire platform. Under a composable design you remove the logging component and swap in another, while collection, storage, and alerting continue to run. The passages describe this as the main perk: you are no longer committed long-term to a particular tool. The cost is a more complex architecture, which the author judges to be far outweighed by the benefits.

• If a single tool handles all five facets adequately and your needs are stable, keep it, but still check that it exposes an API so you can replace pieces later.
• If one facet is the source of most of your pain, replace only that facet with a specialized tool instead of migrating the platform.
• If you monitor network gear, use SNMP-style pull for it and don't force a push model onto it. If you monitor an application, prefer a /health endpoint or metrics emission, whichever your team can support.
• If a candidate tool has no API or only emits unstructured data, treat it as a monolith in disguise and avoid it.
• If monitoring has been stable for two to three years, expect to rearchitect anyway and review which components still fit.

• Buying a specialized tool with no usable API, so it can't be loosely coupled. The tell: swapping it out means rebuilding the integrations around it.
• Splitting every facet at once and ending up with unmanageable complexity. The tell: nobody can say which tool owns which of the five facets.
• Treating the push-versus-pull debate as the main decision. The tell: weeks spent on think-piece arguments while nothing ships. The passages say to use what works and move on.
• Producing unstructured logs and data, which blocks correlation across tools. The tell: you can't turn log entries into metrics or line them up with other data.
• Building a custom platform too early. The tell: you are writing in-house tooling without having outgrown SaaS or FOSS tools such as Graphite, InfluxDB, Sensu, or Prometheus.

• ▸Facet: data collection, data storage, visualization, analytics and reporting, or alerting.
• ▸Current tool: which product handles this facet today.
• ▸Monolith or specialized: is it one tool covering several facets, or a dedicated tool.
• ▸Integration interface: API or data format (e.g., JSON) used to connect it to the other facets.
• ▸Collection model: push or pull, with the reason (e.g., SNMP for network gear).
• ▸Pain or limit: what this tool no longer does well.
• ▸Replacement candidate: the specialized tool you would swap in.
• ▸Swap cost: what else must change if this component is replaced.
• ▸Next review date: when you will re-check fit (every two to three years at most).

1. 1List every point where a user touches the app: web front end, API endpoints, checkout or signup flows. Instrument these first, before any node, database or worker. The source's rule is to start 'as close to the user as possible', not 'deep in the bowels of the infrastructure where most people start'.
2. 2Collect HTTP response codes at each touchpoint and separate out the 5xx class. The source calls 5xx 'one of the most effective things to monitor'. Track the 5xx count and its share of total responses over time, per endpoint or service, so one failing service doesn't hide inside a site-wide average.
3. 3Add request time (latency) second. If you can, store it so you can read percentiles, because the source points to percentile math for latency in its footnotes. Treat latency and 5xx as symptom detectors only: they say that something is wrong and that users are affected, not what is wrong.
4. 4Write down the user questions the data must answer: 'Is the site up?' and 'Are users impacted?' Chapter 5 names these as the questions people actually ask. Check that each question has at least one graph that answers it directly.
5. 5Add business KPIs next to the technical signals, starting with the ones the company has already declared important to its health (a KPI is a metric on 'lines the company has deemed important'). Put them on the same dashboard so a latency rise and a business-metric change can be read together.
6. 6Build one dashboard per service or product, with a high-level overview on top. Have the people who understand the service best create and maintain it, as the source suggests (email admins build the email dashboard). Use line graphs for time series. Avoid pie charts, which show no history or trend.
7. 7Only after the user layer works, expand inward to web nodes, worker nodes and databases. Test every new internal metric with the source's question: 'How will these metrics show me the user impact?' If you can't answer it, collect the metric for investigation but don't alert on it.
8. 8Decide alerting separately from collection. The source says monitoring is for asking questions and that not every metric needs an alert. Alert on user-impact signals (5xx, latency), and keep internal metrics as diagnostic graphs unless they predict user impact.

The source's own scenario: your database server's CPU starts to spike. In a node-first setup, that pages someone. In a user-first setup, you check the user-facing signals first. Look at the 5xx rate on the web tier: if it is flat, users are getting correct responses. Look at request latency: if it is flat, users are not waiting longer. Then look at the business KPIs: if they are steady, nothing is hurting the business. The source's question applies: 'If your database server's CPU has started to spike, but the user isn't impacted, do you really have a problem?' You keep the CPU graph for investigation but don't wake anyone. Now reverse it. 5xx responses climb and latency rises, but no single node looks obviously broken. The user-layer metrics tell you something is wrong and users are affected, though not what. You then use the component dashboards (web nodes, worker nodes, database) to find the cause. The user layer detects, and the infrastructure layer diagnoses.

• If a metric moves but 5xx rate, latency and business KPIs are unchanged, then treat it as a diagnostic graph and don't alert on it.
• If 5xx responses or latency degrade, then treat it as user impact right away and open the component dashboards to find the cause. The user-layer signal won't tell you what is wrong.
• If you want to add a new internal metric, then require a one-sentence answer to 'How will this show me user impact?' before you collect or alert on it.
• If you are choosing between deep infrastructure coverage and user-touchpoint coverage, then finish the user touchpoints first, then expand to web and worker nodes.
• If you are setting availability targets, then weigh the cost of each extra nine. Many customers can't tell 99% from 99.9%, so don't promise more than users will notice.

• Starting with node-level CPU, memory and disk graphs because that is where most people start. The tell: you can describe every server's health but can't answer 'Are users impacted?' without logging in somewhere.
• Alerting on every collected metric. The tell: pages fire for internal spikes (like database CPU) while users see nothing, and on-call stops trusting alerts.
• Reading 5xx and latency as diagnoses. The tell: the team treats a rising error rate as a root cause and starts restarting nodes instead of drilling into component dashboards.
• Stopping at the user layer. The tell: you can see that users are hurt but have no web-node, worker or database metrics to find out why. The source says to expand quickly and go as deep and wide as needed.
• Using one sprawling dashboard, or pie charts, for health. The tell: nobody can tell at a glance which service is degrading, and the chart shows a snapshot with no history or trend.

• ▸Service/product: name the single service or app this map covers.
• ▸User touchpoints: list each place users interact with it (pages, API endpoints, flows).
• ▸User questions: write the questions this must answer (e.g., 'Is the site up?', 'Are users impacted?').
• ▸HTTP response signal: note where 5xx counts and rates are collected, per touchpoint.
• ▸Latency signal: note where request times are collected and how you will read them (e.g., percentiles).
• ▸Business KPIs: list the company-defined KPIs to show on the same dashboard.
• ▸Alert on: state which user-impact signals page someone, and the condition for each.
• ▸Internal components to add next: list web nodes, workers, databases, each with the sentence 'This shows user impact by...'.
• ▸Dashboard owner: name the person who understands the service best and maintains the dashboard.

1. 1Inventory every alert that pages a human and tag it as either a symptom of 'working' (user-facing behavior) or a low-level OS metric (CPU, memory, system load). The source's rule: unless you have a specific reason to alert on OS metrics, stop doing it. Keep collecting them for diagnostics and trend analysis, but remove the page.
2. 2Build one high-level check per service that defines 'working'. For a webapp, run HTTP GET / and record the response code, expect HTTP 200 OK, expect specific text on the page, and record request latency. Alert on these four signals first. They are leading indicators that something is wrong, not diagnoses.
3. 3Add more specific checks and alerts only as you learn the service. Each new alert should come from an observed failure mode, not from a checklist or a compliance requirement.
4. 4Run an ignored-alert audit. If people consistently ignore an alert because it is a false alarm more often than not, treat it as the checkbox-monitoring symptom. Fix its threshold or delete it. Noisy alerts make monitoring untrustworthy, which the source calls probably worse than no monitoring.
5. 5Check collection resolution against the failures you need to catch. Checking every five minutes or less often is a checkbox-monitoring sign. A latency spike every 30 seconds disappears at five-minute resolution, so an alert on that data cannot fire. Increase collection frequency for alerting metrics.
6. 6Keep historical metric data. Not storing it is a listed sign of checkbox monitoring. History is what lets you judge whether a threshold fits the service's normal behavior instead of copying a static number.
7. 7For each alert, write down the manual steps a responder takes first. If those steps can be scripted, have the monitoring tool run them before it alerts you, and page only if they fail. The source says to automate these steps and execute them before alerting.
8. 8Wire alerts into incident handling: monitoring identifies the problem and automatically opens a ticket. Route pages through an on-call tool (PagerDuty, VictorOps, OpsGenie) so escalation paths, schedules, and incident records exist for later review.

A new engineer tells her lead that CPU on a MySQL server is high and asks what to do. The lead asks, 'Is the server still doing what it's supposed to?' It is, with acceptable response times. 'Then there's not really a problem, is there?' The CPU alert gets removed from paging, and the metric stays on a dashboard for diagnostics. In its place, the team sets up the webapp's HTTP GET / check. It records the status code, expects a 200, expects specific text on the page, and records latency. Later, latency climbs while the response stays 200. That signal alone says something may be wrong, and CPU never needed to cross a line. In another failure, the 200 comes back but the expected text is missing, which also flags a problem. Both alerts describe user-visible 'working', and neither pages someone for a resource-hungry service that is behaving normally.

• If a metric is a low-level OS measure (CPU, memory, load) and the service still behaves correctly, then do not page on it. Keep it for diagnostics and trending, and alert on what 'working' means instead.
• If an alert is a false alarm more often than not and people ignore it, then fix or delete it. Leaving it in place trains the team to ignore real pages.
• If the first response to an alert is a repeatable manual step, then automate it and have the monitoring tool run it before paging a human.
• If a failure can occur and clear within less than your collection interval (a spike every 30 seconds against five-minute checks), then shorten the interval before trusting any alert built on that metric.
• If a new alert has no specific reason or observed failure behind it, then do not add it. Start with the high-level check and add specifics as you learn the service.

• Alerts exist to satisfy a requirement or compliance rule (checkbox monitoring). The tell: metrics are recorded, yet the service goes down without anyone knowing why.
• CPU or memory thresholds page people at night for services that are fine. The tell: responders check the service, find it healthy, and go back to sleep.
• Alerts are consistently ignored as false alarms. The tell: pages are acknowledged without investigation, and real incidents are discovered late.
• Metrics are collected every five minutes or less often, so short spikes never trip alerts. The tell: users report latency problems that no alert or graph shows.
• No historical metric data is kept (the Nagios complaint). The tell: nobody can say what normal looks like, so thresholds are guesses and cannot be tuned.
• Responders repeat the same manual fix on every page. The tell: the same steps recur across incident tickets and are never automated.

• ▸Alert name and service: name the alert and the service it protects.
• ▸What 'working' means: state the user-facing behavior this alert detects (status code, expected text, latency), not an OS metric.
• ▸Signal type: label it symptom of 'working' or low-level OS metric. If OS metric, write the specific reason it is worth waking someone.
• ▸Threshold and basis: state the threshold and what historical data supports it, rather than a bare static number.
• ▸Collection interval: give the check frequency and confirm it is short enough to catch the failure you care about.
• ▸Auto-remediation tried first: list the scripted steps the monitoring tool runs before paging, or write 'none' and why.
• ▸Responder first steps / runbook link: record what the on-call person does first, in writing.
• ▸Last-90-days fire record: count fires and how many were false alarms. If false alarms exceed real ones, mark 'tune or delete'.
• ▸Ticket and routing: confirm a ticket is opened automatically and name the on-call escalation path.

1. 1Audit how monitoring changes get made today. Count the requests shaped like 'Can you add this to monitoring?' over the last month. Each one is a manual dependency to remove, and the count is your baseline.
2. 2Switch from per-host thinking to per-class thinking. In cloud architectures you monitor entire classes of things and analyze aggregates of groups, so define checks against a role (all web servers), never against an individual machine.
3. 3Make service registration automatic. Choose a mechanism where nodes self-register and deregister (the source names Sensu for instant self-registration and deregistration), or drive Nagios from config management. The test: a new server appears in monitoring with no human action.
4. 4Time how long it takes to add a new check across the whole fleet. The source's benchmark is a few minutes for every web server. If it takes longer, fix the configuration path before adding more monitoring, because slow configuration makes you stop bothering.
5. 5Handle short-lived servers explicitly. Servers that exist for hours or minutes break static tooling, so confirm that deregistration is as automatic as registration and that departed nodes do not linger as alerts.
6. 6Review every runbook. If it is a plain list of steps ('run this command, check this information, run this other command'), it signals missing automation. Script those steps.
7. 7Wire the scripted steps into the monitoring tool so they run before a human is alerted (automated self-healing first). Page a person only if the automated remediation fails.
8. 8Re-review alert history monthly with one question: what automation can make this alert obsolete entirely? Delete, retune, or automate each recurring alert, and keep maintenance periods narrow rather than issuing blanket silences.

The source describes a pattern from the author's consulting work: in most monitoring implementations he is called into, the team spends more time on configuration than on monitoring. Every new check or node is a manual task, so building better monitoring becomes frustrating and, after a while, people stop bothering. The fix is to push configuration into automation. Self-registering nodes (Sensu) or Nagios coupled with config management mean a new web server appears in monitoring on its own. Adding a check for every web server takes a few minutes, so the team is no longer hesitant to add more. The same logic applies to remediation. An alert whose runbook is just 'run this command, check this information, run this other command' becomes a script the monitoring tool executes first, and it pages a human only if the script does not resolve the problem. The team's time shifts from maintaining configuration to improving what is monitored.

• If someone asks 'Can you add this to monitoring?', treat it as a defect in your process and automate that path rather than just fulfilling the ticket.
• If your infrastructure is cloud-based or its servers live for hours or minutes, require 100% automated registration and deregistration. Manual registration cannot keep up.
• If a runbook is only a sequence of commands, automate it and run it from the monitoring tool before alerting a human.
• If adding a check across the fleet takes more than a few minutes, fix the configuration tooling before adding more checks.
• If an alert keeps recurring, ask what automation would make it obsolete before you only tune its threshold.

• Automating only part of the chain: nodes self-register but checks are still hand-written. The tell is that coverage lags behind new services and the team spends its time on configuration.
• Static, per-host configuration in a dynamic fleet. The tell is alerts for servers that no longer exist, or new servers with no monitoring at all.
• Leaving runbooks as step lists. The tell is on-call staff executing the same documented commands on every page.
• Blanket silencing to cope with noise or maintenance. The tell is that unknown dependency failures go unnoticed, hiding things the source says you would want to learn.
• Adding more monitoring to a finicky service instead of stabilizing it. The tell is a growing check count around the same recurring failure.

• ▸Monitored item/class: name the group (e.g., web servers) rather than an individual host.
• ▸Registration method: self-registration tool or config management, or 'manual'.
• ▸Deregistration method: how departed or short-lived nodes are removed automatically.
• ▸Time to add a new check fleet-wide: measured minutes; target is a few minutes.
• ▸Manual-request count last month: number of 'add this to monitoring' asks.
• ▸Alert with step-list runbook: name the alert and list the steps.
• ▸Automated remediation script: what runs before paging, and its owner.
• ▸Escalation condition: when the script fails and a human is paged.
• ▸Obsolescence review: which recurring alerts could automation remove, and the next review date.

1. 1Name the failure first: list every service or system where one person (or one role) is the only one who creates alerts, dashboards, or checks. That list is your Anti-Pattern #2 exposure, and it is the work queue.
2. 2Assign instrumentation to the people closest to each system. Network engineers define what to monitor in the network and where its hot spots are. Software engineers design the monitoring for the applications they write. Do not route these requests through one monitoring specialist.
3. 3Start software engineers with small, concrete app metrics rather than an APM purchase or a specialist project: time database queries, time external vendor API responses, count logins per day. These are easy and quick, and the passages say instrumenting becomes addictive once begun.
4. 4Make monitoring a first-class part of building a service and enforce the gate: it is not ready for production until it is monitored. Add this as a check in your release or readiness process, with the owning team named.
5. 5Make each team responsible for the alerts on its own services, not just the dashboards. Whoever builds the service creates its alerts and responds to them, consistent with the DevOps tenet that everyone is responsible for production.
6. 6If you have a central observability team, scope it to building and providing self-service monitoring tools. Write down that it is not responsible for instrumenting applications or creating alerts. Service teams own those.
7. 7Teach the shared skill by raising baseline literacy across the team. Make sure everyone can read basic host signals such as the CPU utilization line in top (us + sy + ni + hi + si; wa and st excluded) and the second row of free (used minus buffers/cache), so they can judge whether their own alerts are meaningful.
8. 8Audit for checkbox-monitoring symptoms in each team's setup: service goes down without explanation, alerts routinely ignored as false alarms, five-minute-or-slower checks, no stored history. Each symptom points to an owner who does not understand the system, so fix it by moving ownership to the people who do.

The author describes working at a large enterprise with fully specialized roles: one person for log collection, one for Solaris servers, and one whose job was to create and maintain monitoring for all of it, and he was that third person. The problem he names is blunt: how can you build monitoring for a thing you don't understand? The remedy is to treat monitoring as a skill everyone has to some degree, like config management or database administration. Applied here, the network engineers would define what to watch in the network, and software engineers would instrument their own applications, starting with query timings, vendor API response times, and daily login counts. A small observability team could still provide the self-service tooling, but it would not instrument apps or write alerts. The expected result per the source is more robust monitoring with a better signal-to-noise ratio, and likely better signal than before. The same source links the opposite arrangement (a single person setting up monitoring without understanding the system) to checkbox monitoring, where the simplest possible checks get set up.

• If one person or role is the only one who writes alerts or instruments services, then redistribute those tasks to the engineers who build and run each service, and keep the specialist as an advisor.
• If you are deciding who designs monitoring for an application, then choose its software engineers, because they know the application better than anyone else.
• If a team proposes a central group, then permit it only as a self-service tooling provider (an observability team); it must not own instrumentation or alerts.
• If a service is headed to production without metrics and alerts, then treat it as not ready; do not ship and promise to monitor later.
• If engineers say app instrumentation is too hard or needs a specialist, then start with one simple metric (database query time, vendor API latency, or logins per day) rather than a large initiative.

• Monitoring-as-a-Job: one person owns monitoring for everything. The tell: that person cannot explain how some of the systems they monitor actually work, and others say 'ask them' about any alert.
• Rebranding the observability team as the owner of everything: the tooling team also instruments apps and writes alerts. The tell: software engineers never touch metrics code, and the company has effectively rested monitoring on a few shoulders.
• Checkbox monitoring from non-owners: monitoring exists to satisfy a requirement or compliance rule. The tell: the service still goes down without anyone knowing why, alerts get ignored as false alarms, checks run every five minutes or less often, and no historical metrics are stored.
• Applications left as a black box: server, security, and network monitoring are strong while apps have none, often because teams assume it needs specialized skills. The tell: you can see CPU and memory but cannot say how long queries or vendor calls take.
• Shared skill without shared literacy: engineers read host numbers wrongly, such as treating the first row of free as 'used' memory when buffers and cache are reclaimable. The tell: alerts fire on memory 'usage' that the second row shows is available.

• ▸Service / system: name of the thing being monitored.
• ▸Owning team: the team that builds and runs it (not the monitoring specialist).
• ▸Domain expert(s): the engineers closest to it, such as the network or application engineers.
• ▸Instrumentation owner: who adds metrics to the app or system, and the first metric chosen (e.g., DB query time, vendor API latency, logins per day).
• ▸Alert owner: who creates and responds to alerts for this service.
• ▸Production-readiness check: yes/no — is it monitored before it ships?
• ▸Central tooling dependency: which self-service tools from an observability team it uses, if any.
• ▸Checkbox-monitoring symptoms present: unexplained outages, ignored alerts, slow check interval, no stored history.
• ▸Next action and date: one small step to move ownership to the closest engineers.