Measure And Run Technology Operations

1. 1Classify each candidate metric into one of the three types before designing it: Technology metrics (component/application performance, availability), Process metrics (CSFs, KPIs, activity metrics for service management processes), and Service metrics (end-to-end IT service results computed from component metrics). This tells you what layer the metric governs and prevents mixing a component reading with a service-level claim.
2. 2Anchor every KPI to a Critical Success Factor and to one of the four questions KPIs answer — quality, performance, value, or compliance. If a proposed KPI does not map to a CSF and one of those four, drop it; it is noise, not measurement.
3. 3Make each metric SMART by specifying the exact quantity, direction, baseline, and timeframe. Follow the qualitative pattern from the passages: CSF = 'Improving IT service quality'; KPI = '10% increase in customer satisfaction rating for handling incidents over the next 6 months.'
4. 4List the raw data required to compute the KPI before you commit to it — for the incident satisfaction example you need the original customer satisfaction score, the ending score, and the collection dates. If you cannot name the source data, the metric is not yet operational.
5. 5Distinguish specialized from generic metrics. Specialized metrics are valid only within their domain and cannot be benchmarked or interpreted outside it; only generic metrics aggregate across subject areas or business units. Label each metric accordingly so no one benchmarks a domain-bound number against an unrelated unit.
6. 6Align the metric set to strategic objectives that start broad (financial and organizational performance for commercial providers) and cascade down. Confirm each KPI ultimately traces to a business imperative that depends on service management, mirroring the ROI business-case requirement.
7. 7Build the dashboard so it answers a decision, not decorates a wall — use the supporting templates (Examples of Service Quality Metrics, Availability Reporting, KPIs of the Value of Service Management, High Level Goals and KPIs) as starting structures and adapt them to your bespoke context.
8. 8Test each metric for perverse incentives against customer expectations: since a defect is 'anything outside of customer expectations,' a metric that looks good internally but sits outside what the customer expects is measuring the wrong thing.

Take the qualitative example from the passages. The CSF is 'Improving IT service quality.' The KPI is '10% increase in customer satisfaction rating for handling incidents over the next 6 months.' This passes the SMART test: it names the metric (customer satisfaction for incident handling), the magnitude (10%), the direction (increase), and the timeframe (6 months). The required data is explicit — the original customer satisfaction score for handling incidents and the ending customer satisfaction score — so computation is unambiguous. Because it maps directly to the 'quality' question that KPIs answer and to a named CSF, it belongs on the dashboard. On a dashboard you would show the baseline score, current score, the 10% target line, and days remaining in the 6-month window, so a manager can see at a glance whether intervention is needed.

• If a KPI cannot be traced to a CSF and to one of quality/performance/value/compliance, then remove it from the dashboard.
• If you cannot name the raw data needed to compute a metric, then it is not yet SMART — defer it until the source data is defined.
• If a metric is specialized (domain-bound), then never benchmark or interpret it across other units; if it is generic, then aggregation across business units is permitted.
• If a metric can improve while the customer experience worsens (falls outside customer expectations), then it is perverse — redesign it around the customer's expectation.
• If a proposed measure does not trace to a broad business objective (financial/organizational performance) or a business imperative depending on service management, then it fails strategy alignment.

• Reporting a component/technology metric as if it were a service metric. The tell: a green 'availability' number while users still report the service as down — you measured a component, not the end-to-end service.
• Writing vague CSFs with no quantified KPI. The tell: a dashboard label like 'improve quality' with no baseline, target percentage, or timeframe attached.
• Benchmarking a specialized, domain-bound metric against an unrelated unit. The tell: comparisons that provoke 'that's not apples to apples' objections because the metric was never valid outside its domain.
• Chasing short-term ROI-style numbers that ignore business impact. The tell: metrics optimize a quarterly figure while the strategic service imperative is unaddressed, and the number 'either meets or does not meet' with no context.
• Designing metrics around internal convenience rather than customer expectations. The tell: internal scores rise while complaints and low survey results (customer-related performance) rise in parallel.

• ▸Metric type: Technology / Process / Service — pick the layer this metric governs.
• ▸Linked CSF: the Critical Success Factor this metric supports.
• ▸KPI question served: Quality / Performance / Value / Compliance.
• ▸SMART statement: quantity + direction + baseline + timeframe (e.g., '10% increase ... over 6 months').
• ▸Required raw data: list each source input needed to compute it (e.g., original and ending scores).
• ▸Scope: Specialized (domain-only) or Generic (aggregatable) — state benchmarking limits.
• ▸Strategy trace: the broad business objective or service-management imperative it supports.
• ▸Perversity check: could this improve while customer expectations are unmet? Yes/No + mitigation.
• ▸Dashboard element: what is displayed (baseline, current, target line, time remaining) and the decision it drives.

1. 1Separate your data pipeline into the three functional subsystems the source names: a feature system (produces characteristics of input data), a human annotation system, and a metadata system. Assign clear ownership to each so failures are traceable.
2. 2Distinguish facts from features before you build. Store raw facts (immutable records of what happened) separately, then derive features by encoding on top of them; this lets you re-derive features without re-collecting data when you change feature logic.
3. 3Design collection and governance up front, not after the model works. Treat data collection practices, governance, and data quality as first-class integration concerns alongside UI/UX and business goals, since the source lists them as co-determinants of ML product success.
4. 4Balance the four Vs—volume, velocity, variety, validity—for your specific business context rather than maximizing all four. Decide which dimension dominates (e.g., validity for a spoilage-sensitive pipeline) and provision infrastructure accordingly.
5. 5Build for repeatability: because ML pipelines fail in ways that are hard to detect and hard to measure, instrument every stage so a corrupted file, stuck data, or subtly changed input surfaces as an alert, not a silent quality regression.
6. 6Centralize metadata so every run's data version, features, and lineage are recorded. This is what makes model validity checks (will the new model break the system?) answerable before you flip the switch to production.
7. 7Practice scalability pragmatically: start with vertical scalability (a bigger machine running existing scripts) and only reengineer for horizontal scale when realistic near-term load would collapse the pipeline. Don't divert quality work into premature distributed computing.
8. 8Gate any model swap on two questions the source sets out—validity (does it break the system?) and quality (is it any good?)—using robust, documented processes and cross-functional review rather than a single engineer's judgment.

Consider Alex, who runs data-science scripts on a laptop, and Bowie, who worries the Python scripts won't handle arbitrarily large data. Both concerns are valid: Bowie is right from the engineering side (laptop scripts break under growth), and Alex is right that the scripts are adequate for today and the near future while quality of results matters more. A wise leader (Harper) takes the balanced path: make the code just scalable enough to survive realistic near-term loads, then let Alex spend remaining time ensuring the validity and quality of the data and results. This mirrors the dairy analogy—the farmer whose milk spoils (loses validity) or reaches an oversupplied market shows that volume and velocity are worthless without validity. The lesson for data management: don't over-invest in scale before the data and results are correct, and don't ignore scale until it collapses the pipeline.

• If the same data has to feed multiple derived characteristics, then store facts once and encode features on top, so feature changes never force recollection.
• If a model is being swapped into a critical production setting, then answer both validity (will it break the system?) and quality (is it any good?) through a documented process before flipping the switch.
• If your business tolerates no bad or spoiled results, then prioritize the validity dimension over volume and velocity when provisioning infrastructure.
• If the existing scripts handle today's and near-term data, then favor vertical scalability (bigger machine) over reengineering, reserving effort for data quality.
• If a pipeline failure is hard to detect or measure, then add explicit instrumentation and metadata tracking at that stage rather than trusting silence.

• Treating ML as pure algorithm work and neglecting data collection and governance—the tell: the model trains fine but the product fails to integrate with the rest of the business.
• Assuming data and systems are reliable; disks fail, files corrupt, APIs error, and data changes subtly—the tell: model accuracy quietly degrades with no alert firing.
• Maximizing scale prematurely and diverting attention from result quality—the tell: hardened distributed infrastructure serving a model whose outputs were never validated.
• Conflating facts and features so feature logic changes require full data recollection—the tell: every feature tweak triggers an expensive re-ingest.
• Swapping a new model into production without answering validity and quality separately—the tell: the new model breaks the surrounding system even though its own metrics looked good.

• ▸Subsystem owner: name the owner for feature, annotation, and metadata systems
• ▸Fact source: where immutable raw records are stored and how they are versioned
• ▸Feature encoding: how each feature is derived from facts and where the logic lives
• ▸Dominant V: which of volume/velocity/validity/variety governs this use case and why
• ▸Failure instrumentation: what alert fires for corrupted, stuck, or subtly changed data
• ▸Scalability stance: current vertical/horizontal approach and the load that would trigger reengineering
• ▸Validity check: the documented test that answers 'will the new model break the system?'
• ▸Quality check: the metric and reviewers that answer 'is the new model any good?'

1. 1Before flipping any switch to production, split evaluation into two distinct questions and answer them separately: model validity (will the new model break our system?) and model quality (is the new model any good?). Do not conflate them—a model can be high quality but still break the serving system.
2. 2Establish validity first: verify the new model is compatible with the serving path, produces well-formed outputs, and can replace the currently running version without downstream breakage. Treat this as a gating check that a poor-quality but valid model still passes.
3. 3Establish quality second: measure offline accuracy against a held-out evaluation set and compare against the currently running production model as the baseline, not against zero. The question is whether the new model is better than what it replaces.
4. 4Treat model development as '1% inspiration and 99% verification'—budget the overwhelming majority of effort toward verifying the model, not toward the algorithmic novelty that produced it. Assume the most exciting algorithm this year is next year's mundane executable.
5. 5Convene the right expertise: validity and quality investigations 'may require deep investigation' and 'collaboration among folks with various areas of expertise'—pull in data scientists, business analysts, and ML engineers rather than letting one developer self-certify.
6. 6Codify this as a robust, repeated organizational process rather than an ad hoc check, because ML applications are 'never really done' and every model update re-enters the same loop and must clear the same validity/quality gates.
7. 7When quality is below the threshold, route to the improvement loop: understand the failures, then modify the training pipeline—changing features, adding data—rather than shipping and hoping.

A team has a model running in production and a data scientist declares an updated version 'ready to go.' Before swapping it in, they run the validity question first: does the new model return outputs in the shape the serving system expects, and will substituting it break anything downstream? It passes—the plumbing holds. They then run the quality question against a held-out evaluation set, comparing the new model's offline accuracy directly to the currently running model rather than to an absolute bar. The new version does not clear the accuracy threshold, so instead of flipping the switch, the data scientists, business analysts, and ML engineers collaborate to understand the failures. They modify the training pipeline—changing some features and adding data—and re-enter the evaluation loop, treating this iteration as expected rather than as a setback, since ML applications are never really done.

• If the new model breaks or malforms the serving system, then it fails on validity and never reaches the quality question—stop regardless of accuracy.
• If validity passes but offline accuracy does not beat the currently running model against the threshold, then do not deploy; send it back to the improvement loop.
• If the system-level behavior must be consistent, robust, and reliable independent of model type, then evaluate the whole system's behavior, not just model-only metrics.
• If a single developer wants to self-certify 'ready to go,' then require cross-functional review because these questions 'may require deep investigation' and multiple areas of expertise.
• If you are tempted to spend your budget chasing another accuracy percentage point, then reallocate toward verification—development is 99% verification.

• Conflating validity and quality: shipping a high-accuracy model that breaks the serving path, or blocking a valid model over quality concerns that belong in the improvement loop. Tell: the go/no-go discussion mixes 'it's more accurate' with 'it doesn't crash' in one undifferentiated verdict.
• Grading against an absolute bar instead of the running production model. Tell: nobody in the review can state the current model's baseline numbers.
• Letting the model developer self-declare 'ready to go' with no independent verification. Tell: the only evidence is the developer's assertion, with no cross-functional sign-off.
• Under-investing in verification because the algorithm work feels more exciting. Tell: elaborate model novelty paired with a thin, hurried pre-deployment check.
• Treating evaluation as a one-time gate and forgetting the loop. Tell: teams act surprised when a shipped model needs re-evaluation, having assumed they'd 'train a model only once.'

• ▸Model version under review: name the new model and the production version it replaces.
• ▸Validity check result: state whether the new model breaks the serving system (pass/fail + evidence).
• ▸Output conformance: confirm outputs are well-formed and compatible with the serving path.
• ▸Quality baseline: record the currently running model's offline accuracy as the comparison point.
• ▸New model offline accuracy: record the metric on the held-out evaluation set.
• ▸Threshold decision: does new accuracy beat the baseline and clear the threshold? (yes/no).
• ▸Reviewers and expertise: list the data scientist, ML engineer, and business analyst who signed off.
• ▸Failure/improvement notes: if not shipping, record which features or data to change in the pipeline.
• ▸Loop status: mark whether this model is deployed or returned to the improvement loop.

1. 1Start with the assumption of abundance, not scarcity: architect on top of cloudlike foundations (Amazon S3 for storage, EC2 for compute) where storage is virtually unlimited and compute scales elastically, so your design optimizes for cost-effectiveness and productivity instead of rationing.
2. 2Identify which building block needs to scale before adding resources: the algorithm (add CPU cores, GPUs, RAM on one instance), the task/OS process (use all cores and RAM on the instance), the workflow (fan out tasks across multiple instances), or parallel workflow versions (needs a scalable orchestrator plus many instances).
3. 3Configure the compute layer beneath Metaflow — for example, set up AWS Batch — so tasks execute on cloud instances rather than the laptop, then request capacity per step with @batch and @resources decorators specifying CPU, memory, and GPU.
4. 4Follow the gradual-optimization path: begin with vertical scalability (scale up one instance's capacity) since off-the-shelf libraries like Scikit-Learn and TensorFlow already use all available CPU/GPU/RAM, and only move to horizontal scaling (dynamic branches fanning out to many instances) when one instance is insufficient.
5. 5Isolate every workload so failures have a limited blast radius: run tasks as containers on the compute layer so an ill-behaved or failing experimental workload causes minimal collateral damage to neighbors.
6. 6Add failure handling as decorators once workloads run remotely: @retry to recover from transient errors, @timeout to kill zombie tasks, and @catch as the decorator of last resort.
7. 7Remove human coordination overhead: rely on the infrastructure to grant relatively open access to compute rather than a capacity-planning team rationing fixed clusters, which restores data scientist autonomy over the full prototyping-to-production loop.
8. 8Pay only for what you use — start from laptop-sized capacity (a small compute layer used occasionally can cost less than a physical laptop) and scale resources elastically as needs grow.

Consider hyperparameter search, the passages' horizontal-scalability example. A single model-training run scales vertically: Scikit-Learn already uses all CPU cores, GPUs, and RAM on one instance, so you scale it by picking a bigger instance via @resources. But a search over many hyperparameter configurations is inherently parallel — each configuration is an independent task. You express this with a dynamic branch in Metaflow, which spawns an arbitrary number of tasks, one per configuration. The workflow fans these tasks out across multiple compute instances through the compute layer (AWS Batch), so dozens of variants train concurrently instead of sequentially. Because the cloud supplies effectively unlimited capacity, you don't coordinate with colleagues or wait weeks for provisioning; you request the instances, pay for the runtime consumed, and release them. Isolation via containers means one bad configuration that crashes does not take down the other trials.

• If a single algorithm just needs more data or model complexity handled, then scale vertically — request a bigger instance with @resources — before attempting horizontal fan-out.
• If the work is embarrassingly parallel (independent tasks like hyperparameter configurations), then use dynamic branches to fan out across many instances horizontally.
• If provisioning more capacity would previously have taken weeks from a capacity-planning team, then treat the cloud as elastic instead — request capacity per task and stop treating it like a fixed-size cluster or mainframe.
• If workloads are experimental and cannot be assumed well-behaved, then enforce isolation (containers) so a hard failure has a limited blast radius rather than adding a human supervision layer.
• If a task hits transient errors, use @retry; if it hangs, use @timeout; only fall back to @catch when no cleaner recovery exists.

• Treating the cloud like a fixed-size cluster or mainframe — the tell is teams still coordinating access to 'their' provisioned resources and quadratic communication overhead that scarcity used to force.
• Premature optimization — jumping to horizontal scaling and complex parallelism before exhausting simple vertical scaling; the tell is added complexity with no corresponding capacity limit actually being hit.
• Skipping isolation so one experimental workload interferes with neighbors — the tell is a single bad task or query taking down shared resources for all users (the shared-database analogy).
• Assuming a single universal compute layer serves every workload — the tell is forcing latency-sensitive, GPU, and cost-optimized jobs through one configuration when different workflows have different compute requirements.
• Provisioning always-on large capacity instead of paying for what you use — the tell is idle expensive instances when a small compute layer used occasionally would cost less than a laptop.

• ▸Scaling block: which layer needs to scale — algorithm, task, workflow, or parallel versions?
• ▸Scaling direction: vertical (bigger instance) or horizontal (fan out via dynamic branches)?
• ▸Resource request: CPU cores, RAM, and GPU to specify in @resources
• ▸Compute layer: local laptop, cloud workstation, AWS Batch, or GPU cluster
• ▸Isolation plan: how each task is containerized to limit blast radius
• ▸Failure handling: which of @retry / @timeout / @catch applies and thresholds
• ▸Concurrency limit: max parallel tasks or workflow versions to run at once
• ▸Cost check: estimated pay-per-use runtime vs. keeping capacity idle
• ▸Coordination removed: what human hand-off this provisioning eliminates

1. 1Instrument the ML system, not just the algorithm — you are monitoring systems doing ML, not using ML to monitor systems (that latter is AIOps, a different concern). Start by mapping every functional subsystem: feature system, serving/prediction path, and the metadata layer.
2. 2Adopt Greg Poirier's operating definition: monitoring is the action of observing and checking the behavior and outputs of a system and its components over time. Translate this into signals for both the serving infrastructure (latency, error rate, resource use) and the prediction outputs themselves.
3. 3Layer in ML-specific signals on top of generic production signals: input feature drift, prediction quality, and user-perspective KPIs (in the yarnit.ai case, whether predicted purchase likelihood actually tracks purchases). Generic infra health alone is not enough because the service can go down or degrade silently while CPU and memory look fine.
4. 4Make monitoring a shared responsibility, not a checkbox. Under the DevOps tenet that everyone owns production, the software/ML engineers who know the model instrument it and define its alerts; an observability team may build the self-service tooling but does not own instrumenting the applications.
5. 5For continuous ML systems that receive streaming or periodic batch data and retrain on a trigger, add lifecycle instrumentation around each retrain-and-push cycle: monitor the incoming data, the new model version's validity and quality before it swaps in, and the serving behavior after. A once-trained model needs far less of this than a continuously updated one.
6. 6Wire monitoring into the pre-production gate. Before flipping the switch on a new or updated model, verify validity (will it break the system?) and quality (is it any good?) — the same signals you monitor in production should be checkable at deploy time so rollout is smooth and reversible.
7. 7Tune for signal-to-noise so people trust the alerts. Treat 'it's not ready for production until it's monitored' as the rule, and treat noisy, ignored alerts as a defect to fix, not an inevitability.

Take the yarnit.ai online store. The team ships a model that predicts the likelihood a given user will purchase a given product, and serves it so predictions flow back to the systems that need them. Generic production monitoring covers the serving path: request latency, error rates, and whether the prediction endpoint stays up. But that is insufficient — the endpoint can return 200s while predictions quietly degrade, so you also track the distribution of incoming features (drift), the model's prediction quality against realized purchases, and the business KPI (conversion). Because yarnit.ai's data arrives continuously and triggers retraining, each new model version is gated on validity ('will it break our system?') and quality ('is it any good?') before it replaces the running version. After the swap you keep watching the same signals to confirm the update was safe and reversible. The engineers who built the model own its alerts; if an observability team exists, it supplies the dashboards and tooling, not the instrumentation itself.

• If the service can go down or return degraded predictions while CPU, memory, and system load all look normal, then your infra metrics are checkbox monitoring — add output- and prediction-quality signals.
• If the model retrains on streaming or periodic batch data and pushes new versions, then instrument every retrain cycle (data in, validity/quality gate, serving out); if it is trained once and deployed, lighter monitoring suffices.
• If one person or team is solely responsible for all monitoring, then you have shirked the responsibility — reassign instrumentation and alert design to the engineers who own each service.
• If a new model version is about to replace a running one, then gate the swap on both validity and quality checks before flipping the switch, and keep the path reversible.
• If alerts are noisy and no one trusts them, then treat the signal-to-noise ratio as a defect and fix it rather than adding more alerts.

• Checkbox monitoring: recording system load, CPU, and memory purely to say you have monitoring, while the service still goes down. The tell — the dashboard is green during the outage and nobody was paged by the thing that actually failed.
• Confusing monitoring ML with using ML to monitor (AIOps). The tell — the team buys anomaly-detection tooling but never instruments the model's own prediction quality or drift.
• Resting all monitoring on a single person or team. The tell — engineers say 'that's the monitoring team's job' and no one can explain what a given model's alerts mean.
• Treating a model as trained-once-and-deployed when it is actually continuous. The tell — new model versions get pushed to serving with no per-cycle validity/quality checks and drift goes unnoticed between retrains.
• Noisy, untrusted alerting. The tell — people mute or ignore pages, and real incidents are found by customers before the monitoring flags them.

• ▸Subsystem — name the component (feature system, serving path, metadata, retrain pipeline).
• ▸Generic signal — the infra metric tracked (latency, error rate, resource use) and its threshold.
• ▸Prediction-quality signal — how you measure whether predictions are still good, and against what ground truth.
• ▸Drift signal — which input features or output distributions you watch and the alert condition.
• ▸User/business KPI — the user-perspective metric this model is supposed to move (e.g., conversion).
• ▸Owner — the engineer/team who instruments it and defines its alerts (not the observability team).
• ▸Deploy gate — the validity and quality checks run before a new version swaps in.
• ▸Continuous-ness — is the model trained once or retrained on streaming/batch triggers, and what is monitored per cycle.
• ▸Signal-to-noise status — is this alert trusted; last time it fired, was it actionable?

1. 1Make monitoring a first-class citizen: treat a service as unfinished until it's monitored. Assign instrumentation to the people who know the system best — network engineers for the network, software engineers for their applications — rather than resting all of it on one person.
2. 2Distinguish a valid observability team (builds and cultivates self-service monitoring tools others use) from the anti-pattern of a single person owning all monitoring. Keep tool-building central, but push instrumentation, alert creation, and ownership out to the teams.
3. 3Run a daily on-call review: compile every alert that fired the previous day and, for each, ask whether the signal can be improved, the threshold tuned, the underlying check redesigned, or the alert deleted entirely. Do this every on-call day.
4. 4Audit a month of alert history against three columns: what fired, what action was taken, and what the impact was. Any alert with no action and no impact is a deletion or automation candidate.
5. 5Replace brittle static thresholds with methods that match the failure you actually care about: percent change/derivative (e.g., 'disk grew 50% overnight'), or with a capable metrics backend like Graphite, moving averages, confidence bands, and standard deviation.
6. 6Separate a signal problem from a firefighting problem. If dozens of alerts fire daily and they are all legitimate, stop tuning and fix the underlying systems — monitoring doesn't fix anything.
7. 7Institutionalize resiliency work: make it on-call's duty to improve system stability when not fighting fires, and/or explicitly plan resiliency work in the next sprint using the previous on-call week's data.
8. 8Define what 'working' means for the service (its core functionality) and monitor that directly, instead of only recording system load, CPU, and memory — metrics that can look green while the service is down.

A team has a disk-space alert wired to a static threshold. The threshold either fires too late to act or fires so often that on-call has learned to think 'I've seen that alert before, it'll clear itself up.' That thought is the tell of alert fatigue — the signal is being ignored. Instead of a fixed threshold, they switch to a derivative check that reports 'disk usage has grown by 50% overnight,' which surfaces the trajectory rather than a snapshot. During the daily on-call review they find the old static alert never produced an action, so they delete it. With a Graphite-class backend they layer confidence bands so genuine anomalies stand out from normal daily variation. The result is fewer, higher-signal alerts that people trust and act on again.

• If an alert requires no human action, then delete it or automate it away — don't route it to a person.
• If someone catches themselves thinking 'I've seen this alert, it'll clear itself up,' then treat it as alert fatigue and fix or remove that alert now.
• If the alerts are all legitimate but there are dozens a day, then it's a firefighting problem, not a signal problem — spend the effort on system resiliency, not tuning.
• If a single person or team is solely responsible for all monitoring (not just tool-building), then redistribute instrumentation ownership to the engineers who know each system.
• If your dashboards are green on CPU/load/memory but the service still goes down without explanation, then you have checkbox monitoring — define and monitor actual core functionality.

• Checkbox monitoring: systems exist only to say they exist (compliance or a mandate from above). Tell — the service goes down without your knowing why while metrics look fine, and you're not even storing historical data.
• Alert fatigue from the treacherous middle ground: a mix of actionable and non-actionable alerts erodes trust. Tell — slowed human response, ignored alerts, impacted sleep, and staff burnout.
• Monitoring-as-a-job: one person sets up the simplest, easiest checks and ticks the box because they don't understand how the system works. Tell — shallow instrumentation that misses the real failure modes.
• Treating a firefighting problem as a signal problem: endlessly tuning alerts that are all legitimate. Tell — you keep tuning but the volume of real incidents never drops because the underlying systems stay broken.
• Chasing 100% alerting accuracy as a prerequisite. Tell — paralysis; you never ship improvements because you can't reach perfection, when appreciable reduction was achievable.

• ▸Alert name — the exact alert that fired
• ▸Times fired (last 24h / last month) — count to spot noise
• ▸Action taken — what the responder actually did, or 'nothing'
• ▸Impact — real user/service impact, or 'none'
• ▸Verdict — keep / tune threshold / redesign check / automate / delete
• ▸Signal fix — specific change to improve signal (e.g., switch to derivative check)
• ▸Firefight flag — yes if legitimate-but-frequent, routing it to resiliency work
• ▸Owner — the engineer who knows this system and owns the fix

1. 1Enumerate the failure classes ML systems actually experience before an incident occurs: disk failures, file corruption, machine breakage, network outages, API calls returning errors, data getting stuck or changing subtly, and models degrading from once-accurate to less-so. Use this list to build detection coverage.
2. 2Distinguish whether the affected system is trained-once-and-deployed or a continuous ML system, because continuous systems receive streaming or periodic-batch data that triggers automated retraining and pushes to serving — meaning a bad data batch can silently propagate a broken model. Raise the bar on automated verification for continuous systems specifically.
3. 3Instrument for the three ML-specific decay signals that ordinary software incident management misses: subtle data changes, model accuracy drift, and world changes where previously rare events become commonplace. Alert on these, not just on hard outages.
4. 4When an incident fires, first recognize the situation correctly — classify it as infrastructure (disk/machine/network/API) versus data (stuck/changed) versus model (accuracy loss) versus environment (distribution shift), since mitigation differs by class.
5. 5Mitigate by exploiting the reversible path to production: because serving is a separate structure that asks the model for predictions, roll back to a known-good model version to stop harm before you diagnose root cause.
6. 6Apply standard incident management practices — declaring an incident, assigning roles, communicating — since best practices are common across many computer systems; layer ML-specific handling on top rather than reinventing.
7. 7Close the loop by learning: write down what you faced, how you dealt with it, what you learned, and what you'd do differently next time, in the practitioners' own words, so the capability compounds across incidents.

Consider a continuous ML system like the yarnit.ai store's purchase-likelihood model, which is retrained on periodic batches and pushed to serving. Suppose an upstream data feed gets 'stuck' — it keeps delivering yesterday's inventory as if it were current. The training system reads this data, imposes its usual consistency checks, and produces a new model version that is technically valid but representing a stale world, so recommendations quietly degrade rather than throwing errors. Detection here does not come from a downed disk or a failed API call; it comes from monitoring model accuracy and watching for subtle data changes. The right first move is to use the reversible path to production and roll serving back to the last known-good model version, halting customer harm. Only then do you trace the incident to the stuck feed, fix ingestion, and add an automated verification gate so a stale batch cannot trigger a push again. Afterward, the team documents the timeline and what they'd do differently, feeding the next round of prevention.

• If the system is continuous (streaming or periodic-batch retraining), then treat automated verification as mandatory before any model reaches serving, because bad data auto-propagates.
• If the symptom is degraded predictions with no infrastructure failure, then suspect data change or model decay first, not a hardware or network fault.
• If a good model has become less accurate over time, then check whether the external world changed — previously rare events becoming common — before assuming a code or data bug.
• If harm is ongoing and root cause is unknown, then roll back to a known-good model version immediately, since the path to production should be reversible.
• If the incident is a generic outage (disk, machine, network, API error), then apply standard cross-system incident management practices rather than inventing ML-specific process.

• Treating training as a one-time event; the tell is a team with no monitoring on model accuracy or data drift because they assume the model, once deployed, stays correct.
• Missing subtle data corruption because alerts only cover hard failures; the tell is 'all systems green' while predictions quietly worsen.
• No reversible path to production; the tell is that when a bad model ships, the only option is a slow forward fix instead of an immediate rollback.
• Ignoring world change as a cause; the tell is repeatedly retraining on the same features while accuracy keeps sliding because the environment shifted, not the pipeline.
• Never writing up incidents; the tell is the same failure recurring because there's no record of what happened, what was learned, and what to do differently.
• Applying only generic software incident management and skipping ML-specific detection; the tell is a mature on-call process that catches outages but never catches model decay.

• ▸Incident class: infrastructure / data / model / environment — pick the primary one
• ▸System type: trained-once vs continuous (streaming or periodic-batch)
• ▸Detection signal: what fired (hard failure, data change, accuracy drop, or world change)
• ▸Immediate mitigation: known-good model version to roll back to, and whether the path is reversible
• ▸Automated verification gap: which check would have caught this before serving
• ▸Root cause: the specific failure (disk, corruption, stuck/changed data, decay, shift)
• ▸What we learned: one sentence in the responder's own words
• ▸What we'd do differently next time: concrete prevention added

1. 1Separate the two failure classes before acting: software system failures (infra, pipeline, dependency issues) versus ML-specific failures (data distribution shifts). Only the latter is fixed by refreshing the model — don't retrain when the real problem is a broken feature pipeline.
2. 2Distinguish the types of data distribution shift you're facing, since the source separates general shifts from specific types. Ask whether the input distribution changed, the output distribution changed, or the input-output relationship changed, because each implies a different remedy.
3. 3Stand up detection before you stand up retraining. Monitor ML-specific metrics (not just system latency), and compare current production data distributions against the training distribution so drift surfaces as a signal rather than a support ticket.
4. 4Run the data-freshness experiment described in the source: train separate model versions on different rolling time windows in the past, then test all of them on the most recent data to quantify how much performance you gain per unit of freshness.
5. 5Set retraining cadence from that measured gain, not from intuition. If a model trained on data from a quarter ago performs much worse than one trained on a month ago, do not wait a quarter to retrain.
6. 6Make retraining infrastructure fast and cheap first, then increase frequency. The question 'how often to update' only becomes answerable once updating quickly is possible; then tighten the loop from monthly to weekly to daily as the freshness experiment justifies.
7. 7Gate every model update behind a continual-evaluation pipeline and test-in-production techniques. Writing the update function is easy; ensuring the updated model actually works properly in production is the hard part.

Facebook ran a data-freshness experiment for ad click-through-rate prediction. They measured how much the model improved as the retraining window moved closer to the present, and found they could reduce the model's loss by 1% by going from retraining weekly to retraining daily. That 1% was significant enough that they switched their retraining pipeline from weekly to daily. Following the source's generic method: you take data from 2020, train version A on January–June, version B on April–September, version C on June–November, then test all three on December data. The performance spread across versions tells you the marginal value of fresher data. The passages note that with today's more diverse online content and faster-shifting user attention, the value of freshness for ad CTR is even higher — and some companies with sophisticated infrastructure found enough gain to retrain every few minutes.

• If production data is 'constantly shifting' while your model was benchmarked on static research data, then assume performance decays over time and build monitoring — do not treat deployment as a finished state.
• If the model trained on a quarter-old window performs much worse than one on a month-old window, then retrain more frequently than quarterly.
• If measured performance gain from fresher data is small (e.g., near zero going weekly to daily), then keep the slower cadence and spend effort elsewhere.
• If a metric drops but the input distribution matches training, then suspect a software system failure (pipeline, feature inconsistency), not a distribution shift — refreshing the model won't help.
• If online content diversity and user attention change fast in your domain, then bias toward higher freshness because the value of recency compounds.

• Assuming any performance drop is a distribution shift. Tell: you retrain, the new model behaves identically, and the real cause was a broken feature pipeline or dependency.
• Setting retraining cadence by opinion ('people keep saying fresher data is better') instead of measurement. Tell: nobody can state the performance gain in numbers for their own system.
• Training-serving skew — features used during training differ from features at inference. Tell: offline metrics look fine but production accuracy quietly lags.
• Building the fast update function without a continual-evaluation gate. Tell: an updated model ships and degrades production because 'working properly' was never verified before rollout.
• Monitoring only system-level signals (latency, uptime) and no ML-specific metrics or distribution comparisons. Tell: the model rots for weeks and the first alert is a business-metric complaint, not a monitoring alarm.

• ▸Failure class: is this a software system failure or an ML-specific (distribution shift) failure?
• ▸Shift type: input changed / output changed / input-output relationship changed?
• ▸Detection method in place: how are you comparing production distribution to training distribution?
• ▸ML-specific metrics monitored: list the metrics that would flag degradation early.
• ▸Freshness experiment windows: the past time windows you train separate versions on.
• ▸Test set: the recent data slice all versions are evaluated against.
• ▸Measured gain: performance delta per step-up in retraining frequency (e.g., weekly→daily).
• ▸Chosen cadence + threshold: retrain every ___; trigger early if metric drops below ___.
• ▸Pre-deploy gate: which continual-evaluation / test-in-production checks must pass before rollout?

1. 1Measure delivery on two axes the book separates: tempo and stability. Tempo = deployment frequency and lead time for changes; stability = change fail rate and time to restore service. Capture all four; do not report tempo alone.
2. 2Define lead time narrowly as the time from code committed to code running in production — the delivery half, not the design/feature-definition half (see Table 2.1 Design vs. Delivery). This keeps the metric bounded and comparable across teams.
3. 3Instrument deployment frequency at the team/service level, then normalize by team size using deploys per developer per day (Figure 5.1) so a large team's raw volume doesn't mask low per-person throughput.
4. 4Track change fail rate as the percentage of changes to production that result in degraded service and require remediation (hotfix, rollback, patch), alongside time to restore service when an incident does occur.
5. 5Trend all four measures year over year (Figures 2.2 tempo, 2.3 stability) rather than reading a single snapshot; the book's finding is that high performers improve on both axes simultaneously — throughput does not trade off against stability.
6. 6Drive the numbers by implementing the specific capabilities the research links to delivery performance — continuous delivery, trunk-based development, loosely coupled architecture, Lean management, integrated infosec — rather than declaring a 'maturity level' reached.
7. 7Connect the four metrics to the outcomes chain the book documents (Figure 2.4): software delivery performance feeds organizational performance — productivity, profitability, market share — so report them to leadership as business indicators, not just ops trivia.

The research operationalizes delivery performance across the 2016 and 2017 cohorts (Tables 2.2 and 2.3) using exactly four measures: deployment frequency, lead time for changes, change fail rate, and time to restore service. High performers deploy on demand, multiple times per day, with lead times measured in less than an hour; lower performers deploy weekly-to-monthly with lead times in the range of weeks. Crucially, the high performers also restore service faster and have lower change fail rates — stability rises with tempo rather than falling. When the authors normalized throughput by team size (Figure 5.1, deploys per developer per day), they found deploys per developer scaled with the number of developers for high performers but flattened or dropped for others, exposing where added headcount stopped adding throughput. The upstream driver they identify is architecture: loosely coupled systems let teams deploy independently, which is what preserves per-developer deploy rates as the org grows. This is how the same four numbers diagnose both current performance and its architectural cause.

• If you can report deployment frequency but not time to restore service and change fail rate, then you are measuring tempo only — stop and instrument stability before claiming a performance level.
• If adding developers lowers deploys per developer per day (Figure 5.1), then your architecture is too tightly coupled — invest in decoupling before hiring more.
• If someone proposes ranking teams on a fixed maturity level, then reject it in favor of a capability model — the research shows high performers never consider themselves 'done' and keep improving both axes.
• If lead time includes feature-definition and design work, then re-scope it to commit-to-production (Table 2.1) so the metric stays bounded and comparable.
• If a change to raise deployment frequency also raises change fail rate, then treat it as a regression, not progress — the finding is that both improve together.

• Optimizing tempo while ignoring stability — the tell is dashboards full of deploy counts and lead times with no change fail rate or restore-time column.
• Measuring lead time from idea to production, mixing design with delivery — the tell is wildly variable lead times dominated by prioritization queues, not engineering flow (contradicts Table 2.1's design/delivery split).
• Reading throughput as raw deploy volume so big teams look elite — the tell is that per-developer deploys were never computed (Figure 5.1).
• Adopting a maturity model and declaring the transformation finished — the tell is a static target state and a stalled improvement curve, which the book explicitly warns against.
• Treating these as ops-only metrics disconnected from business — the tell is that delivery performance never appears in conversations about productivity, profitability, or market share (Figure 2.4).

• ▸Deployment frequency: how often you deploy to production (per day / week / month)
• ▸Lead time for changes: median time from code committed to running in production
• ▸Change fail rate: % of production changes that degrade service and need remediation
• ▸Time to restore service: median time to recover from a production incident
• ▸Deploys per developer per day: total deploys ÷ team size ÷ working days
• ▸Measurement scope note: confirm lead time is commit-to-prod, not idea-to-prod
• ▸YoY trend: this period's value vs. prior period for tempo and stability separately
• ▸Coupling flag: is per-developer deploy rate falling as headcount rises? (yes/no)
• ▸Business linkage: which org outcome (productivity/profit/market share) this feeds

1. 1Treat the ML system as a whole data processing pipeline, not just the algorithm — instrument every stage from data ingestion to serving, because ML pipelines 'fail in different ways' and many failures are 'difficult to detect.'
2. 2Before swapping any new model into production, answer the two gating questions in order: model validity ('will the new model break our system?') then model quality ('is the new model any good?'). Block the release if validity fails, regardless of quality gains.
3. 3Build monitoring per Greg Poirier's definition: observe and check the behavior and outputs of the system and its components over time. Define what 'healthy' output looks like at each pipeline stage so silent failures surface.
4. 4Stand up the three data subsystems that feed reliability — a feature system, a human-annotation system, and a metadata system — and version their outputs so a model can be reproduced and a bad model can be rolled back along a reversible path to production.
5. 5Establish robust verification processes that pull in the needed cross-functional expertise; budget for it as '1% inspiration and 99% verification' rather than treating validation as a final formality.
6. 6Design serving so the system can ask the model for predictions on new examples and return them to the people or systems that need them, and monitor serving latency and availability the same way you already monitor web server performance and database performance.
7. 7Practice safety in updating: make every model swap reversible so that when the pagers fire you can restore a known-good version quickly.
8. 8Diagnose from the perspective the book targets — when 'the alarms start to go off or the pagers start to fire,' first determine whether the failure is system-level (independent of model type) or genuinely model-related, since the fix differs.

Take the yarnit.ai online store. The team has built a model that predicts the likelihood a given user will buy a given product, and now needs to serve it — create a structure so the system can ask the model to make predictions on new examples and return them to whoever needs them. Before flipping the switch on this model or any updated replacement, the team runs validity first: does the new model integrate without breaking the serving path, feature pipeline, or downstream store logic? Only after validity passes do they assess quality: is the new model actually better at predicting purchases? Because model development here is '1% inspiration and 99% verification,' the team invests most effort in that check, pulling in data and serving experts. They keep the currently running version in place so the swap is reversible, and they monitor serving output over time so a degradation in prediction behavior is caught rather than running silently.

• If model validity fails (the new model will break the system), then do not deploy — even if quality metrics improved.
• If a failure alarm fires and the symptom is independent of model type or quality, then treat it as a system-level/infrastructure issue (serving, storage, pipeline) before touching the model.
• If you cannot reproduce a model from its feature, annotation, and metadata records, then it is not ready for a critical setting — repeatability precedes promotion.
• If a new model cannot be swapped out along a reversible path, then hold the release until rollback is possible.
• If you are choosing a new monitoring tool to solve reliability problems, then stop — define the mission and capabilities first, because a tool-driven team is captive to its tools' limitations.

• Focusing on squeezing out another percentage point of model accuracy while system-level behavior degrades — the tell: metrics improve in the lab but production still pages.
• Tool obsession: buying a monitoring or 'better monitoring' toolset instead of defining the mission — the tell: someone blames the current toolset and proposes evaluating new ones without stating what capability is missing.
• Assuming ML pipeline failures announce themselves like conventional log-processing failures — the tell: no alarm fired, yet output quality quietly drifted because success is hard to measure and many failures are difficult to detect.
• Treating verification as a rubber stamp rather than 99% of the work — the tell: models promoted quickly with little cross-functional review, then broken behavior discovered in production.
• Deploying without a reversible path — the tell: when a swapped-in model misbehaves, the team scrambles because there is no clean rollback to the previously running version.

• ▸Model & version: name the candidate model and the version it replaces.
• ▸Validity check: state whether the new model integrates without breaking serving/pipeline (pass/fail).
• ▸Quality check: record the metric and whether the new model is demonstrably better.
• ▸Reproducibility: confirm feature, annotation, and metadata records let you rebuild this model.
• ▸Reversibility: describe the exact rollback path to the currently running version.
• ▸Monitoring hooks: list which outputs/behaviors will be observed over time post-deploy.
• ▸Failure detection: note how a silent (non-alarming) degradation would be caught.
• ▸Sign-off: name the cross-functional reviewers who verified validity and quality.

1. 1Fix your scope before instrumenting: decide whether you measure all changes across the org or just your program, and whether infrastructure changes count. Apply that same scope to all four metrics — if infra changes are in lead time and deployment frequency, then infra-induced outages must be in change failure rate and time to restore.
2. 2Identify the pipelines in scope: those that listen for code and config changes in a source repository, act on them (compilation, automated testing, packaging), and deploy into production. Exclude CI tasks that aren't deliveries, like database backups.
3. 3Set the commit timestamp (clock start) as the moment code hits main. If teams hold changes on branches, either capture the true commit time or use the pipeline trigger from merge-to-main as a proxy — knowing the proxy hides branch wait time.
4. 4Set the deployment timestamp (clock stop) as the moment the final deployment-to-production pipeline completes. If you insist on counting manual smoke testing, add a manual checkpoint at the pipeline's end that QA presses once satisfied.
5. 5Calculate total time to run as elapsed time between clock start and clock stop; that gives lead time for changes, and counting deployments over a period gives deployment frequency.
6. 6Instrument the third data point — service failures — via a 'change failure' ticket: opening it starts the restore clock, closing it (when service is restored, not necessarily root-caused) stops it. Rolling back to serve customers counts as restored.
7. 7Capture manually first (a spreadsheet is acceptable), then automate. Balance the two pairs — throughput (deployment frequency, lead time) against stability (change failure rate, time to restore) — because improving one pair while degrading the other is unbalanced and won't yield sustainable benefit.
8. 8If you're not yet in production, define your 'highest environment' (SIT, pre-prod, or staging), treat testers and collaborating teams as users who define failures, and treat test failures as seriously as real ones.

On one client engagement (all-in on Azure DevOps), the author built a dashboard in Microsoft PowerBI. After wrestling with dates and times, they captured raw data, ran calculations, and built visuals. Deployment frequency was a bar graph with dates on the x-axis and number of deploys per day on the y-axis, with average deployments per day pulled into a summary box and highlighted green to show 'DORA Elite'; they also plotted mean, 95th, and trend lines as dotted overlays. Lead time for changes was a bar graph showing the daily mean lead time, with the period mean highlighted (rating 'DORA High') and the single longest lead time called out in a box. Because they kept asking 'did we do a lot of deploys that day?', they shadow-plotted deploy counts in light gray behind the lead times rather than adding more trend lines. Change failure rate got yet another bar graph. This home-baked effort became the most fully featured dashboard the author had ever worked with.

• If a metric includes infrastructure changes, then every other metric must include the corresponding infrastructure effects — keep scope identical across all four.
• If teams hold changes on branches, then either capture the real commit time or accept the merge-to-main proxy — but expect a guilty conscience and adopt trunk-based development and pair programming when the proxy causes real suboptimization.
• If a rollback restores service, then close the change-failure ticket even though root cause is unaddressed — you're measuring stability, not diagnosis.
• If you improve throughput but degrade stability (or vice versa), then you're improving in an unbalanced way — treat only across-the-board positive movement as success.
• If you aren't in production, then designate the shared environment closest to production as your 'highest environment' and treat it exactly like production.

• Mixing scopes across metrics — counting infra changes in lead time but ignoring infra-induced outages. Tell: throughput looks great while failure rate seems suspiciously clean.
• Including non-delivery CI tasks (like database backups) in your pipeline set. Tell: deployment frequency inflates with events that never reached users.
• Silently adopting the merge-to-main proxy and forgetting it hides branch wait time. Tell: lead time looks short but real feedback cycles feel long to developers.
• Chasing one pair of metrics — pushing deployment frequency up while service stability quietly degrades. Tell: more deploys but rising change failure rate and longer restore times.
• Delaying rollout waiting to automate capture. Tell: no metrics exist yet because manual capture — the acceptable and recommended starting point — was skipped.

• ▸Scope: name exactly what changes count (all org / this program; include infra? yes/no)
• ▸Pipelines in scope: list repos/pipelines that build+test+package+deploy to prod
• ▸Clock start rule: commit timestamp definition (true commit time or merge-to-main proxy)
• ▸Clock stop rule: final prod-deployment completion (plus optional QA manual checkpoint)
• ▸Failure definition: what counts as a service failure that stops users completing tasks
• ▸Failure ticket rule: open = start restore clock; close = service restored (rollback allowed)
• ▸Highest environment (if not in prod): name it and who acts as 'users'
• ▸Capture method: manual (spreadsheet) now, automation target later
• ▸Balance check: current values for all four, flagged if any pair moves against the other

1. 1Treat security as a continuous-delivery capability, not a downstream gate: apply the 'shift left on security' practice (Chapter 6) so security is built into the delivery process from the start rather than added as a separate phase.
2. 2Integrate infosec personnel into the delivery loop as a feedback source. The passages list 'infosec personnel' alongside customers, production monitoring, and team members as valid feedback channels — route their input into design and build, not just into a final review.
3. 3Store all environment and application configuration in version control so builds, tests, and deployments run in a fully automated fashion; this lets security-relevant changes be applied consistently and reproducibly, leaving room for manual approvals that then execute automatically.
4. 4Add security remediation to your quality measures. The 2016 research extended quality measurement to include defects, rework, and security remediation — track how long security fixes take as a first-class delivery metric.
5. 5When failures or security incidents occur, run blameless investigations: treat 'human error' as the start of the investigation, not the end, and ask how to improve information flow or tooling — avoid the pathological 'throat to choke' response.
6. 6Build fairness into decision-making processes, not just outcomes: use rules applied uniformly so no one gets preferential or discriminatory treatment (Schwartz), while keeping the mission primary so rule-following never overrides achieving the goal.
7. 7Address fairness in the work environment as a burnout risk factor. Maslach lists 'absence of fairness' in decision-making as one of six predictors of burnout — fix the process, not the person.

In the 2016 research year, the team set out to answer a specific, contested question: does integrating security into software development and delivery help the process or slow it down? Rather than assume security is a drag, they measured it — extending quality measures to include security remediation alongside defects and rework. They folded security in with other technical practices like trunk-based development and test data management as part of the same delivery model. The result confirmed their hypothesis: integration helped rather than hindered, so 'shift left on security' entered the list of 24 key capabilities that drive delivery performance. The practical mechanism is treating infosec staff as a feedback source inside the delivery loop and versioning all configuration so security changes deploy automatically. This is the paid difference from a summary: security stops being a phase you pass through and becomes a capability you measure and improve.

• If security is currently a downstream approval phase, then move it upstream into build and design as a continuous-delivery capability and measure remediation time.
• If an incident investigation stops at 'human error,' then it is incomplete — restart it to find the information-flow or tooling gap that let the error propagate.
• If a rule prevents achieving the mission rather than guaranteeing fairness, then the culture has slipped toward rule-oriented/pathological — re-anchor on the mission.
• If infosec personnel only see the product at the end, then reroute their feedback into the same fast loop used for customers and production monitoring.
• If a decision process feels arbitrary or preferential to the people affected, then treat it as a burnout and fairness risk and standardize the rules applied to it.

• Bolting security on as a final gate — the tell: security review is the slowest, most-dreaded step and generates late rework instead of early design changes.
• Looking for a 'throat to choke' after a security incident — the tell: investigations name a responsible person and stop, rather than mapping the complex interplay of contributing factors.
• Trying to fix the person instead of the environment when fairness or burnout surfaces — the tell: interventions target individuals while unfair decision processes stay untouched (Maslach found environment fixes succeed more often).
• Confusing rule-following with good governance — the tell: teams optimize for compliance with rules even when it defeats the mission, the hallmark of a rule-oriented culture.
• Not measuring security at all — the tell: you debate whether security slows delivery instead of tracking security remediation as a quality metric the way the 2016 research did.

• ▸Security touchpoint stage: name where in the delivery process security currently enters (aim: build/design, not final gate).
• ▸Infosec feedback channel: how and how often infosec input reaches developers.
• ▸Config in version control: yes/no — can environments and deploys run automatically from version control?
• ▸Security remediation metric: current median time to remediate a security finding.
• ▸Automated security tests in CI: which security checks trigger on every commit.
• ▸Incident investigation rule: state your blameless standard (human error = start, not end).
• ▸Fairness-of-process check: which decision rules are applied uniformly to all cases.
• ▸Mission-vs-rule flag: note any rule that currently overrides achieving the mission.
• ▸Owner: person accountable for keeping security and fairness built in, not bolted on.

1. 1Understand your business processes and identify which ones are critical to service delivery (Step 1); do not attempt improvement on everything at once — process improvement discipline starts with scoping the critical few.
2. 2Assess current process maturity before you commit to improvement targets or ISO/IEC 20000 certification. Use the CMMI maturity level definitions (or another assessment tool) to locate where each process sits on the scale; ISO 20000 certification requires processes at a fairly consistent maturity level first.
3. 3Define what is to be measured — the hardest element — and make it involve the customer. Separate the three metric types: technology metrics (component performance/availability), process metrics (CSFs, KPIs, activity metrics answering quality, performance, value, compliance), and service metrics (end-to-end results computed from component metrics).
4. 4Run improvement through DMAIC discipline: define requirements with the customer, measure the process against DPMO, analyze variation, improve to hit target, then control. In the control phase develop process control plans reflecting the new targets and train operators on statistical process control charting to keep variables on target and within acceptable ranges.
5. 5Resource the effort with defined roles: Green Belts as trained team members driving cultural change within their existing responsibilities, Black Belts as subject-matter experts leading improvement teams, Master Black Belts as full-time teachers who review and mentor Black Belts.
6. 6Measure results, not efforts — focus on organizational output and the actual contribution, and report the true service level as the customer experiences it rather than component-level reports that protect or blame.
7. 7Use tension metrics to balance shared goals against delivering to business requirements within time and budget, so an unbalanced focus does not drive budget increases or lower product quality.
8. 8Model the business process (Business Process Modeling) to make visible all work sequences, interactions and interdependencies; use the model to avoid duplication, optimize workflows, and anchor systems, procedures and quality controls.

Six Sigma began in Motorola's manufacturing areas where Statistical Process Control was already accepted, but the passages insist it applies equally to service — 'Most service areas don't have a clue as to how well they are doing now.' Take an IT service desk process: first you define what to measure with the customer, treating a defect as anything outside customer expectations. You then measure defects per million opportunities (DPMO): to reach Six Sigma the process must not produce more than 3.4 defects per million opportunities, i.e. be 99.99966% error-free. A Black Belt leads the improvement team while Green Belts apply the tools inside their own work; a Master Black Belt mentors the Black Belts. In the control phase you build a process control plan reflecting the new targets and train the operators on SPC charting so the variables stay on target and within acceptable ranges. The passage notes the payoff: productivity increases leading to direct cost savings.

• If you are heading toward ISO/IEC 20000 certification, then first bring your service management processes to a fairly consistent maturity level (assessed via CMMI or equivalent) before starting the certification process.
• If a process still produces more than 3.4 defects per million opportunities, then it has not reached Six Sigma — measure against DPMO rather than declaring success on impression.
• If you are choosing improvement targets, then measure results and organizational output, not efforts — a metric describing activity volume does not tell you about contribution.
• If you can only report at component/system/application level, then you have not delivered a service measurement; combine component metrics into the end-to-end service level as the customer experiences it.
• If focus is unbalanced toward one goal, then apply tension metrics to rebalance, because unbalance leads either to budget increase or lower product quality.

• Assuming Six Sigma 'only works in manufacturing.' The tell: service areas run 'processes' but have no SPC or DPMO data, so they cannot say how well they are performing now.
• Monitoring and reporting on component levels to protect yourself or shift blame. The tell: reports show component/system/application status but never the true service level the customer experiences.
• Measuring efforts instead of results. The tell: dashboards fill with activity counts while no one can state the organizational output or contribution.
• Skipping maturity assessment before improvement or certification. The tell: an ISO 20000 attempt stalls because processes sit at inconsistent maturity levels.
• Letting ROI calculations focus on the short-term. The tell: adoption decisions ride on a single near-term numerical criterion while longer-term service management benefits are ignored.
• Treating defining requirements as trivial. The tell: measurement is defined without the customer, so 'defect' is not anchored to customer expectations.

• ▸Critical process & why it matters to service delivery: name the process and its link to the customer-experienced service.
• ▸Current maturity level (CMMI): state assessed level and the tool used.
• ▸Customer-defined defect: write what counts as 'outside customer expectations'.
• ▸Baseline DPMO / defect rate: record current defects per million opportunities.
• ▸Target DPMO & error-free %: set the goal (e.g., 3.4 DPMO / 99.99966%).
• ▸Metric set: list the technology, process (CSF/KPI), and service metrics you will track.
• ▸Tension metrics: name the balancing measures across goals, time and budget.
• ▸Roles: assign Green Belt(s), Black Belt lead, Master Black Belt mentor.
• ▸Control plan & SPC training: define the control-plan targets and who is trained on SPC charting.

1. 1Anchor each metric to a business objective before you measure it — if it does not map to a business outcome or an expected outcome, discard it; relevance to business objectives is the single non-negotiable filter.
2. 2Segment your metrics by horizon: operational (process operation), tactical, and strategic, so feedback drives decisions at the level that owns them. Route operational process measures to line managers, KPIs-against-SLAs to Service Level Management, financial trends to Financial Management.
3. 3Build KPIs from negotiated SLAs and present them as exception reports — surface only breaches or trends heading the wrong way rather than raw dashboards, so steering effort goes where targets are missed.
4. 4Verify every tool-supplied 'out of the box' metric against its actual formula before you steer on it; confirm what the package is really counting matches what you intend to improve.
5. 5Design integrated sets, not isolated numbers — a workable set lets you tell whether performance is genuinely improving; run individual metric design (6.4) then integrate into sets (6.5) and score/prioritize them (7.3).
6. 6Use trends over time as the steering signal: watch KPI trends in SLA reports to show whether IT's ability to deliver is improving, and watch cost trends to spot movement in the wrong direction.
7. 7Take corrective action on adverse cost trends before over-run, not after — have Financial Management monitor organizational costs overall once the cost model matures, and act while the deviation is still small.
8. 8Hold off on converting process measures into monetary steering (ROI, ROCE, EVA) until IT Financial Management reaches high maturity; until then steer on process efficiency plus KPIs against SLAs as the most complete picture.

Take Configuration Management as the metric-choice example from the guide. You start by asking why each candidate metric exists rather than accepting the hundreds a tool offers 'out of the box'. You build a data model for measuring the metrics, then prioritize and score them so only the ones tied to a real management decision survive. For steering, one early use is cost allocation: costing surfaces an SLA that serves only a small part of the business. If that allocation is not a business priority, you act on the feedback and make a large cost saving. As the business matures, you hand Financial Management the job of monitoring overall organizational costs to catch adverse trends and correct before serious over-run — but you resist putting costing on an accurate basis too early, because that pressure is only bearable once process-based thinking is the norm.

• If a metric does not map to a business objective or expected outcome, then drop it regardless of how easy it is to measure.
• If a metric is easy to measure but might measure something you don't need to improve, then treat that ease as a red flag, not a reason to adopt it.
• If a cost trend moves in the wrong direction, then trigger corrective action before it reaches over-run rather than waiting for the year-end figure.
• If IT Financial Management has not reached high maturity, then steer on process-efficiency measures and KPIs-against-SLAs instead of monetary measures like ROI/ROCE/EVA.
• If an SLA is found to serve only a small part of the business and is not a business priority, then reallocate or cut it to capture the cost saving.

• Adopting tool-supplied metrics wholesale — the tell is a dashboard with hundreds of numbers and no one able to name the business decision each one drives.
• Steering on a metric whose formula is subtly different from what you assume — the tell is reported numbers that don't reconcile with the outcome you expected.
• Forcing accurate costing too early — the tell is organizational pushback and pressure the organization can't yet bear because process-based thinking isn't the norm.
• Building isolated metrics instead of integrated sets — the tell is that you can't answer whether performance is actually improving, only whether one number moved.
• Reporting raw measurement instead of exceptions and trends — the tell is business readers who can't tell from the report whether IT is getting better or worse.

• ▸Metric name — the specific measure as it will be reported.
• ▸Business objective link — the objective or expected outcome it supports; blank means discard.
• ▸Horizon — operational / tactical / strategic level that owns the decision.
• ▸Formula check — confirm what is actually counted matches intent (Y/N).
• ▸Target / SLA basis — the negotiated threshold this KPI is judged against.
• ▸Trend direction — improving / flat / adverse based on time series.
• ▸Corrective trigger — the deviation level at which you act, and who acts.
• ▸Priority score — from prioritizing/scoring the metric set (7.3).

1. 1Build metrics only on top of defined processes: before publishing any metric, confirm the underlying process exists and is documented, because measuring how things happen without a process gives no consistent method of improvement or way to discover what prevented a metric occurring.
2. 2Anchor every metric to a business objective: for each candidate metric, name which of the four objective classes it serves (align business with IT, achieve compliance, drive operational excellence, or prove value creation) and discard metrics that map to none.
3. 3Map each metric to its stakeholder using the customer relationship diagram, then write a communication plan that states the metric's purpose, its proper use, and what the result means for service delivery — co-construct and review this plan with the stakeholders themselves.
4. 4Make performance visible at the point of work: put process-appropriate metrics on prominent display in the IT department, the way factory floors show current performance graphs, so everyone is reminded of the goals daily.
5. 5Set reporting cadence to match the decision horizon: report daily/weekly for tactical decisions (e.g. Service Desk next week) and quarterly/annual for strategic decisions, and only record a data point when the process status actually changes.
6. 6Tier the report format by audience: graphical summaries for strategic/tactical consumers, tabular detail for a process manager who needs to build an action plan, choosing chart type by comparison purpose (pie for component share, bar for item comparison, column for time series, line for distribution, scatter for correlation).
7. 7Engage staff in setting and negotiating their own metrics, and rebuild job descriptions, performance evaluations and rewards around agreed metrics so achievement is expected, measured and rewarded — then reinforce meaning continuously in weekly or monthly staff meetings.
8. 8Add 'check usefulness and accuracy of metrics' to the Service Improvement Plan: physically visit the people working the process to confirm each metric measures a genuine process variable tied to the service, not a bureaucratic intrusion.

Two processes are scored on weighted sums of their integrated metrics, where 'customer' is defined as the main beneficiary of each process output. Availability Management scores 8.5 across its ten metrics; Change Management scores 10.8 across its metrics. Even though the two processes are quite different in nature, the comparable scoring lets you read directly that Availability Management is less in control of its processes than Change Management. This transparency turns opinion into evidence: the Change Manager, who must give final approval of changes and is exposed to political undermining, can point to the score as proof of doing a good job, backed by a corporate sponsor. Well-chosen metrics also expose which organizational areas are less cooperative than they could be, so you work with them to iron out process difficulties. The clarity comes from the integrated, weighted structure and an accurate customer-satisfaction component, not from any single raw number.

• If a proposed metric has no documented process behind it, then do not implement it yet — you cannot improve or diagnose what has no process.
• If a decision is tactical (what the Service Desk does next week), then use daily/weekly data; if it is strategic, then use quarterly/annual data — never make strategic calls off a daily report that may be an anomaly.
• If a report goes to a strategic audience, then present it graphically for visual impact; if it goes to a process manager needing an action plan, then present tabular detail.
• If a metric cannot be tied to a stakeholder need or a business objective, then cut it — 'out of the box' tool metrics that measure nothing meaningful add noise.
• If staff had no involvement in setting a metric, then expect resentment and resistance when it is measured; involve them first to build ownership and responsibility.

• Using metrics as a stick: consumers scan only for red items to beat the responsible process owner. Tell — reviews fixate on last period's exceptions while long-term trends and preventive decisions go undiscussed.
• Agreeing to a process then carrying on as before: without appropriate process metrics this stays hidden. Tell — the metric was 'adopted' but behaviour and outputs are unchanged.
• Costing too early and too precisely: putting accurate cost models in place before process-based thinking is the norm creates dangerous pressure. Tell — cost figures are challenged and destabilize adoption instead of guiding it.
• Aesthetic reporting over meaning: attractive charts chosen for looks rather than fit obscure the message. Tell — pretty visuals that don't match the comparison purpose (e.g. pie chart used for a time series).
• Rewarding empire over efficiency: culture rewards managers of many staff over those delivering great value with few, and ignores trends showing over-capacity. Tell — process managers who cut overhead are undervalued while headcount is celebrated.

• ▸Metric name — the single thing being measured
• ▸Underlying process — the documented process it measures (leave blank = do not proceed)
• ▸Business objective served — align / compliance / operational excellence / value creation
• ▸Stakeholder & need — who consumes it and what they need from it
• ▸Purpose & proper use — one line stating how it should and should not be used
• ▸Reporting cadence — daily/weekly (tactical) or quarterly/annual (strategic)
• ▸Report format & chart — tabular or graphical, plus chart type by comparison purpose
• ▸Visible display location — where front-line staff see it prominently
• ▸Reward linkage — what behaviour it rewards and how achievement is recognized

1. 1Separate inherent from incidental complexity before optimizing: inherent complexity is the natural state of the world (real data messiness, statistical uncertainty); incidental complexity is the unwanted artifact of your chosen approach (boilerplate, spaghetti pipelines, dependency hell, organization-driven interfaces). Only the latter is worth eliminating.
2. 2Default to simplicity as the first-order tool for cognitive load: write code so simple that anyone new to the project can read the source and quickly comprehend how it works. Treat 'simple but slow' as acceptable and 'complex but fast' as suspect until proven necessary.
3. 3Choose expressiveness over machine efficiency when human-time costs more than computer-time: use a high-level, productivity-boosting language like Python instead of C++, accepting slower workloads, because the bottleneck is human throughput and organizational scalability.
4. 4Structure applications as workflows (a DAG of @step methods) instead of arbitrary Python classes, functions, and modules, so non-software-engineers can reason about the application as sequential steps — 'notebook cells on steroids' — rather than piecing together module relationships.
5. 5Make state transfer between steps nearly transparent so the framework handles persistence automatically (as in Metaflow, where an instance variable like x is persisted when a step finishes and reloaded before the next starts). This removes the boilerplate decision of 'what to persist and how.'
6. 6Structure the workflow around the logical shape of the application (separate steps for preprocessing and training), never merge steps just to dodge state-transfer boilerplate, and persist the maximum useful state so a production failure gives you maximum, not minimal, diagnostic information.
7. 7Integrate the layers of the stack into one cohesive user experience so the practitioner never has to understand and operate each layer independently — reduce infrastructure cognitive overhead so their limited cognitive bandwidth goes to data science itself.
8. 8Increase complexity only when it is absolutely clear the application requires higher scalability or performance, and then only proportionally to the demonstrated need.

Consider the state-transfer problem in figure 3.5, where the state is a single variable x, initialized in step A and incremented by steps B and C. In a framework that is unopinionated about persistence, each step carries explicit code to load x from a database and store it back — flexible, but it adds boilerplate and, worse, cognitive overhead, because the data scientist must decide what to persist and how. The observed failure pattern: conservative scientists, faced with that explicit choice, either pack many unrelated operations into one step to avoid the load/store boilerplate, or persist only the bare outputs a downstream consumer demands. Both choices degrade the application: the workflow structure no longer reflects the logical shape (preprocessing and training get merged), and when it fails in production you have minimal information to debug. Metaflow removes the choice: x is persisted automatically when a step finishes and reloaded before the next step starts, so state moves transparently and the scientist structures steps for readability, not for dodging boilerplate. The cognitive load shifts off infrastructure and back onto data science.

• If a complexity is not necessitated by the problem itself but is an artifact of your chosen approach, then treat it as incidental and eliminate it; if it reflects the real messiness of the world, then expose and manage it.
• If human-time is more expensive than computer-time (true for most data scientists), then choose the more expressive, productivity-boosting tool even at the cost of runtime efficiency.
• If a solution is fast but complex (hard for a newcomer to understand), then prefer the simple-but-slow alternative until scalability limits are actually hit — 'simple no obvious bugs' beats 'complex no obvious bugs.'
• If a complex interface between systems exists only to follow an organizational boundary (e.g., between data scientists and data engineers) rather than a technical necessity, then it is incidental complexity to be challenged.
• If you're tempted to merge steps or under-persist state to avoid boilerplate, then fix the framework's ergonomics instead — structure must optimize for logical clarity and maximum debug information.

• Treating all complexity as a nuisance to abstract away — the tell: you can no longer distinguish real problems from imaginary ones because inherent and incidental complexity are lumped together.
• Being too frugal with persisted state to avoid load/store boilerplate — the tell: a production failure leaves you with minimal diagnostic information about what went wrong.
• Packing unrelated operations into a single step to dodge state-transfer code — the tell: the workflow's step boundaries don't match its logical structure, so other readers can't follow it.
• Optimizing prematurely for speed with C++ or non-obvious clever code — the tell: the solution scales technically but no one else on the team can read or maintain it, so it fails organizationally.
• Building a one-size-fits-all high-level solution (models with one button click) — the tell: everyone can operate it but it's too inflexible for the variety of real projects and chokes on large data.

• ▸Component/interface name: the piece of infrastructure or code under review.
• ▸Complexity source: name the specific artifact (boilerplate / spaghetti pipeline / dependency hell / org-boundary interface).
• ▸Inherent or incidental?: is this required by the problem or by your chosen approach/org chart?
• ▸Cognitive cost: what decision or knowledge does it force onto the practitioner outside their core expertise?
• ▸Human-time vs computer-time: does this trade scarce human throughput for cheaper machine efficiency, or the reverse?
• ▸Newcomer test: can someone new read the source and quickly comprehend it? (yes/no)
• ▸Simpler alternative: the slow-but-simple version that still works.
• ▸Justification to add complexity: is it absolutely clear the app needs more scale/performance right now? (evidence required)
• ▸Owner + review date: who removes or defends this and when it's re-audited.

1. 1Measure burnout and deployment pain as first-class outcomes alongside delivery metrics, not as afterthoughts. Treat them as latent constructs measured through multiple survey items, the same way you measure culture, so a single noisy question does not mislead you.
2. 2Instrument your failure-notification sources separately. Distinguish notifications from outside automated processes (customer reports, the NOC) from proactive failure notification from systems (log alerts, threshold warnings like CPU exceeds 90%, rate-of-change warnings like CPU usage up 25% over 10 minutes). Track which source dominates.
3. 3Shift the notification mix toward proactive, system-generated alerts. Being told about failures first by customers or the NOC signals reactive firefighting; build logging, monitoring, threshold, and rate-of-change alerting so systems tell you before users do.
4. 4Apply continuous delivery practices to attack deployment pain at its root: build quality in so issues are caught when cheap to fix, and work in small batches so each deployment carries less risk and less anxiety.
5. 5Adopt technical practices proven to affect these human outcomes — version control, deployment automation, continuous integration, trunk-based development, test automation, test data management, and shift-left on security — since the research shows continuous delivery has a measurable impact on burnout and deployment pain.
6. 6Remove the change approval board as a control ritual. It does not improve delivery performance and is negatively correlated with tempo and stability, so it adds pain without buying safety.
7. 7Use the measures inside a learning culture, never a pathological or bureaucratic one. Set targets with teams for lead time, deployment frequency, time to restore service, and change fail rate rather than imposing numbers, because 'whenever there is fear, you get the wrong numbers.'
8. 8Re-run psychometric checks periodically. If a notification source changes behavior, your validity tests will catch the drift and alert you before your burnout and deployment-pain signals decay.

The research team wanted to measure failure notification and drafted five items: notified by customer reports, notified by the NOC, alerts from logging and monitoring systems, threshold warnings (CPU exceeds 90%), and rate-of-change warnings (CPU usage up 25% over 10 minutes). A pilot test with about 20 technical professionals showed all five loading together as one construct. But the larger, final data collection failed to confirm a single construct — statistical tests split them into two. The first two items measured 'notifications from outside automated processes,' and the last three measured 'proactive failure notification' from systems. Had they asked only one blanket question about monitoring for failures, they would never have learned that the source of alerts matters. The proactive failure notification construct turned out to be a technical capability predictive of software delivery performance — the difference between teams that firefight from customer complaints and teams whose systems warn them first.

• If customers or the NOC are your primary failure-notification source, then treat alert fatigue and firefighting as active risks and invest in system-generated proactive notification.
• If a deployment causes anxiety, then work in smaller batches and build quality in earlier rather than adding more inspection or approval gates.
• If you are considering a change approval board to reduce risk, then don't — it is negatively correlated with tempo and stability and does not improve delivery.
• If your organization's culture is pathological or bureaucratic, then fix the culture before deploying measurement, because fear produces the wrong numbers.
• If a notification or monitoring measure changes behavior, then re-run your validity checks before trusting the burnout/deployment-pain signals it feeds.

• Measuring burnout with a single survey question — the tell: your results feel binary and can't distinguish causes, because one item can't capture a latent construct.
• Conflating all failure alerts into one bucket — the tell: you count 'we monitor for failures' as a win while your team is still learning about outages from customers.
• Adding change approval boards to feel safer — the tell: tempo and stability drop while sign-off queues grow, yet failures don't decrease.
• Treating technical practices as secondary to management/team practices during an Agile adoption — the tell: burnout and deployment pain persist despite ceremony changes, because CI, test automation, and trunk-based development were never adopted.
• Using metrics as control in a fear-driven culture — the tell: numbers look clean but contradict what people privately report, because 'whenever there is fear, you get the wrong numbers.'

• ▸Primary notification source: customer reports / NOC / system alerts — name which dominates today
• ▸Proactive alert coverage: list logging, threshold (e.g. CPU>90%), and rate-of-change (e.g. +25%/10min) alerts in place
• ▸Batch size: describe typical size of a deployment and whether it can be split smaller
• ▸Quality-build-in gaps: where do you still rely on late inspection instead of catching issues early
• ▸CD capabilities present: check version control, deployment automation, CI, trunk-based dev, test automation, test data management
• ▸Change approval mechanism: is a CAB in place? note its effect on tempo/stability
• ▸Culture type: learning / bureaucratic / pathological — is it safe to publish these metrics
• ▸Target metrics with team: agreed targets for lead time, deployment frequency, time to restore, change fail rate
• ▸Re-check date: next scheduled validity review of your notification and burnout measures

1. 1Keep tactical planning models current on a fixed cadence so strategic pivots require minimal notice: update all production, DC, freight, and demand inputs to your optimization model at least every 6 months, as AO did with its PDCF model.
2. 2Build flexible internal DSS capability alongside turnkey commercial tools — reserve custom-built loops for the sudden, unforeseen problems (like Pfizer's overnight storage deficit) that no packaged tool anticipates.
3. 3Measure agility with the two SCOR adaptability KPIs: upside adaptability (maximum sustainable percent increase in quantity delivered achievable in 30 days) and downside adaptability (reduction in ordered quantity sustainable at 30 days prior to delivery with no inventory or cost penalty).
4. 4Structure your models so an existing tactical model can be re-scoped for a strategic question by swapping inputs — e.g., add candidate new DC locations, cost, and capacity estimates directly into the running PDCF model rather than building a new one.
5. 5Maintain hierarchical frameworks (operational/tactical/strategic) with feedback loops so any proposed new process or tool is evaluated against existing interfaces: ask what new communications it requires and whether it duplicates or truly improves a capability.
6. 6Use advanced analytics (mathematical optimization, stochastic simulation), not spreadsheets, so each iteration produces a defensible quantitative assessment of resource allocation rather than high-speed arithmetic.
7. 7Institutionalize the loop — the process, tools, and expertise must be permanent so agility produces a continuing stream of results, not a one-time save.

AO ran its supply chain on an agile hierarchical framework anchored by a tactical PDCF (production-DC-freight) optimization model holding production costs/rates/capacities by product family, DC costs and capacities, plant-to-DC and DC-to-region freight, and forecast demand by region. Because AO's planning team refreshed every input at least every 6 months, it always had current network economics on hand. When it became clear that DC storage and throughput capacity had to expand to meet rising demand, AO did not start a fresh study: it estimated costs and capacities of candidate new DC locations, added them to the standing PDCF model, and evaluated how each option would reshape the entire network. The currency and availability of the tactical model let AO launch a strategic network-expansion analysis rapidly, at any time, with minimal notice — while still running its scheduled annual strategic review. The iteration speed came from the discipline of maintenance, not from heroics during the crisis.

• If demand is unpredictable and supply is evolving, then choose an agile supply chain strategy — flexible enough to track shifting demand while minimizing supply disruption; if demand is predictable and supply stable, an efficient strategy suffices.
• If a sudden operational problem arises that no commercial DSS tool covers (e.g., overnight capacity deficit after a merger), then build a flexible internal custom DSS rather than forcing a turnkey tool.
• If tactical model inputs are older than 6 months, then refresh them before running any strategic scenario — stale inputs invalidate the rapid re-scoping capability.
• If a proposed new tool or process only duplicates an existing capability within your framework, then reject it; only adopt it if it truly improves a function or adds a needed interface.
• If the analytical question involves complex resource allocation, then use optimization or stochastic simulation, not a spreadsheet.

• Relying exclusively on turnkey commercial DSS tools — the tell: when an unforeseen event hits (a merger, a capacity shock), no tool fits and there is no internal flexible capability to improvise.
• Letting tactical models go stale between updates — the tell: a strategic request arrives and the team spends weeks re-gathering current costs and capacities instead of re-scoping in days.
• Making 'one-off' ineffective supply chain decisions with no framework — the tell: nobody can say how the change interfaces with existing operational/tactical/strategic activities or whether it duplicates something.
• Defaulting to spreadsheets for allocation problems — the tell: analyses are 'high-speed arithmetic with eye-catching graphics' that cannot handle stochastic or optimization complexity, so the answer fails in practice.
• Treating agility as a one-time crisis save rather than institutionalized capability — the tell: the process and expertise dissolve after the win and the next shock finds the firm unprepared.

• ▸Core tactical model name — the optimization model that would be re-scoped for strategic questions
• ▸Input refresh cadence — required update interval and date of last full refresh (target: <6 months)
• ▸Input inventory — list of live inputs (production, DC, freight, demand) and each one's last-updated date
• ▸Upside adaptability — maximum sustainable % increase in quantity deliverable within 30 days
• ▸Downside adaptability — % order reduction sustainable at 30 days prior with no inventory/cost penalty
• ▸Custom DSS gap list — unforeseen operational scenarios no turnkey tool covers, and who owns the internal build
• ▸Re-scoping steps — the specific edits needed to turn the tactical model into a strategic-scenario model
• ▸Framework interface check — for any proposed tool/process: new interfaces required, and duplicate-or-improve verdict

1. 1Separate your metrics into three tiers: technology metrics (component/application performance, availability), process metrics (CSFs, KPIs, activity metrics for service management processes), and service metrics (results of the end-to-end IT service). Only service metrics represent what the customer experiences.
2. 2Understand your business processes first and identify which ones are critical to delivering value to the business; anchor IT goals and objectives so they support those business goals directly.
3. 3Define what success looks like before you measure — state what you are trying to achieve and how you will know when you have achieved it (the 'Origins' step of service measurement).
4. 4Build service metrics by computing them from the underlying component metrics: identify which systems, applications and components the service depends on, monitor those, then roll them up into a single service-level view as the customer experiences it (see the Availability Reporting example on page 99).
5. 5For each measurement, define the CSF, the KPI, the metrics required, and the specific measurements that feed them — do not stop at the KPI. Use the four KPI questions (quality, performance, value, compliance) to test each indicator before implementing it.
6. 6Focus measurements on results and organizational output, not on efforts; get a clear idea of the contribution being made rather than counting activity for its own sake.
7. 7Report the true service level experienced by the customer, not a component/system/application report that lets you point blame elsewhere. Replace protective component reporting with a meaningful end-to-end service view.
8. 8Use the metrics as input to identify improvement opportunities, and remember service measurement looks at the future as well as the past so individuals can make operational, tactical or strategic decisions.

Take the qualitative example from the workbook: the CSF is 'Improving IT service quality' and the KPI is 'a 10% increase in customer satisfaction rating for handling incidents over the next 6 months.' To operationalize this as a customer-experience metric you need two required metrics — the original customer satisfaction score for handling incidents and the ending customer satisfaction score. Those metrics are fed by concrete measurements: the incident handling survey score and the number of survey scores collected. Notice this measures the service as the customer experiences it (their satisfaction with incident handling), not a component metric like server uptime. You capture a baseline score, run the improvement effort over six months, then re-survey and compare against the 10% target. If the ending score is up 10% or more, the KPI is met; if not, the survey data itself points to where handling still falls short.

• If a report only shows component, system or application status, then it is not a service measurement — add the roll-up to the true service level the customer experiences before you publish it.
• If a proposed KPI cannot answer at least one of quality, performance, value, or compliance, then discard or redefine it.
• If a metric measures effort rather than organizational output/result, then rework it to measure the contribution instead.
• If you are measuring at the component level to protect yourself or assign blame, then stop — refocus on providing a meaningful view of the IT service as the customer experiences it.
• If a business process is not critical to delivering value to the business, then deprioritize measuring the IT services supporting it and put activity where change and improvement matter most.

• Reporting component/application metrics (uptime, performance) and calling it service quality — the tell is a green dashboard while customers still report a broken experience.
• Stopping at the KPI without defining the required metrics and the underlying measurements — the tell is a stated target like '10% increase' with no baseline score and no data source to compute it.
• Measuring efforts instead of results (hours logged, tickets touched) — the tell is reports full of activity counts that no one can tie to a business contribution.
• Using measurement to protect oneself or point blame elsewhere — the tell is siloed reports each showing 'my component was fine' while the end-to-end service failed.
• Defining metrics inside one subject area and then benchmarking or interpreting them outside their valid domain — the tell is comparing specialized metrics across business units where only generic, aggregable metrics are valid.

• ▸Business process supported: name the critical business process this service delivers value to.
• ▸CSF: state the critical success factor in customer/service terms (e.g. Improving IT service quality).
• ▸KPI: state the target with magnitude and timeframe (e.g. 10% increase in customer satisfaction for incident handling over 6 months).
• ▸KPI question tested: mark which of quality / performance / value / compliance this KPI answers.
• ▸Metrics required: list the baseline and ending values needed to prove the KPI (e.g. original and ending satisfaction score).
• ▸Underlying measurements: list the raw data sources that feed the metrics (e.g. incident handling survey score, number of survey scores).
• ▸Component-to-service mapping: list the systems/applications/components whose metrics roll up into this service metric.
• ▸Result vs effort check: confirm the metric captures organizational output, not activity — note the contribution it demonstrates.
• ▸Decision use: state whether this measurement supports operational, tactical or strategic decisions.

1. 1Classify each product line before choosing a cost posture: functional products (stable demand, low margins) get a cost-efficiency supply chain; innovative products (unpredictable demand, high margins) get a responsive one. Cost efficiency is only correct where demand is predictable.
2. 2Build the annual plant/DC/family (PDCF) optimization model that outputs planned operating rates (capacity utilization) per plant, per family, per time period — utilization is the primary lever on cost per unit, so make it an explicit model output, not an afterthought.
3. 3Run each annual production plan through a family weekly production scheduling model to confirm feasibility at weekly buckets, including the cost impact of weekly changeovers, since plans that look efficient annually often turn infeasible (and more expensive) weekly.
4. 4Assign sourcing (sales regions to DCs and plants, plant-to-DC and inter-DC lanes) inside the same optimization so distribution cost is traded off against production utilization rather than optimized in isolation.
5. 5Install DSS at strategic, tactical, and operational levels and PMS metrics on every function to measure how efficiently the chain runs and flag emerging cost problems early — frameworks without these tools cannot deliver the savings.
6. 6Build feedback loops from operational back to tactical levels so realized cost and utilization data correct the higher-level plans, converting one-time savings into an institutionalized stream.
7. 7Quantify results in two currencies: operating cost reduction (percent of annual OpEx) and revenue loss avoided, so efficiency gains are defended against capacity-stress scenarios, not just steady-state.

A ceramic tile manufacturer implemented a hierarchical manufacturing and logistics planning framework spanning strategic, tactical, and operational horizons over a multiechelon network. By coordinating plant production plans, DC sourcing assignments, and operating rates through the framework, it reduced annual operating costs by approximately 10 percent while simultaneously improving customer service fill rates — showing efficiency and service are not automatically a trade-off. A separate confectionery division of a diversified consumer products and pharmaceutical firm ran a global manufacturing planning decision support system whose strategic and tactical components identified over $5 million in potential annual operational savings. A third firm, a large pharmaceutical and consumer health care company, used its existing planning frameworks and DSS to respond to a sudden stress on logistics capacity, maintaining very high fill rates and avoiding $20 million in potential revenue losses. Note the pattern: the savings came from the framework plus DSS/PMS acting together, and the authors stress institutionalizing the process so results continue rather than occur once.

• If demand is predictable and supply sources are stable, then pursue a cost-efficiency (process/functional logistics) strategy focused on controlling costs; if demand is unpredictable, do not optimize purely for cost.
• If a plan appears feasible at the annual level, then still validate it in weekly buckets including changeover impact before committing capacity — assume hidden infeasibility until proven otherwise.
• If a product enters the stable or decline phase, then look to extend it into a new geographic market where it re-enters growth, and transition its strategy from responsive back to cost efficiency once demand is understood.
• If you have strong frameworks but weak DSS/PMS (or vice versa), then expect failure — only the combination delivers efficiency gains.
• If efficiency savings are reported as a one-time project outcome, then reject them as incomplete; require the tools and expertise to be institutionalized for a continuing stream of results.

• Applying a cost-efficiency posture to innovative, high-margin products with unpredictable demand — the tell is chronic stockouts and lost margin despite low unit cost.
• Optimizing capacity utilization at a single plant without the multiechelon sourcing view — the tell is high local utilization but rising inter-DC shipping and total network cost.
• Trusting annual-level feasibility and skipping the weekly scheduling check — the tell is production plans that break during execution and force expensive expediting and changeovers.
• Buying DSS/PMS tools without the framework to organize them (or building frameworks with no supporting tools) — the tell is heavy managerial resource spend with minimal measurable results.
• Treating a savings project as done at go-live with no feedback loops — the tell is savings that erode within a year because plans stop being corrected by operational data.

• ▸Product classification: functional or innovative — determines cost vs responsive posture
• ▸Planned operating rate per plant, per family, per period — current vs target capacity utilization
• ▸Sourcing assignment: sales region -> DC -> plant — lane and cost per unit
• ▸Weekly feasibility flag: pass/fail after changeover-inclusive scheduling check
• ▸Annual OpEx baseline and target % reduction — e.g., 10% benchmark
• ▸Identified annual savings ($) — strategic + tactical components tallied
• ▸Revenue loss avoided ($) under capacity-stress scenario
• ▸DSS coverage: strategic/tactical/operational — present or gap
• ▸PMS metrics and feedback loop owner — who corrects the higher-level plan

1. 1Establish the two architectural tests you must satisfy: (1) 'We can do most of our testing without requiring an integrated environment' and (2) 'We can and do deploy or release our application independently of other applications/services it depends on.' Survey your teams against both statements; agreement predicts high performance.
2. 2Audit cross-team dependencies by asking whether a team can make large-scale changes to their system's design without permission from anyone outside the team and without creating significant work for other teams. Every 'no' is a coupling defect to remediate.
3. 3Measure communication load: track how much a delivery team must communicate and coordinate with people outside their team to complete work. In loosely coupled teams little such communication is required — treat high external coordination as the coupling signal.
4. 4Bring integrated-against custom software in-house where you're a low performer relying on custom software developed by another company; the research ties that external dependency to low performance.
5. 5Grant teams the authority to choose their own tools based on what best supports their work, rather than mandating an approved list, since tool choice is a distinct contributor to delivery performance.
6. 6Enable on-demand deployment: teams should deploy and release their product or service regardless of the other services it depends upon, and perform deployments during normal business hours with negligible downtime.
7. 7Validate the payoff by tracking deploys per day per developer as you add people — the target is that individual deploy frequency holds constant or increases with team size rather than degrading.

In the 2017 survey the authors expanded their analysis to test whether a loosely coupled, well-encapsulated architecture drives IT performance, and it did — it was the biggest single contributor to continuous delivery, larger even than test and deployment automation. The measured behaviors were concrete: teams making large-scale design changes without outside permission, without depending on other teams, deploying on demand regardless of dependencies, and testing on demand without an integrated environment. To see the scaling effect, they computed deploys per day per developer for respondents deploying at least once per day (Figure 5.1). Low performers deployed with decreasing frequency as developers were added; medium performers held constant; high performers deployed at significantly increasing frequency. This contradicts the orthodox view that individual productivity must fall due to communication and integration overhead. The difference is architecture: loosely coupled systems let teams test and deploy individual components even as the number of systems grows, so the business moves faster as it adds people.

• If a team cannot deploy or release independently of the services it depends on, then treat the architecture as tightly coupled and prioritize decoupling over adding CD tooling.
• If most testing requires an integrated environment, then testability is failing and you cannot validate components independently — fix this before scaling the team.
• If large-scale design changes require permission or create significant work for other teams, then you have an organizational coupling problem, not just a code problem — realign team boundaries with system boundaries.
• If deploys per developer per day fall as you add people, then you are scaling like a low performer — stop hiring into the bottleneck and remediate coupling first.
• If you are a low performer integrating against custom software developed by another company, then bring that capability in-house.

• Investing in test and deployment automation while leaving the architecture tightly coupled — the tell: automation exists but teams still need permission or integrated environments to change anything.
• Treating loose coupling as purely a code property while ignoring team coupling — the tell: little code sharing but constant cross-team meetings to coordinate releases.
• Mandating an approved tool list to reduce environment complexity and boost vendor purchasing power — the tell: teams cannot experiment with approaches best suited to their problems, and delivery performance stalls.
• Assuming DevOps/CD only works for greenfield web systems and giving up on mainframe, firmware, or big-ball-of-mud environments — the tell: no attempt to decouple legacy systems despite the finding that high performance is possible with all system types when loosely coupled.
• Adding developers expecting linear speedup without decoupling — the tell: deploy frequency per developer flattens or drops as headcount rises, revealing communication and integration overhead.

• ▸Team/System name: the delivery team and the system it owns.
• ▸Independent deploy: can you deploy/release independently of dependent services? (yes/no).
• ▸Independent test: can you do most testing without an integrated environment? (yes/no).
• ▸Change without permission: can you make large-scale design changes without outside approval? (yes/no).
• ▸Change without burdening others: can you change design without depending on or creating work for other teams? (yes/no).
• ▸External coordination level: how much cross-team communication is needed to finish work? (low/medium/high).
• ▸Tool autonomy: can the team choose its own tools for this work? (yes/no).
• ▸Deploys per developer per day: current measured value and trend as team grows.
• ▸Top coupling defect + remediation owner: the single dependency to break next and who owns it.

1. 1Instrument technical debt as fitness functions, not opinions: pick metrics like cyclomatic complexity, code coverage, relative cyclicity, and SDI, then attach a threshold each metric must stay above or below.
2. 2Build a broad base of triggered atomic fitness functions in your CI/CD pipeline — static code analysis (cyclomatic complexity) and coverage — because they are cheap, well-established, and easy to run on every developer action.
3. 3Do not maximize the base: unlike unit tests, each fitness function adds overhead, so keep the bottom layer broad but deliberate, and add middle/holistic layers only where context justifies the cost.
4. 4For cyclic dependencies, weight each link (lower weight = preferred cut), compute SDI = 10 * numberOfLinksToCut + Σ weightOfLinksToCut per cycle group, then accumulate to module and system level.
5. 5Track SDI together with relative cyclicity over time; set the goal near zero and watch the trend, since continuous growth is the actual debt signal, not the absolute value at any single point.
6. 6Assess architectural debt against the three cognitive principles — modularity (chunking), hierarchy (building hierarchies), and pattern consistency (building schemata) — because architectures that follow them are perceived as understandable and are cheaper to maintain and extend.
7. 7Frame findings for non-IT budget managers as an accumulating cost: pile up debt and every adjustment becomes an incalculable cost screw, so continuous work is required across the whole service life to keep debt low.

Take the passage's cycle example: 10 components arranged in a simple cycle. To break the cycle you only need to cut one link. Assuming that link has a weight of 1, SDI = 10 * 1 + 1 = 11. The constant multiplier (10) reflects that someone must figure out how to break each dependency — for instance by inverting it using the dependency inversion principle — and each usage may add further effort. Accumulate this per cycle group up to module and system level, and let a tool like Sonargraph-Architect rank cycle groups by how easy they are to fix. Pair the SDI value with relative cyclicity and drive both toward zero; a continuously rising SDI is the tell that code cancer is developing.

• If a metric only checks a limited part of the system (e.g., cyclomatic complexity via static analysis), then classify it as atomic and place it in the cheap bottom layer; if positive verification means end users can use the system as intended, then it is holistic and belongs higher up.
• If you are choosing which cyclic dependency to break, then prefer cutting the lowest-weight links so the algorithm minimizes SDI and rework.
• If SDI stays near zero, then debt is contained; if it grows continuously, then treat it as a spreading structural problem requiring intervention now.
• If a metric must be evaluated against live production behavior (e.g., service response time), then make it a continual fitness function under system monitoring, not a triggered CI check.
• If you are reporting to budget managers, then express debt as rising future maintenance cost and the need for continuous upkeep, since that framing is comprehensible to non-IT stakeholders.

• Treating the fitness-function base like unit tests and creating as many as possible — the tell is mounting pipeline overhead and slow runs with no proportional insight gain.
• Reading SDI or complexity as a one-time snapshot instead of a trend — the tell is a 'passing' number today while the trajectory is climbing (code cancer) unnoticed.
• Measuring only atomic metrics and assuming the whole system is healthy — the tell is green static analysis while end users still can't complete workflows, which only holistic functions would catch.
• Ignoring the cognitive principles (modularity, hierarchy, pattern consistency) and chasing raw numbers — the tell is metrics that pass but an architecture developers still describe as hard to understand and maintain.
• Letting debt accumulate silently until every change is expensive — the tell is that small adjustments turn into an incalculable cost screw and both developers and management stop enjoying the system.

• ▸Metric name: the debt indicator (e.g., cyclomatic complexity, relative cyclicity, SDI, coverage)
• ▸Atomic or holistic: how much of the system this feedback covers
• ▸Trigger type: triggered (CI action/nightly) or continual (production monitoring)
• ▸Threshold: the value the metric must stay above/below to pass
• ▸Current value & trend: latest reading plus direction over time
• ▸Cognitive principle served: modularity, hierarchy, or pattern consistency
• ▸Cycle/SDI detail: links to cut, their weights, computed SDI per group
• ▸Owner & remediation: who fixes it and the intended break technique (e.g., dependency inversion)

1. 1Start from the business processes, not the technology: identify which processes are critical to delivering value to the business, then derive IT goals and objectives that support those business goals and objectives.
2. 2Define what success looks like before you measure: state what you are trying to achieve and how you will know when you have achieved it (the Origins step of service measurement).
3. 3Translate each business goal into a Critical Success Factor, then a KPI, then the specific metrics and measurements required — keep the chain explicit so every measurement traces back to a business objective.
4. 4Categorize each KPI against the four questions metrics can answer: quality, performance, value, and compliance; discard measures that map to none of these.
5. 5Set targets for every measure via SLA or OLA agreed internally within IT, ensuring the target level is what the business actually requires end-to-end, not just component performance.
6. 6Measure and report the true service level as experienced by the customer (end-to-end), computing service metrics from component metrics rather than reporting only against a component, system, or application.
7. 7Assign roles and responsibilities for each measure (who does what), define supporting policies, and decide criteria for continual improvement including when targets should be raised.
8. 8Where using Six Sigma, deploy it across all areas — product development, R&D, order fulfillment, back-office — to align resources against critical business problems and the organization's strategic objectives and key performance measures.

Take the qualitative CSF 'Improving IT service quality.' You derive a KPI of a 10% increase in customer satisfaction for handling incidents over the next 6 months. The metrics required are the original customer satisfaction score for handling incidents and the ending score; the measurements are the incident handling survey score and the number of survey scores. Contrast with the high-level goal 'Manage availability and reliability of a service': its KPI is percentage improvement in overall end-to-end availability, categorized as Value and Quality, measured as end-to-end service availability built from component availability (AS/400 availability, application availability) against a 99.995% target. Ownership is assigned to Technical Managers, Technical Analyst, and the Service Level Manager. In both cases the measure exists only because a stated goal demands it, and the availability example deliberately rolls component metrics up into the service the customer actually experiences.

• If a proposed metric does not answer one of the four questions — quality, performance, value, or compliance — then it does not belong in the framework.
• If IT goals and objectives do not demonstrably support the business goals and objectives, then realign or drop them before measuring anything.
• If you are reporting only against a component, system, or application, then you have not yet reported the true service level the customer experiences — build the end-to-end service metric first.
• If a metric is specialized to one subject area, then do not benchmark or interpret it outside that domain; use generic metrics when you need to aggregate across business units.
• If a strategic plan cannot be linked to operational implementation nodes, then expect the 50-75% benefit-failure rate — ground it in operational practicality via business process modeling before committing.

• Reporting component health instead of the customer-experienced service level — the tell is dashboards full of green component uptime while customers still complain about the service.
• Treating strategic analysis on too broad a scale without focus on implementation issues — the tell is excellent plans that never deliver expected benefits (quoted failure rates of 50-75%).
• Calling procedures 'business processes' — the tell is that your 'process' describes internal task steps rather than a sequence of linked activities that recognizes a need and delivers a solution.
• Choosing metrics that create negative behavior or that aren't relevant to meeting objectives — the tell is people optimizing the number rather than the outcome, and no improvement opportunity emerging.
• Adopting Six Sigma or a measurement program in isolated pockets rather than aligning it throughout the organization to strategic objectives — the tell is improvement projects that solve local problems but never touch critical business problems or strategic key performance measures.

• ▸Critical business process: name the process critical to delivering value to the business
• ▸Business goal/objective: state the overarching goal this process serves
• ▸Supporting IT goal (CSF): the IT objective explicitly derived from the business goal
• ▸KPI: the specific indicator (e.g., 10% increase in satisfaction over 6 months)
• ▸KPI category: mark one of quality / performance / value / compliance
• ▸Metrics required: list the quantities needed to compute the KPI
• ▸Measurements: name the raw data points and their source (end-to-end, not component-only)
• ▸Target and source: the target value and the SLA/OLA it is agreed against
• ▸Owner / how: role responsible and method (who does what)

1. 1Demonstrate commitment in actions, not words: provide the three concrete resources the research names—time, actions, and resources—consistently over time rather than through a single announcement. A memo does not count as support.
2. 2Reject the top-down mandate model. Kissler names 'trying a top-down mandate to adopt Agile' as a common trap; instead treat the change as building a learning organization that is 'never done,' not a program with an end date.
3. 3Model 'honoring and extracting reality': create conditions where teams feel comfortable sharing risks and failures. If your team won't share risks with you, you never see reality, so make failure lead to inquiry (per the Westrum model) rather than blame.
4. 4Show up as genuinely curious continuously, not only when something breaks. Kissler states that a leader who only appears at failures is failing as a senior leader—so build trust through routine presence.
5. 5Limit the blast radius: start with one to two teams rather than 'boiling the ocean.' Use these early teams to generate the external and internal examples you need to influence skeptics.
6. 6Align organizational values with individual lived values to fight burnout—remember it is the everyday, lived values employees feel that count, not mission-statement placards.
7. 7Use data to drive actions and decisions: know cycle time via a value stream map, and consolidate into a single backlog because NFRs are features and reducing technical debt improves product stability.
8. 8Persist across methodologies and environments. The guidance is methodology-agnostic and applies to mainframe, packaged-software, and product teams, so refuse the 'we're different' exemption.

Courtney Kissler's Nordstrom transformation, begun summer 2011, is the worked case. The IT organization was optimized for cost, and her 'aha' moment was shifting to optimizing for speed as digital became the growth engine. She stepped in the common traps—top-down Agile mandate, treating it as one-size-fits-all, not measuring the right things, leadership behavior not changing, and running it as a program instead of a learning organization. Recovering, she moved to outcome-based team structures, mapped the value stream to know cycle time, limited the blast radius to one or two teams, used data to drive decisions, and consolidated to a single backlog. She modeled 'honoring and extracting reality' and made failure lead to inquiry, and when skeptics claimed 'Of course this worked for the mobile app team—they are a unicorn,' she used external examples and leaned on mentors to influence the discussion. The result: teams practicing these techniques earned the highest eNPS scores across the technology organization.

• If your commitment is expressed only in words or placards, then it does not count—provide time, actions, and resources, because lived values are what employees feel.
• If you consider a top-down mandate to force adoption, then stop—build a learning organization instead and start with one to two teams.
• If your team does not surface risks or failures to you, then treat it as a trust failure on your part; make failure lead to inquiry so you can see reality.
• If skeptics dismiss a success as a 'unicorn,' then bring external examples and mentors into the discussion rather than arguing from authority.
• If organizational and individual values are misaligned, then expect burnout in demanding technical work; realign values before pushing harder.

• Support in words, not actions. Tell: leadership issues statements or mission placards but provides no time, actions, or resources—employees' lived experience contradicts the stated values.
• Top-down mandate. Tell: an executive orders 'adopt Agile' as one-size-fits-all across all teams at once instead of starting small and adapting.
• Treating the transformation as a finite program. Tell: there is a completion date and a project close-out rather than an ongoing, 'never done' learning organization.
• Absentee curiosity. Tell: the leader only shows up when there's a failure and isn't genuinely curious otherwise, so risks stay hidden and reality is never known.
• Not measuring, or measuring the wrong things. Tell: no value stream map, unknown cycle time, and separate backlogs for features, technical debt, and operational work instead of a single backlog.

• ▸Resource commitment—list the specific time, actions, and resources you have provided this quarter (not words).
• ▸Lived vs. stated values—note where employees' felt values diverge from official statements.
• ▸Blast radius—name the one to two teams you are starting with.
• ▸Risk-sharing signal—record the last risk or failure a team surfaced to you unprompted.
• ▸Curiosity cadence—log your non-failure touchpoints with teams this month.
• ▸Measurement baseline—state current cycle time from your value stream map.
• ▸Backlog status—confirm a single backlog (features + tech debt + ops) yes/no.
• ▸Skeptic response—capture the external example or mentor you used to influence a specific skeptic.
• ▸Outcome signal—track eNPS or equivalent for teams practicing these techniques.

1. 1Measure your current position with the six Westrum Likert-type questions (Figure 3.1), each scored 1–7. Have the team answer, then take the mean across all questions to produce a single culture score you can track over time and analyze statistically.
2. 2Read the score against Table 3.1's three types: are messengers 'shot,' neglected, or trained? Is cooperation low, modest, or high? Is failure met with blame or inquiry? This tells you whether you sit at pathological, bureaucratic, or generative.
3. 3Act your way to a better culture by implementing the concrete technical and management capabilities, not by running a 'culture program.' The passages assert there is 'no mystical culture magic'—improving practices like continuous delivery and lean management drives Westrum culture upward (Figure 3.3, drivers).
4. 4Engineer information flow to meet Westrum's three tests: information answers the receiver's actual question, arrives timely, and is presented so the receiver can use it. Audit your dashboards, incident reports, and status updates against these three characteristics.
5. 5Make failure blameless and decisions reversible: keep the team open and transparent rather than closed and hierarchical, so bad decisions get surfaced and reversed quickly. Discover and address problems rapidly instead of protecting turf.
6. 6Protect experimentation by limiting the blast radius—start with one or two teams rather than boiling the ocean—and let teams freely test and discard hypotheses. Pair this with a single unified backlog (features, tech debt, and operational work together) so NFRs get treated as real work.
7. 7Have senior leaders demonstrate commitment in actions, not words. As Courtney Kissler models: 'honor and extract reality'—if the team doesn't feel comfortable sharing risks, you never learn the truth, and generativity collapses.
8. 8Re-survey on a cadence, watch the trend, and correlate it against outcomes (software delivery performance, organizational performance, job satisfaction) so culture change is data-driven, not asserted.

Courtney Kissler's Nordstrom journey (2011) shows the norm in practice. Early on the team fell into common traps: a top-down Agile mandate treated as one-size-fits-all, no measurement of the right things, leadership behavior not changing, and running the transformation as a finite 'program' instead of building a never-finished learning organization. They corrected course by moving to outcome-based team structures, understanding cycle time through value stream mapping, limiting the blast radius by starting with one to two teams, and unifying backlogs so technical debt and operational work counted as features. It took heavy experimentation and adjusting along the way—no single change happened overnight. The proof surfaced in the data: when they measured eNPS, the teams practicing these techniques had the highest scores in the whole technology organization. Kissler ties it directly to leadership behavior: a senior leader who can't make people comfortable sharing risks will never surface reality.

• If messengers who bring bad news are punished or ignored, then you are pathological or bureaucratic—fix the response to bad news before expecting information to flow.
• If following the rules is treated as more important than achieving the mission, then you are bureaucratic; if rules exist to guarantee fairness and encode accumulated expertise while the mission stays primary, that same rule-following can still be generative.
• If a decision turns out wrong and the team resists reversing it, then treat that friction as a culture signal—open, transparent teams reverse bad decisions easily; closed hierarchical ones don't.
• If someone proposes a 'culture change program' with no accompanying practice changes, then redirect: you act your way into culture via the 24 concrete capabilities, not the reverse.
• If you're introducing experimentation, then start with one or two teams to bound the blast radius rather than mandating across the whole org.

• Top-down Agile mandate as one-size-fits-all. Tell: leadership issues the directive but their own behavior never changes, and teams comply on paper only.
• Treating transformation as a finite program. Tell: there's an end date and a 'done' state—the passages flag this as 'never done,' because the goal is a continuous learning organization.
• Splitting work into separate backlogs (features vs. tech debt vs. ops). Tell: technical debt and operational work perpetually lose to features because they aren't counted as real work; NFRs get starved.
• Leaders supporting the culture in words but not actions. Tell: teams don't feel safe sharing risks, so reality is never extracted and problems surface late.
• Skipping measurement or measuring the wrong things. Tell: culture is asserted as 'good' with no Westrum score, no eNPS, and no value-stream cycle-time data to confirm it.

• ▸Team name & survey date: identify the unit and the point-in-time snapshot.
• ▸Westrum mean score (1–7): average of the six Likert questions across all respondents.
• ▸Messenger response test: how was the last piece of bad news treated—shot, neglected, or trained?
• ▸Information flow check: does recent key info answer the receiver's question, arrive timely, and get presented usably? (Y/N each)
• ▸Reversibility check: name one recent decision reversed after new evidence—if none, note why.
• ▸Blast-radius scope: which one or two teams are running the current experiment?
• ▸Single-backlog audit: are features, tech debt, and ops work in one prioritized backlog? (Y/N)
• ▸Leader action-not-words item: one behavior a senior leader will visibly model this cycle to make risk-sharing safe.
• ▸Outcome metric to re-check: eNPS / delivery performance figure to compare against next pulse.

1. 1Design processes for your organization first, then design metrics to measure them, then decide how to implement them in software — starting from the software's built-in metrics lets the tail wag the dog and staff end up owning goals the vendor chose.
2. 2Assign every metric to exactly one named individual who holds the authority to influence its outcome; never split one metric across two owners, because if two are responsible neither is. Test each assignment against the Canute rule: does this person actually control the tide?
3. 3Distinguish department ownership from process ownership. Give department heads (Service Desk, ICT Infrastructure Management) metrics for departmental efficiency plus the processes their department runs (e.g. Service Desk manager owns Incident Management metrics); for processes that span departments end-to-end, appoint a process owner who may have no direct reports.
4. 4Work with Human Resources to rewrite job descriptions and performance evaluations so performance is expected, measured, and rewarded against the appropriate metrics — then communicate continuously exactly what each metric means, what behavior it rewards, and how achievement will be rewarded.
5. 5Make metrics visible: put current-performance graphs prominently in the IT department the way factory floors display them, and run weekly or monthly staff meetings focused on relative process performance, rewarding teams exceeding targets.
6. 6Route front-line staff into the Continuous Service Improvement Programme (SIP) — solicit their improvement suggestions and reward effective ones handsomely, since a process improvement beats buying more people or computers.
7. 7When a red-light item appears, ask the responsible staff to suggest fixes to prevent recurrence and frame it as an improvement opportunity, not grounds to blame an individual.
8. 8Include delivery-team staff when negotiating or changing metrics so they can visualize impact, flag unrealistic targets before they are imposed, and offer collaborative ways to meet challenges rather than resenting top-down mandates.

Consider the Service Desk, one of the two departmental entities every IT organization has. Its department head is made the process owner responsible for the Incident Management process metrics, satisfying the single-owner rule and the Canute test because that manager has the authority to influence incident resolution. Separately, an administrative department that produces both ITIL process reports and SLA compliance reports gets metrics testing the accuracy and timeliness of those reports, owned by its manager, plus ownership of the communication plan and its own success measures. On the floor, current KPI performance graphs are posted prominently and reviewed in the monthly staff meeting, where the team beating its incident-resolution target is publicly rewarded. When an incident-backlog metric turns red, staff are asked how to prevent recurrence instead of being blamed — and one operator's SIP suggestion improves the process, delivering the service gain that would otherwise have required more headcount.

• If two people are responsible for the same metric, then reassign it to one — because neither is actually responsible and each can block the other's result.
• If a candidate owner cannot influence the metric's outcome, then do not assign it to them (the Canute rule); find someone with the necessary authority.
• If a process runs end-to-end across multiple departments, then appoint a designated process owner rather than a departmental manager, even if that owner has no direct reports.
• If performance produces a red-light item, then treat it as an improvement opportunity and ask staff for prevention methods — do not use it to blame individuals.
• If you are about to change or tighten a metric, then involve the delivery team first so unrealistic requests get averted collaboratively instead of resented.

• Relying only on the metrics the software tool ships with — the tell is that reports describe what the vendor's designer thought mattered, not what business-IT alignment demands.
• Rewarding fire-fighters for last-minute daring saves while ignoring those who prevent crises — the tell is celebration of heroic recoveries and no conspicuous recognition of money saved by defusing future problems.
• Splitting metric ownership or leaving it vague — the tell is that when a red item appears, no single person is accountable and actions of different owners work against each other.
• Imposing new or tighter metrics top-down without consulting delivery teams — the tell is resentment and resistance rather than suggestions, and targets that turn out unrealistic in practice.
• Treating morale, staff turnover and staff engagement as too intangible to bother measuring — the tell is well-defined metrics but service levels below what they ought to be because staff are not satisfied or engaged.
• Culturally rewarding managers of large teams over those who deliver great value with few — the tell is process owners avoiding efficiency gains because shedding resource looks like lost status.

• ▸Metric name: the exact, versioned name (note if changed from prior period)
• ▸Process or department measured: which ITIL process or departmental entity it covers
• ▸Single responsible individual: one named person, no shared ownership
• ▸Authority/influence check: how this person can actually move the outcome (Canute test)
• ▸Owner type: departmental manager or designated process owner (with/without direct reports)
• ▸Rewarded behavior: what the metric is meant to steer staff toward
• ▸Reward mechanism: how achievement is recognized (meeting, visible graph, incentive)
• ▸Visibility location: where the current-performance graph is displayed for the team
• ▸Red-item response protocol: who is engaged and how prevention (not blame) is captured into the SIP

1. 1Establish the causal chain you're managing: software delivery performance drives organizational performance (profitability, productivity, market share) — and for non-commercial contexts, effectiveness, efficiency, and customer satisfaction. Measure delivery, then trace it to business outcomes.
2. 2Measure software delivery performance first, using the four established metrics implied by the tempo/stability dimensions: lead time, deployment frequency (tempo), plus change fail rate and time to restore (stability). Use Likert-type survey items (Strongly Disagree to Strongly Agree) rather than yes/no to capture nuance.
3. 3Drive improvement through the 24 capabilities across five categories — continuous delivery, architecture, product and process, lean management and monitoring, and cultural — rather than chasing a maturity-model level. Pick capabilities that are currently constraining your delivery.
4. 4Reject the maturity-model framing; treat improvement as continuous iteration, incremental change, and revalidation. Re-measure after each intervention to confirm the effect holds in your context.
5. 5Add human-capital measures alongside business metrics: eNPS (employee Net Promoter Score), work identity, and burnout. Track whether the teams practicing the capabilities also show higher satisfaction, because sustained performance depends on people.
6. 6Limit the blast radius: start with one or two teams instead of boiling the ocean, prove the outcome, then expand. Move toward outcome-based team structures and a single unified backlog where NFRs and technical-debt reduction are treated as features.
7. 7Change leadership behavior explicitly — transformational leadership is one of the 24 capabilities and measurably impacts technical and lean capabilities. Do not run this as a time-boxed program; build a learning organization instead.

Courtney Kissler's Nordstrom journey (starting 2011) shows the construct in practice. Nordstrom made a strategic decision to treat digital as the growth engine, which forced a shift from an IT organization optimized for cost to one optimized for speed. Kissler names the traps that broke the business-value linkage: a top-down Agile mandate, treating transformation as one-size-fits-all, not measuring the right things, unchanged leadership behavior, and running it as a program rather than a never-finished learning organization. What worked was understanding the value stream map to know cycle time, limiting the blast radius to one or two teams, using data to drive decisions, and consolidating features, technical debt, and operational work into a single backlog. She applied these across mainframe, packaged-software, and product teams — evidence the outcomes are methodology-agnostic. As proof the performance and people gains moved together, the teams practicing these techniques scored highest on eNPS across the organization.

• If someone claims high IT performance and strong business performance, then demand the underlying analysis and methods — correlation dressed as science is a tell for 'bogus twaddle' (Fowler's test); trust it only when the reasoning holds.
• If you're tempted to pursue a maturity-model 'level,' then switch to selecting specific capabilities that relieve your current delivery constraint — the evidence exposes maturity models as a myth.
• If you want to improve culture, then implement the concrete technical and lean practices, because you can act your way to a better culture rather than waiting for culture magic.
• If a transformation is being run as a bounded program with an end date, then reframe it as building a learning organization — the work is 'never done.'
• If you maintain separate backlogs for features, technical debt, and operational work, then merge them into one, because NFRs are features and reducing technical debt improves product stability.

• Top-down Agile mandate treated as one-size-fits-all — the tell: leadership hands down a framework, team context is ignored, and adoption stalls.
• Not measuring, or measuring the wrong things — the tell: you can't state your cycle time or trace delivery to profitability/market share, so improvements are anecdotal.
• Leadership behavior never changes while teams are told to transform — the tell: managers demand new practices but keep old command patterns; transformational leadership scores stay low.
• Boiling the ocean instead of limiting blast radius — the tell: the rollout targets all teams at once, failures are large and un-contained, and momentum collapses.
• Optimizing solely for cost when the strategy needs speed — the tell: digital is declared the growth engine but the IT org's incentives and structure still reward cost reduction, starving the strategy.

• ▸Strategic outcome: the business/mission result you're optimizing for (profitability, market share, effectiveness, customer satisfaction).
• ▸Delivery metrics baseline: current lead time, deployment frequency, change fail rate, time to restore.
• ▸Constraining capability: which of the 24 capabilities most limits delivery right now, and its category.
• ▸Pilot scope: the one or two teams and value stream you'll start with (blast radius).
• ▸Value stream cycle time: measured end-to-end cycle time from the value stream map.
• ▸Single backlog check: confirm features, technical debt, and operational/NFR work are unified (Y/N).
• ▸Leadership behavior change: the specific transformational-leadership behaviors leaders will alter.
• ▸People metrics: current eNPS, work-identity, and burnout readings for pilot teams.
• ▸Revalidation date: when you'll re-measure delivery and business/people outcomes to confirm the effect.

1. 1Separate the four scalability dimensions before choosing a solution: technical scalability (bigger data/algorithms), performance (speed), and organizational scalability (understandable by the most people). You can have one without the others, so name which you are optimizing.
2. 2Default to simplicity as your organizational-scalability tool. When you evaluate a proposed approach, ask whether the average team member can read, run, and modify it — not whether the most experienced engineer can. Reject the highly optimized C++ solution if only one person understands it.
3. 3Separate 'what, how, and where' into independent layers so teams don't collide: the architecture layer (business logic the data scientist owns), the versioning layer (infrastructure-owned, uniform for everyone), the job scheduler (topological execution), and the compute layer (where tasks land).
4. 4Make the infrastructure opinionated about versioning and permissive about architecture. Place the architecture layer at the top of the stack (least constraining, per-project freedom); enforce one shared versioning approach so concurrent prototypes and deployments don't conflict (the Alex/Finley conflict scenario).
5. 5Hunt for incidental complexity introduced by organizational boundaries — complex interfaces between data scientists and data engineers that exist for org-chart reasons, not technical ones. Remove interfaces that follow boundaries rather than need.
6. 6Provide a toolbox of good-enough scalability options (@resources, @batch, @retry, @catch, @timeout) rather than one mandatory framework, so a wide variety of projects fit without over-engineering.
7. 7Standardize the environment to reduce debugging overhead as headcount grows: use cloud instances over per-person laptops so environments are homogeneous, monitorable, and remotely supportable by one support person using standard tools.

Alex builds a workflow estimating cupcake delivery times, ingesting all historical orders from the company data warehouse. Loading the data takes longer than building the model itself — the bottleneck is that ML workflows were forced through the same data path built for dashboards. Bowie recognizes this and builds a separate fast data path for ML access. The result is organizational, not just technical: Alex can now train and test at least 10 model versions daily instead of 2. The point is that removing the incidental interface (reusing the dashboard path) multiplied one person's iteration throughput without adding coordination overhead. The fix scales because any data scientist on the team uses the same fast path with the same simple tooling.

• If only one person on the team understands a solution, then it is not organizationally scalable regardless of its technical performance — reject or document/simplify it.
• If a proposed interface exists to satisfy an org boundary rather than a technical constraint, then treat it as incidental complexity and remove it.
• If the choice is between letting each project decide (architecture) versus enforcing a shared standard (versioning), then make architecture free and versioning uniform.
• If the concern is 'more people, more projects' rather than 'more data, faster,' then reach for simplicity first, not more optimization.
• If updates are needed no more than every ~15 minutes, then model it as a batch workflow and avoid the added complexity of streaming.

• Optimizing technical performance while destroying organizational scalability — the tell: a fast solution that only its author can maintain, and PRs stall waiting on that one person.
• Mistaking inherent complexity for incidental, or vice versa — the tell: teams argue about 'imaginary problems' (boilerplate, spaghetti pipelines) while the real domain complexity goes unmanaged.
• Letting each data scientist run their own laptop environment — the tell: 'works on my machine' bugs that are hard to reproduce, monitor, or support remotely as the team grows.
• Over-engineering to look scalable — the tell: complex infrastructure added before any project needs it, consuming cognitive bandwidth that limited teams can't spare.
• Coupling ML workflows to a data path built for another purpose — the tell: data loading dominates iteration time (like Alex's warehouse load exceeding model build time), throttling per-person throughput.

• ▸Project/component name: what you are evaluating
• ▸Scalability dimension targeted: technical / performance / organizational (pick and justify)
• ▸Comprehension breadth: how many current team members can run and modify this?
• ▸Incidental complexity found: boilerplate, spaghetti, dependency hell, or org-boundary interface?
• ▸Layer ownership: is architecture free per-project and versioning uniform for all?
• ▸Environment homogeneity: laptop-specific or shared cloud instance?
• ▸Toolbox fit: does an existing robust decorator/pattern cover this, or is a bespoke build needed?
• ▸Simpler alternative: the simplest version that is still adequate ('as simple as possible, but no simpler')