The Art of Capacity Planning Scaling Web Resources in the Cloud

1. 1Segment the architecture so each backend role (database, web server, cache) can be tested and measured on its own. The source says an easily segmented architecture helps you find the limits of the current hardware configuration.
2. 2Find the failure point for each role under its specific hardware configuration. Ask how many queries per second (QPS) it can manage, and how many it can serve before performance degradation affects end-user experience. Treat the point of failure as a mandatory input to the calculation.
3. 3Define the ceiling as the load at which the resource becomes unresponsive, fails over, or degrades the user experience. Record it as a 'red line' metric per resource class: CPU, disk, memory, network bandwidth, and whole hosts on very large sites.
4. 4Measure the typical peak utilization from your own graphs. Use the spikes you have actually seen, because the source says to base safety margins on observed spikes of usage.
5. 5Subtract a comfortable percentage (the safety factor) from the ceiling. Adjust the ceiling value for periodic spikes before you forecast, so the forecast uses the reduced ceiling and not the theoretical maximum.
6. 6Pick the margin per resource, not one number for everything. The margin is an educated guess. Caching systems tolerate spikes better than other resources, so you can be less conservative there.
7. 7Set alert thresholds from the red-line number. Use it to predict what adding or removing similar servers does, and to decide when to begin sizing the next order of capacity.
8. 8Re-check headroom at the bottleneck whenever the architecture or traffic changes. A new release, a lost server, a changed microservice dependency, or an event such as the Super Bowl can shift the bottleneck and shrink headroom. If you reallocate headroom to drain a queue, confirm that high-priority traffic is still covered.

The source's own example sets web servers at 85 percent CPU as the upper limit, reserving 'enough headroom to handle occasional spikes'. That leaves a 15 percent margin of safety. The ceiling for forecasting is 85 percent, not the 100 percent at which the server fails. Say your busiest observed peak on a web server is 70 percent CPU. Against the 85 percent working ceiling you have 15 points of headroom. Against 100 percent you would wrongly count 30. The source asks why 15 percent and not 10 or 20, and answers that the number is a slippery educated guess. You justify it by looking at the spikes you have seen. The same logic applies to the database example. Find the QPS at which end-user experience degrades, subtract a comfortable percentage for periodic spikes, and the resulting single number characterizes that configuration for that role. That number sets the alert threshold and tells you when to start sizing new database capacity.

• If a resource tolerates spikes well, as caching systems do, then use a smaller safety margin. If it fails hard or fails over, then keep a larger margin.
• If observed spikes grow beyond what the margin was based on, then lower the working ceiling and recompute the forecast.
• If a release, a lost server, or a dependency change affects a bottleneck, then re-measure its headroom before trusting the old plan, because bottlenecks can shift or be masked.
• If consuming headroom would drain a queue but risk high-priority traffic, then keep the headroom and use priority queues or graceful degradation. Watch the other side too: protecting headroom at the expense of queuing can violate SLAs.
• If finance pushes back on the cost of extra headroom, then justify it as risk avoidance: show the user-experience and bottom-line impact of a spike without ready capacity.

• Using 100 percent as the ceiling. The tell is a headroom figure that looks comfortable while the system fails under a routine spike, since the source says resources fail at critical levels such as 100 percent CPU or disk.
• Skipping the point-of-failure test and borrowing a number. The tell is that you cannot state the QPS at which end-user experience degrades for your own hardware configuration.
• Picking a margin by gut with no spike evidence. The tell is that you cannot answer why 15 percent and not 10 or 20 from your own graphs.
• Measuring headroom on the wrong component. The tell is plenty of headroom on most services while a bottleneck with small headroom limits the high-level goal, which is always subject to the bottleneck only.
• Treating the headroom figure as permanent. The tell is a stale plan after microservice dependencies, releases, or traffic patterns changed.

• ▸Role / component: name the server role or service being assessed (e.g., database server, web server).
• ▸Resource class: CPU, disk I/O, memory, network bandwidth, or host count.
• ▸Hardware configuration: record the specific configuration the limit applies to.
• ▸Point of failure / red-line metric: the load (e.g., QPS) at which it becomes unresponsive, fails over, or degrades user experience.
• ▸Observed typical peak utilization: the highest regular usage from your graphs, with the date range.
• ▸Largest observed spike: the spike on which the safety margin is based.
• ▸Safety factor (%): the margin you subtract, with a one-line reason (e.g., cache tolerates spikes).
• ▸Working ceiling: red-line value minus the safety margin; use this for forecasts and alerts.
• ▸Headroom remaining: working ceiling minus typical peak; note the trigger to start sizing new capacity.
• ▸Last re-check and change triggers: the date, and any release, server loss, dependency change, or event that prompted it.

1. 1Define the clock explicitly. Start it when the capacity need is identified and stop it when the new capacity is deployed, configured, tested and serving production traffic. Delivery of hardware or an instance does not stop the clock. Write both endpoints down so every request is timed the same way.
2. 2Break the pipeline into its stages and timestamp each handoff: justification to the budget holder (often the CFO), approval, quote or order (a call to a hosting provider, a request for quotes from a vendor, or a purchase at a local store), supply-chain delivery, racking and provisioning, configuration, testing, and the cut-in to production behind the load balancer. A single end-to-end number hides which stage is the bottleneck.
3. 3Treat procurement as part of capacity planning, not an afterthought. Add the measured lead time to every forecast: the capacity must be ordered while current capacity is still sufficient, so the order date equals the date you forecast the shortfall minus the total pipeline time.
4. 4Keep separate lead-time figures per path: owned datacenter (long supply-chain cycles plus provisioning and configuration overhead), hosting provider, and cloud (capacity added at the push of a button, e.g. through a CD platform such as Spinnaker). Do not apply one number to all three.
5. 5On the cloud path, measure the human and control-point time that remains: due diligence on sizing, approval, and cost review. The button is fast, but over-provisioning or heavyweight instance types follow from skipping the sizing work. Add team-level cost reporting so capacity growth gets feedback, as Netflix does.
6. 6Shorten the testing stage with production load testing. Use the load balancer to add and remove a single machine to and from production, and use that to establish a server's upper limit. Knowing that limit before you order tells you how many units to buy and removes guesswork from the test stage.
7. 7Pre-stage the approval path. Agree in advance who must sign off (the CFO, in the passage's framing) and what evidence they need: measurements, usage estimates and predictions. A clearly defined procurement and approval system is one of the four basic concepts the book lists.
8. 8Review the pipeline after every purchase. Compare actual stage durations with the figures you planned around, update the lead-time table, and re-check the forecast trigger date.

The book describes an operations engineer told they have one week to prepare for an unknown amount of extra traffic, for instance from a new feature launch or a link from a major site. Twitter's 2014 FIFA World Cup goal of zero downtime is a comparable case, since its match-driven tweet spikes once took the service down. Run the pipeline test against a one-week window. On the cloud path, the provisioning clock is short, but you must still size correctly and get approval. The pricing table shows why: an m4.4xlarge costs $0.80/hour on-demand versus $0.496/hour on a 1-year reserved term, so the reserved rate is about 62% of on-demand. A reserved commitment is itself a procurement decision that needs sign-off. On the owned-datacenter path, long supply-chain cycles and provisioning overhead mean a one-week window is almost certainly too short, so the choice is already made by the lead time. Before the event, add and remove a single server through the load balancer to find the per-server limit. Then divide the forecast spike by that limit to get the unit count. The book also asks whether to deploy capacity for a one-time event that will sit underutilized most of the year. Answer that question while timing the stages, not after.

• If your total pipeline time exceeds the forecast horizon you can trust, then you cannot buy to the forecast, so shorten the pipeline (cloud path, pre-approval) or hold headroom.
• If the capacity is for a one-time event that would sit underutilized most of the year, then prefer the fast-provisioning path (cloud, on-demand) over owned hardware, and cost the idle period explicitly.
• If approval is the longest stage, then fix it with pre-agreed evidence and thresholds. Smaller, nimbler organizations get hardware faster because their approval process is less cumbersome, even at a worse price.
• If cloud provisioning is nearly instant, then do not skip due diligence. Require sizing evidence and team-level cost reporting so over-provisioning and cluster underutilization do not replace the old delay.
• If the testing stage dominates, then establish a server's upper limit through production load testing with a single machine behind the load balancer, rather than relying on synthetic benchmarks.

• Stopping the clock at delivery. The tell: capacity is 'provisioned' on paper, yet racked-but-unconfigured or untested machines are not yet serving traffic.
• Using one lead time for all paths. The tell: forecasts assume cloud-style speed for datacenter orders, and you run short despite ordering 'on time'.
• Forgetting approval time. The tell: the technical work finishes in days, but the request sits waiting for the budget holder.
• Treating the cloud's speed as a reason to skip sizing. The tell: heavyweight instance types, over-provisioned clusters and a fast burn rate, which can end in the cloud jail the book describes.
• Never re-measuring. The tell: planning still uses last year's stage durations while the process, vendor or approval chain has changed.

• ▸Capacity need identified (date and trigger): record the date and what forecast or event triggered it.
• ▸Path: owned datacenter, hosting provider, or cloud.
• ▸Justification and approval (days): time from request to sign-off, and who signed.
• ▸Quote or order placed (days): time spent on the vendor RFQ, provider call or cloud purchase decision.
• ▸Delivery or supply-chain (days): time from order to resources available.
• ▸Provisioning and configuration (days): time to deploy and configure.
• ▸Testing and load verification (days): time to confirm the per-server upper limit and readiness.
• ▸Serving production traffic (date): the date the stop-clock condition is met.
• ▸Total pipeline time and slowest stage: sum the stages and name the bottleneck.
• ▸Forecast trigger date: the forecast shortfall date minus total pipeline time.

1. 1Set the goal before the efficiency target. Write down the performance, availability, and reliability targets first (the source's example is serving pages in under three seconds). Treat efficiency as a constraint on meeting them, not the goal. The source states that the goal itself should not be to reduce the footprint or raise utilization, and that opex work must not hurt the end-user goal.
2. 2Pick one defining capacity metric per tier and find its ceiling. For the web servers, the source uses CPU as the single defining metric because it correlates with the work the server does and, hopefully, with site traffic. Measure the ceiling against application-level work, not just system stats.
3. 3Set an upper limit with deliberate headroom. The source sets total CPU at 85 percent, enough to absorb occasional spikes while still using the servers efficiently. Write the limit down per tier so idle capacity can be measured against it.
4. 4Tie system resources to application work. Relate CPU to busy Apache processes (about 1:1 in the source). For databases, chain hardware resources (CPU, disk I/O, memory) to database resources (queries per second, replication lag) to a user-level measure (photos per user per database). This lets you compute needed capacity from load.
5. 5Find the real bottleneck before adding or trimming anything. Look for the root cause, not the effect. In the source's database case, disk utilization hit 100 percent several times, but replication lag jumped only when I/O wait exceeded 40 percent. Size against the resource that actually constrains the service.
6. 6Audit the cluster for underutilization against the ceiling. Compare observed peak usage per host to the stated upper limit. Flag large instances (compute, memory, or network) used unnecessarily and over-provisioned clusters. The source names two causes: not knowing the server's upper limit, and easy cloud horizontal scaling reducing the push for optimization.
7. 7Choose the instance type by price per unit of the bottleneck resource. Prices vary widely across types. The January 2017 source tables list AWS m4.large at $0.10 on-demand and x1.32xlarge at $13.338, and GCE n1-standard-1 at $0.05 up to n1-standard-64 at $3.04. Match the family (standard, highmem, highcpu) to the metric that defines capacity.
8. 8Decide between tuning and buying by weighing engineering time against hardware cost. Scale immediately and skip tuning when an event threatens user experience, such as a sale spike. Tune when opex pressure is high, as at a startup watching its burn rate.
9. 9Cap the cost of your own monitoring data. Set retention so fine detail is kept only for the recent past, as RRDTool does with a user-defined maximum age. Netflix's aggressive retention policies for operations data saved millions of dollars in AWS S3 cost. Re-check the plan against actual usage weekly or even daily.

The source's web tier shows the method. The team picks CPU as the single defining capacity metric because it tracks the work Apache does. They cap total CPU at 85 percent, which leaves headroom for occasional spikes while still using servers efficiently. They find CPU usage to busy Apache processes runs about 1:1, so either number can be computed from the other. That ceiling then becomes the yardstick for the cluster. Any host sitting well below 85 percent at peak is a candidate for consolidation or a smaller instance type. The source's pricing tables show why this matters: moving from AWS m4.large ($0.10 on-demand) to m4.16xlarge ($3.20) multiplies the hourly price 32 times, and GCE n1-standard-1 ($0.05) to n1-standard-64 ($3.04) is roughly 61 times. A database case adds a caution. Disk utilization hit 100 percent several times, but replication lag spiked only when I/O wait passed 40 percent. The tier's real limit is I/O wait, not the headline utilization number.

• If a service's performance, availability, or reliability target is at risk (for example, a sale-driven transaction spike), then add capacity now and defer tuning. Efficiency work must not damage the end-user goal.
• If peak usage on a tier sits well below its defined ceiling (such as the 85 percent CPU limit) and the instances are large, then downsize or consolidate. Unneeded large instances are the common source of cluster underutilization.
• If opex or burn rate is the binding constraint, as at a startup, then invest engineering time in tuning. If the cost of engineering time exceeds the cost of hardware or instances, then buy capacity.
• If the metric you track hits its maximum but the service does not degrade (disk utilization at 100 percent with no lag), then keep looking for the true constraint, such as I/O wait above 40 percent, before sizing against it.
• If operations data is growing and old data no longer informs decisions, then shorten retention. Stale data can mislead decisions and costs real storage money.

• Treating higher utilization as the goal. The tell: cuts to headroom or footprint that lead to missed performance, availability, or reliability targets. The source says the goal is user experience and SLAs, with efficiency subordinate.
• Never establishing a server's upper limit. The tell: no documented per-tier ceiling (such as 85 percent CPU), so no one can say whether the cluster is underused or near saturation.
• Reaching for horizontal scaling by reflex on the cloud. The tell: large, expensive instances running unnecessarily and over-provisioned clusters, with performance optimization never on the agenda.
• Diagnosing from the wrong signal. The tell: effort aimed at disk utilization reaching 100 percent when the replication lag actually tracks I/O wait crossing 40 percent. This is chasing effects rather than the root cause.
• Keeping all monitoring data forever. The tell: storage cost for operational data grows without a retention policy, and old data feeds forecasts it should not.

• ▸Service/tier: name the tier being audited (for example, web servers, MySQL database).
• ▸Target SLA/SLO: the performance, availability, and reliability targets this tier must meet (for example, pages served in under 3 seconds).
• ▸Defining capacity metric: the single metric that tracks work done (for example, CPU), with the application-level measure it correlates to (for example, busy Apache processes).
• ▸Ceiling and headroom: the maximum allowed value (for example, 85 percent CPU) and the reason for the margin.
• ▸Observed peak vs. ceiling: the measured peak per host and the gap to the ceiling.
• ▸True bottleneck evidence: the resource and threshold where degradation actually begins (for example, I/O wait above 40 percent).
• ▸Instance type and unit price: current type and hourly price (on-demand vs. reserved), plus the candidate alternative family.
• ▸Tune-or-buy decision: engineering time cost vs. hardware cost, and whether a user-facing event or opex pressure applies.
• ▸Monitoring data retention: how long fine detail is kept and the point at which data is rolled off.
• ▸Next review date: weekly or daily check of actuals against the forecast.

1. 1Open with the two questions the source treats as the starting point: what performance and what availability does the organization need from the site? Write the answer as numbers, using the metrics the source names: response times, consumable capacity, and peak-driven processing.
2. 2Treat the performance target as a moving one. The source says expected performance keeps rising as end-user attention span shrinks, so assess what users expect in the short term going forward. Re-check the target on a schedule and do not freeze it.
3. 3Factor the client device into the target. The source says meeting expected performance depends on the kind of device (smartphones, smart watches, tablets) and on the growth of digital media and information volume. Record which device classes the target applies to.
4. 4Map the backend before measuring. List every component behind the application: cache, persistent data storage, queue, search service, authentication service, and each service or microservice. The source says high availability and performance call for capacity planning of each one.
5. 5Answer 'How well is the current infrastructure working?' by measuring the workload characteristics of each piece (web server, database server, network), whether in the datacenter or the cloud. Compare each measurement to the performance requirements from step 1.
6. 6Define the peak-driven processing load: the workload your resources see at peak usage. Estimate it by analyzing how large past spikes were relative to normal traffic, which gives a first-cut estimate of the next spike.
7. 7Base conclusions on empirical observation of your own site's usage, not on benchmarks from artificial environments. Do not choose hardware from SPEC CPU2006 results alone, because their characteristics differ from web applications.
8. 8Cap tuning effort. When demand is growing, decide whether to make existing gear faster or add capacity, and put the main effort into finding what you need and when. Treat any single tuning win as a rare gain, not a plan.

Before the 2014 FIFA World Cup, Twitter had a recurring performance and availability problem. In the previous World Cup, every time Brazil or Japan scored, the spike in tweet volume took the service down, and the Fail Whale became popular as a result. The 2014 goals were zero downtime and a blazingly fast mobile app for sharing photos. The source's first-cut method is to analyze the magnitude of past spikes relative to normal traffic and project the next one. It then raises the cost question: should you deploy capacity for one-time events when it will sit underutilized most of the year? It also raises the unplanned case, such as the power failure during Super Bowl XLVII in 2013. The answer is a systematic approach to architectural design and capacity planning, not a one-off fix. Write the target as a number, measure each backend component against it, and size for the estimated spike.

• If traffic is expected to double over the next couple of months, then plan to add capacity rather than tune the existing gear, as with the butcher who buys more trucks instead of speeding up the three he has.
• If a benchmark's workload differs from your web application's, then do not base hardware selection on it. Use observed production usage.
• If tuning has produced little gain after repeated tweak-and-test cycles, then stop and return to determining what the system needs and when.
• If the event is a one-time spike, then compare the cost of capacity that stays underutilized for most of the year against the cost of the spike, before you deploy for it.
• If a component sits behind the user-facing response, such as the cache, queue, search, or authentication service, then measure and plan it separately instead of assuming the front end represents it.

• Setting a fixed performance target and never revisiting it. The tell: users' expectations have risen, but the requirement document is unchanged.
• Choosing hardware from SPEC-style benchmark scores. The tell: the benchmark workload does not resemble your web traffic, yet the purchase decision cites it.
• Building a detailed queuing model that is stale on arrival. The tell: new code ships and usage characteristics change before the model is finished.
• Endless parameter tuning. The tell: weeks spent on tweak-and-test with negligible gain, while the capacity question stays unanswered.
• Measuring only the web tier. The tell: a cache, queue, search, or authentication service saturates, yet your monitoring covered only the front end.

• ▸Application / service name: name the website, API, or background job being assessed.
• ▸Target response time: state the required response time, with the client device class it applies to.
• ▸Expected-performance review date: record when the target is next reassessed, since user expectation keeps rising.
• ▸Backend components: list each one (cache, data store, queue, search, authentication, microservices).
• ▸Current measured workload per component: record the observed characteristics of each piece, in the datacenter or the cloud.
• ▸Peak-driven processing estimate: record the expected workload at peak, based on past spikes relative to normal traffic.
• ▸Gap vs. requirement: for each component, note where measurements fall short of the target.
• ▸Action: choose tune existing gear or add capacity, with the reason.
• ▸Evidence source: note whether the data is from production observation or a benchmark.

1. 1Write the availability target as a number tied to a contract. State the percentage (for example 99.99 percent), then translate it into allowed unavailability: 99.99 percent means 0.01 percent of the time the service can be down and still meet the SLA. Record what counts as 'unavailable' before anyone measures anything.
2. 2Separate availability from performance in the requirement. Define load and capacity requirements with specific metrics: response times, consumable capacity, and peak-driven processing. Peak-driven processing is the workload your web servers, databases and other resources carry at peak usage. A service that responds too slowly under peak is effectively failing, so set the 'degradation affects end-user experience' point explicitly.
3. 3Map the backend and treat each component as its own availability risk. A typical modern app has a cache, persistent storage, a queue, a search service and an authentication service. Plan capacity for each service or microservice individually, and account for upstream and downstream services in an SOA or microservice architecture.
4. 4Find each component's red line. For each piece, such as a database server, determine how many queries per second it can serve before performance degradation affects users. Adjust for periodic spikes, subtract a comfortable percentage of headroom (safety factor), and keep the resulting single number per configuration and role.
5. 5Use the red line to set alert thresholds and growth triggers. The red line tells you the load that makes the component unresponsive or forces a fail-over. It also tells you what to expect from adding or removing similar servers, and when to begin sizing the next order of capacity.
6. 6Include the failure point in every calculation. Treat the point of failure as mandatory input, not optional. Run a segmented architecture so you can isolate the limits of each part of the backend.
7. 7Plan for spikes using past events, then question the cost. Compare the magnitude of past spikes to normal traffic for a first-cut estimate of the next one. Decide explicitly whether to deploy capacity for one-time events that sit idle most of the year, and decide how you will handle unplanned events such as a power failure.
8. 8Cover Disaster Recovery and failover as availability inputs, and re-run the process continuously. The source says expected availability shapes capacity planning in part through DR and failover. It also says that in an Agile environment planning is continuous, flexible and adaptive, and that static theoretical models fail.

The source's real example is Twitter and the FIFA World Cup. Before the 2014 World Cup, Twitter's recurring story was that the service became unavailable during the previous tournament: every time Brazil or Japan scored, the tweet-volume spike took the service down, and the Fail Whale became the symbol of those availability problems. The 2014 goal was therefore stated as absolute zero downtime, alongside a fast mobile app for photo sharing. The method the source suggests is to analyze the magnitude of past spikes relative to normal traffic and derive a first-cut estimate of the next spike. It then raises the cost question: should you deploy capacity for such one-time events when it will mostly sit underutilized? It also points to unplanned events, such as the power failure during Super Bowl XLVII in 2013, which no traffic forecast covers. The conclusion is that high availability in these moments requires a systematic approach to both architectural design and capacity planning, not a larger server count alone.

• If the SLA promises a percentage such as 99.99 percent, then convert it into the allowed 0.01 percent unavailability and treat any component whose failure alone could consume that allowance as a priority for redundancy or failover.
• If a component's load approaches its red line (measured capacity minus spike adjustment and headroom), then begin sizing new capacity and alert before the point where it becomes unresponsive or fails over.
• If an event is predictable (Super Bowl, a show launch), then estimate the spike from past spikes relative to normal traffic, and weigh the idle-capacity cost against the downtime risk before provisioning.
• If you have only a week to prepare for unknown extra traffic, then do not build a queuing model; measure each component's limit, apply headroom, and fix the weakest point first, since models go stale as code and usage change.
• If the architecture is SOA or microservices, then compute capacity per service with its upstream and downstream dependencies included, because one service's ceiling caps the availability of what depends on it.

• Quoting the 'nines' without a definition of downtime or a mapping to components; the tell is that no one can say how many minutes of failure the SLA allows or which component would use them.
• Planning capacity from a static queuing model; the tell is that the model was built weeks ago and code deploys and usage patterns have already changed (Google commits 16,000 human changes and 24,000 automated changes on a typical workday).
• Sizing for average traffic and ignoring peak-driven processing; the tell is outages that coincide with launches, goals scored, or links from sites like Reddit or the New York Times.
• Omitting the failure point from the calculation or using a red line with no headroom; the tell is that the database or other component hits unresponsiveness or fail-over before any alert fires.
• Ignoring unplanned events and dependencies, such as a power failure or a downstream service; the tell is a capacity plan that covers forecast traffic but has no DR or failover provision.

• ▸Availability target (%): the SLA figure, e.g. 99.99 percent.
• ▸Allowed unavailability (%): 100 minus the target, i.e. what the SLA tolerates.
• ▸Definition of 'unavailable': the response-time or error condition that counts as downtime.
• ▸Component and role: for example, database server serving frontend web servers.
• ▸Hardware configuration: the specific configuration this measurement applies to.
• ▸Measured red-line load (e.g. QPS): the load at which performance degrades enough to hurt users.
• ▸Spike adjustment and headroom (safety factor): the periodic-spike allowance and percentage margin subtracted, giving the planning number.
• ▸Failure/fail-over behavior: what happens at the red line and which alert threshold you set.
• ▸Dependencies and DR/failover: upstream and downstream services and the recovery path.
• ▸Next-capacity trigger: the load level at which you begin sizing additional capacity.

1. 1Define the performance and availability targets first, using specific metrics: response times, consumable capacity, and peak-driven processing. Cost is only meaningful against what the infrastructure must deliver, so write these targets down before pricing anything.
2. 2Split the architecture into components that each do one measurable job (microservices, containers, or at minimum separable tiers). If the web server and database share one machine, as in the single-server example in Figure 2-3, you cannot tell which component drives the spend. Use the built-in container monitoring to capture task-level metrics.
3. 3Measure how the current infrastructure performs for each piece (web server, database server, network) and compare it to the targets. Identify what is over-provisioned and what is near its limit.
4. 4Forecast needs from traffic growth and past system performance, remembering that performance does not scale linearly with traffic. Then reconcile the forecast with what you can afford and a realistic timeline: what you need, and when.
5. 5Pipeline procurement rather than buying everything at once. Milestones on product and engineering roadmaps are not aligned, so stage the purchase of different elements (drives, memory, network switches) to match each team's need. Work with product and engineering to see the roadmap.
6. 6Do not size always-on capacity for the burst peak. Handle bursts with queuing and caching, which reduce the capex overhead, and keep the baseline sized for normal load.
7. 7In the cloud, track expected opex continuously, even though instances spin up in minutes. Check instance types (heavyweight types and over-provisioning are the two common causes of a fast burn rate) and autoscaling policies, which are not trivial to set up.
8. 8Reassess hosting at scale: public cloud becomes very pricey at scale, and Dropbox, which started on a public cloud, migrated to its own datacenters. Evaluate serverless (BaaS/FaaS) for cutting operational cost and complexity, weighing vendor dependency and immature supporting services.
9. 9Price availability explicitly. Disaster recovery in two or more locations multiplies infrastructure, nodes to manage, data replication, deployment, and monitoring. Buy that only to the level the SLA and business continuity plan require, then iterate and recalibrate the plan over time.

The Ellen DeGeneres Oscar tweet in 2014 produced a huge spike in views and retweets in a short time, and the outage that followed traced to a missing architecture component in one data store and in the search tier. The cost lesson is that always-on capacity for such a burst is not financially sound, yet an under-designed architecture fails anyway. The fix the passage points to is not buying more hardware but load testing the specific scenario (more than a million people searching for the same tweet and then retweeting it), which would have exposed the gap. Burst handling then comes from queuing and caching, which alleviate the capex overhead, rather than from permanent peak-sized servers. Capacity for the baseline is sized from measured per-component workloads and forecast growth. The spike becomes a forcing function to revisit the design, so the money goes to the architecture gap and not to idle machines.

• If a traffic burst is short-lived and predictable (holiday season, Super Bowl, album launch), then use queuing and caching to absorb it rather than running peak-sized capacity permanently.
• If you are in the cloud and the burn rate is climbing, then check instance types and over-provisioning first, then autoscaling policies, before adding capacity.
• If you operate your own datacenter, then plan further ahead and pipeline procurement, because supply-chain cycles are long and errors in capacity planning are severe; if you are in the cloud, elasticity cuts the reliability risk of under-provisioning, so the question becomes cost.
• If cloud spend becomes very pricey at scale, then evaluate moving to your own infrastructure, as Dropbox did, instead of assuming the cloud stays cheapest.
• If an SLA or business continuity plan calls for a second location, then cost the multiplication of infrastructure, nodes, replication, deployment, and monitoring before committing, and do not justify it with a naive downtime-times-revenue-per-minute figure.

• Over-provisioning or choosing heavyweight instance types out of poor due diligence. The tell is a fast burn rate with low utilization on the components.
• Sizing for the burst peak and running it all the time. The tell is capacity that sits idle most of the year while a queue or cache would have handled the spike.
• Reactive buying: acting only after a shortage appears. The tell is missed deadlines or lost user acquisition or retention because procurement lead times were longer than the warning.
• Treating downtime as lost revenue by straight math (10 minutes at $3,000 a minute equals $30,000). The tell is a cost justification that ignores customers who resume purchases after the outage and the extra customer service spend; it is neither true nor accurate, so use availability targets instead.
• Planning with too little detail, or too much. The tell is either a fatal capex or user-experience miss, or a capacity process so detailed that it delays the plan and risks performance.

• ▸Component / service: name each separable component (web server, database, queue, search, authentication) so its cost can be measured on its own.
• ▸Performance and availability target: state response time, peak-driven load, and SLA percentage the component must meet.
• ▸Current measured workload: record the measured utilization against the target for this component.
• ▸Hosting model and cost type: note cloud, own datacenter, or serverless, and whether the spend is capex or opex.
• ▸Instance or hardware type and fit: record the type chosen and whether it is heavyweight or over-provisioned for the measured load.
• ▸Burst handling method: list the queuing or caching used for spikes, and the load test that validated it.
• ▸Forecast and procurement date: give expected growth (non-linear performance) and when each element must be bought, pipelined with roadmaps.
• ▸Redundancy and DR cost: record the extra locations, nodes, replication, and monitoring the availability target requires.
• ▸Decision and review date: record keep, resize, migrate, or buy, and when to recalibrate.

1. 1Inventory the components by purpose. List each web server tier, database cluster, caching server, storage layer, and any microservice or third-party service (BaaS/FaaS). A component qualifies as divisible only if it has one job and you can name its own metrics. If you can't, it is still part of a monolith.
2. 2Set operational requirements per component before measuring. Write down the target (for example, pages served in under three seconds) and the acceptable speed or reliability of each part of the site. Without a target you cannot work out how many servers a part needs, or how many more it needs as traffic grows.
3. 3Record an SLA/SLO for every external or serverless service you depend on, so you can meet your own targets. Convert each percentage into minutes. 99.9% over 30 days (43,200 minutes) allows 43.2 minutes of downtime a month.
4. 4Instrument each component with its own metrics. For a database, collect concurrent connections, INSERT/UPDATE/DELETE/SELECT rates, and replication lag. For its host, collect disk utilization and %iowait (iostat, every 60 seconds). Graph them side by side (Ganglia, RRDTool) so you can see which resource moves which database metric.
5. 5Find each component's ceiling by chaining three layers: hardware resources (CPU, disk I/O, memory), then component resources (queries per second, replication lag), then a user-facing unit (photos per user per database). The ceiling is the point where the component's own metric degrades, not where a hardware gauge peaks.
6. 6Match the component to its workload. Decide whether it is read-heavy (New York Times) or write-heavy (Facebook, Instagram), then choose storage (HDD or SSD) against the SLA for that component alone.
7. 7Plan replication and redundancy as capacity decisions. Replicas add redundancy and performance, but they bring lag. If all user queries go to slaves, lag becomes a user-visible defect, so include the lag ceiling in the plan.
8. 8Set retention for the monitoring data per component. Keep fine detail for the recent past and let older data coarsen or expire (RRDTool's model). Review forecasts against actuals weekly or even daily, and refit as new code and features change usage.
9. 9Tell product management which components a planned feature will load. New features change the capacity plan, so tie the forecast to the product timeline.

The source's MySQL example isolates one user database during a peak hour. The metrics are connections and INSERT/UPDATE/DELETE/SELECT rates, with a replication-lag graph beneath them. Most metrics show a few spikes, but only one stands out: replication lag peaks at more than 80 seconds. With all user queries going to slaves, a user could annotate a photo, click Submit, and not see the comment, which is an out-of-sync defect. Because the database is a separate, measurable component, the next question is which resource caused the lag. Disk utilization (reported every 60 seconds from iostat's %ioutil) hit 100 percent more than once. Lag jumped only when %iowait exceeded 40 percent. So the real ceiling is I/O wait, not raw disk utilization. The planner ties that ceiling to a user-level unit, photos per user per database, and forecasts when databases will cross it. None of this analysis is possible if the database shares a box and a metric stream with the web tier.

• If you can't state a speed or reliability target for a component, then set one before sizing it. You can't compute server counts or growth increments without it.
• If a metric spikes, then check the related resource before acting. Disk at 100% alone was not the problem. Lag moved only when I/O wait passed 40%, so alert on the metric that actually moves the component's output.
• If the component is read-heavy versus write-heavy, then choose storage (HDD or SSD) per component against its SLA, not globally.
• If you depend on a serverless or third-party service, then record its SLA/SLO and convert the percentage to minutes of allowed downtime. That is the budget your own targets must live within.
• If code and usage change faster than you can build a queuing model, then use measured per-component ceilings and frequent refits instead. The source notes that Google commits about 16,000 human changes a day, so models go stale quickly.

• Splitting by technology instead of purpose: a 'database tier' that mixes workloads. The tell is that you can't attach one ceiling or one user-level unit (photos per user per database) to it.
• Reading hardware gauges as ceilings. The tell is disk at 100% with no user impact, while the real degradation (replication lag) tracks I/O wait above 40%.
• Ignoring replication lag in capacity plans. The tell is user complaints of missing comments or stale data after a write, while the dashboards show healthy throughput.
• Taking SLA percentages at face value. 99.9% sounds reassuring, but it permits 43.2 minutes of downtime every month. The tell is a plan with no downtime budget per component.
• Letting monitoring data grow unplanned, or keeping stale data. The tell is rising storage cost (Netflix saved millions in S3 costs with aggressive retention) and forecasts distorted by old data.

• ▸Component and single purpose: name the component (e.g., user database cluster) and its one job.
• ▸Workload type: mark read-heavy or write-heavy, and the storage type chosen (HDD/SSD).
• ▸Target / SLA / SLO: state the speed or reliability goal, and convert any uptime % to minutes per month.
• ▸Hardware metrics tracked: list CPU, disk utilization, %iowait, memory and the collection interval.
• ▸Component metrics tracked: list queries per second, connections, replication lag, etc.
• ▸Observed ceiling: state the metric and value at which service degrades, and which resource drove it.
• ▸User-level unit: state the ceiling in terms such as photos per user per database.
• ▸Replication/redundancy plan: note the replica count and the acceptable lag.
• ▸Retention and review cadence: note how long fine detail is kept, and when forecasts are refit (weekly or daily).

1. 1Define the process as three linked stages (measure, forecast, procure/deploy) and name an owner for each. The source says capacity planning and management are processes, not tasks, so schedule them as recurring work, not a one-off project.
2. 2Collect system metrics (CPU, disk, memory) and application-level metrics together. Add real-world user-experience metrics such as latency, because latency can reach an intolerable level at high volume before any system metric raises a flag. Do not look for a one-size-fits-all metric.
3. 3Isolate resource usage on multiple-use systems by pairing each application metric with the resource it drives. Plot both over the same window, for example photo processing rate against total CPU over two days. Read the slope as a ratio, then use it to set ceilings.
4. 4Before you forecast, check whether slowness comes from capacity or from the page itself. A page that is too heavy, or a third-party service, can make a site slow or unavailable even with ample capacity. Capacity is one of the more expensive things to change, so rule out these causes first.
5. 5Build a first-cut forecast from history. Compare past spikes with normal traffic to estimate the next spike's magnitude. Label small-sample estimates as rough, and re-check them against a wider range of traffic before committing money.
6. 6Decide explicitly whether to provision for one-time events. The source warns the capacity would likely sit underutilized most of the year. Weigh that cost against the availability goal, and name the alternatives, such as serving lower-resolution images or stale cache, to absorb the peak.
7. 7Run a defined procurement and approval path. Write the justification for each capital expenditure in business terms, and keep a deployment and management system ready so new resources are managed after they go live.
8. 8Re-run the loop continuously. In an Agile environment, code and usage change faster than a static queuing model can be built. Adjust forecasts when new code deploys, a new feature launches, or an external event (a link from a major site, a new API consumer) changes usage.

The source's Flickr example shows the measure-to-forecast loop. Two days of data (Saturday and Sunday) put total CPU next to the photo processing rate. Sunday's peaks ran more than 20 percent higher, but in the evening its rate fell below Saturday's at the same time of day. Distilling the two series showed that every 30 photos processed per minute corresponded to an additional 25 percent CPU utilization. The authors call this extremely rough and based on a small sample, and say confirming the 25:30 ratio rigorously would mean looking at a larger range of traffic and upload rates. Even so, it gave a starting point for determining ceilings. With web traffic and upload rate both tracked, you can work out how many machines are needed if both processes run on each machine. The approach worked for more than a year in planning Flickr's web server capacity. Eventually image processing got its own dedicated cluster that could use multicore CPUs.

• If users report slowness but CPU, disk, and memory look healthy, then check page weight, images, video, and third-party services before buying capacity.
• If a ratio between an application metric and a resource comes from a small sample, then use it only as a provisional ceiling and re-validate it over a wider range of traffic.
• If a forecast peak comes from a one-time event and the capacity would sit idle most of the year, then weigh the idle cost against the availability goal and consider degrading gracefully (lower-resolution images, HD instead of 4K video, stale cache, CDN) instead of provisioning for the peak.
• If two workloads on one system contend for the same resource and one grows large, then split it onto a dedicated cluster sized for its needs, as Flickr did with image processing.
• If new code or a new feature changes usage characteristics, then treat the existing model as stale and re-measure instead of relying on the earlier model.

• Building elaborate queuing models while the system changes underneath. The tell: the model is finished after new code has shipped and usage has already shifted.
• Tracking only system metrics such as CPU and disk. The tell: users complain about latency while the dashboards stay green.
• Buying capacity to fix slowness that comes from a heavy page or a third-party dependency. The tell: the spend does not improve user-perceived speed.
• Treating a rough small-sample ratio as settled fact. The tell: ceilings are set from two days of data with no later validation against a larger range of traffic.
• Having no procurement and approval path or business justification. The tell: you know you need capacity during a spike but cannot get it approved or deployed in time (for example, a week's notice of unknown extra traffic).

• ▸Service/component and owner: name the system and the person accountable for its capacity process.
• ▸System metrics tracked: list CPU, disk, memory, and others, with the collection source.
• ▸Application metrics tracked: list the workload drivers (e.g., photo uploads per minute, API calls) and the user-experience metric (e.g., latency).
• ▸Metric-to-resource ratio: record the observed relationship (e.g., 30 photos/min per +25% CPU), sample window, and confidence level.
• ▸Non-capacity checks done: note whether page weight, media, or third-party services were ruled out as the cause of slowness.
• ▸Forecast and basis: state the expected load, the historical spike-to-normal comparison used, and the date to re-validate.
• ▸Event/spike decision: record whether you provision for the peak or degrade gracefully, with the idle-cost reasoning.
• ▸Procurement and approval status: record the business justification, approver, and request date.
• ▸Deployment and management plan: say how new resources will be deployed and managed after arrival.

1. 1Write down the full provisioning timeline before you automate anything: order equipment, physical installation, OS/application/configuration installation, testing, and deploying into production. Automation shortens only the install-and-configure segment. The passages state that automated still takes time and can still produce problems, so time that segment too.
2. 2Choose a tool from the automated installation and configuration category (the passages name Vagrant, Chef, Puppet, Pallet, Ansible, CFEngine). Express configuration as code, in a high-level or descriptive language, so it provisions infrastructure and deploys, not just repeats steps. The passages draw this line: infrastructure as code differs from infrastructure automation, which only replicates steps across servers.
3. 3Make one central location the only place changes happen, a 'control tower' for all hosts. Push changes from there to the servers they apply to. Don't hand-edit hosts.
4. 4Put every OS, application, and other configuration under version control (Git, Perforce, RCS, CVS, Subversion). Make each change answer who, what, when, and why. Back up and replicate the installation and configuration files from the same central place.
5. 5Aggregate configuration and management logs centrally and treat them as your first troubleshooting resource when a newly provisioned machine misbehaves.
6. 6Standardize on homogeneous hardware. Where you must mix hardware, keep the inventory in the central management environment and define different configuration settings per hardware type.
7. 7For cloud capacity, configure autoscaling in place of physical installation (Chapter 6 covers it). For datacenters, add the physical steps: racking, cabling, and confirming datacenter staff know the machines are arriving.
8. 8Put a new machine into production gradually. Raise its weight in the load-balanced pool, measure its capacity with the same process used for existing servers (Chapter 3), and watch how the added capacity shifts traffic if it relieves a bottleneck.

The source's own arithmetic: your capacity forecast says you run out in six weeks, and adding new hardware takes three weeks, so you have only three weeks of breathing room. Of the stages (ordering, shipping, physical installation, OS/application/configuration installation, testing, production), the one you control most is integrating machines onto the network and beginning operations. Automated deployment and configuration compresses that stage. Suppose you pull a machine out of the crate and it needs OS, application, settings, and data. With a Chef/Puppet-style tool driven from one version-controlled repository, those steps run from code rather than by hand. You then ramp the machine by raising its weight in the load-balanced pool while measuring its capacity as in Chapter 3. The gain is not a fixed number of days; it is a shorter, more predictable install segment inside the three weeks, which widens your margin before the six-week exhaustion date.

• If your forecast shows capacity exhausted sooner than total lead time (order to production), then order now and put your effort into shrinking the segment you control: install and configuration.
• If you are repeating the same install steps on many servers, then move from simple infrastructure automation to infrastructure as code so provisioning and deployment are described, versioned, and reproducible.
• If a change is needed on a host, then make it in the central configuration location and push it. Never change the host directly.
• If hardware differs across machines, then record it in the central inventory and give each type its own configuration settings. Prefer homogeneous hardware to reduce that burden.
• If a new machine is ready for production, then ramp its load-balancer weight gradually and measure it rather than switching on full traffic.

• Treating automation as zero-time: the schedule omits install time and an unexpected problem eats the buffer. Tell: provisioning dates slip even though 'it's automated.'
• Scripting repeated steps without managing configuration as code (infrastructure automation only). Tell: servers that should be identical drift, and no one can say how one was built.
• Changes made directly on hosts outside the central location. Tell: you cannot answer who changed what, when, and why from version control.
• Mixed hardware with no inventory-based configuration. Tell: the same configuration behaves differently across machines and you cannot say why.
• Skipping testing or full-traffic cutover of a new machine. Tell: a bottleneck shift or misconfiguration appears only after the server takes production load.

• ▸Capacity exhaustion date: when your forecast says current capacity runs out.
• ▸Order-to-delivery time: how long ordering and shipping take, including whether the order is trackable.
• ▸Physical installation time: loading dock to racked and cabled, including datacenter staff involvement (N/A for cloud).
• ▸Automated install/config time: measured duration of OS, application, and configuration install using your tool.
• ▸Testing time: QA team and environment available, and time to functionally test.
• ▸Total lead time vs. exhaustion date: sum the stages and subtract from the days remaining to get the breathing room.
• ▸Central change location: the single repo or system where all changes are made.
• ▸Version control and logging: which system tracks who/what/when/why, and where aggregated config logs go.
• ▸Hardware inventory and homogeneity: hardware types in use and the configuration setting for each.
• ▸Production ramp plan: starting load-balancer weight, increase steps, and metrics watched.

1. 1Find the ceiling of each tier before you set any scaling trigger. Use the easily segmented architecture (web, database, cache) to find the failure point of one unit of each tier. The passages call including the point of failure 'mandatory, not optional.' Define failure as an SLA violation, graceful degradation, or true system failure.
2. 2Measure the periodic peaks of each tier. Web server capacity is peak-driven and therefore elastic, unlike storage. Pull hourly, daily and weekly graphs, find the peak period, and ignore resources that stay well within limits at peak (in the Apache example: memory, disk I/O, network). Let the remaining binding resource drive the scaling signal.
3. 3Set the scale-out threshold below the ceiling by a safety factor. The passages stipulate 85 percent CPU as the upper limit for web servers, leaving a 15 percent margin 'to handle occasional spikes.' Treat that margin as headroom you leave unused, per resource class.
4. 4Add lead time to the trigger. If capacity runs out in six weeks and new hardware takes three weeks to add, you have three weeks of breathing room. In a datacenter, supply-chain and configuration delay forces horizontal capacity to be planned ahead. In the cloud, elasticity removes most of that delay, but only if provisioning is automated.
5. 5Make every new node identical and automated. Use infrastructure as code (Chef, Puppet, Ansible, etc.) with homogeneous hardware or instance types, so adding a node is a repeatable, minimal-time operation. Infrastructure as code goes beyond repeating steps by hand on several servers.
6. 6Put the capacity behind a load balancer and add or remove nodes there. Load balancers allow easy expansion and removal of capacity in production. They also let you route varying amounts of live traffic to one server to observe its real resource use. Choose the balancing algorithm deliberately (round robin, weighted, least used, lowest latency, persistence), because not every algorithm fits every situation.
7. 7Pick instance sizes with due diligence, not by default. Heavyweight instance types and over-provisioning are named causes of a fast burn rate. Push the button (e.g. via a CD platform such as Spinnaker), but pair it with team-level cost reporting, as Netflix does on AWS, so growth in capacity gets feedback.
8. 8Re-run the cycle continuously. Capacity planning is not a one-time process. Revisit ceilings, thresholds and instance choices when the technology changes (VMs versus containers) or when traffic patterns shift.

Take the single Apache web server in the passages. Its hourly graph shows no pattern, its daily graph a smooth decline and rise, and its weekly graph shows Monday as the highest-traffic day. Memory, disk I/O and network all stay far from their limits at the Monday peak, so the binding resource is CPU. You set the web tier's upper limit at 85 percent CPU, leaving a 15 percent safety margin for spikes. You place the servers behind a load balancer and push a controlled share of live traffic to one node to confirm where its ceiling sits. The scale-out trigger is then CPU approaching 85 percent during the Monday peak cycle, adding identical, automatically configured nodes. If this were a datacenter, a three-week hardware lead time against six weeks of remaining headroom would leave only three weeks, so you would order early. In the cloud, the same trigger can fire an automated add, bounded by cost reporting.

• If the workload is peak-driven (web servers), then scale it elastically on its binding resource; if it is steadily growing (storage), then plan it by forecast rather than by autoscaling.
• If a resource sits well within its limits at peak, then drop it from the scaling signal and key on the resource that approaches saturation.
• If lead time to add capacity exceeds the time until exhaustion minus your margin, then provision now; automate deployment to shrink that lead time first.
• If you are in a datacenter, then plan against long supply-chain cycles; if you are in the cloud, then rely on elasticity for peak reliability but add cost reporting to guard against over-provisioning.
• If a change makes pages faster (e.g. memcached), then re-check web tier thresholds, because faster pages can bring faster clicks and more load.

• Autoscaling policy set up trivially: the passages say it is not trivial, and ease of scaling hides missing detail. The tell is capacity growing without a stated ceiling or SLA behind the trigger.
• Heavyweight instances and over-provisioning: the tell is a fast burn rate and spend rising faster than traffic.
• Scaling to 100 percent of a ceiling with no safety factor: the tell is that ordinary spikes cause SLA violations because no headroom remains (the passages use 85 percent CPU, a 15 percent margin).
• Letting the cloud lock you in ('cloud jail'): the tell is heavy use of provider-specific services, with switching costly and spend very high at scale.
• Sizing from artificial benchmarks instead of observed usage: the tell is hardware or instance choices justified by SPEC scores that differ in character from your web workload.
• Ignoring load balancer risk: the tell is a single balancer or poorly chosen algorithm whose failure is spectacular and dramatic.

• ▸Tier / resource: name the tier (web, app, database, cache) and the binding resource (e.g. CPU).
• ▸Measured ceiling per node: the load at which one node violates the SLA or fails, found by live testing behind the load balancer.
• ▸Safety factor and trigger threshold: the ceiling minus margin (e.g. 85 percent CPU, 15 percent margin).
• ▸Peak pattern: when the peaks occur (hourly, daily, weekly) and the evidence from the graphs.
• ▸Scale-out / scale-in action: what is added or removed, in what increments, using which identical, automated build.
• ▸Provisioning lead time: time to add a node (datacenter weeks vs. cloud minutes) and time until exhaustion.
• ▸Load balancing algorithm: chosen algorithm and why it fits this tier.
• ▸Instance type and cost guardrail: chosen size, justification, and team-level cost report owner.
• ▸Review trigger: events that force a re-run (technology change, traffic pattern shift, new feature).

1. 1Segment the architecture into roles (for example frontend web servers, database servers, search tier, data stores). The source says an easily segmented architecture is what lets you find the limits of the current hardware configurations, and capacity planning is architecture-specific. Write the role list before you look at any hardware.
2. 2State the performance and availability each role must deliver: the end-user performance you expect in the short term, plus DR and failover needs. Hardware is 'appropriate' only against this target, so write it down per role.
3. 3Pick one hardware configuration (or one cloud instance type) per role, and record its full spec. Do not start from a general benchmark. SPECspeed and SPECrate characteristics are very different from web applications, so basing the choice on them skews the decision. Use them at most as a comparison input, never as the capacity number.
4. 4Find that configuration's failure point under a load that resembles the real workload. For a database, ask: with this exact hardware, how many QPS can it manage, and how many before performance degradation affects the end user? Failure can mean an SLA violation, graceful degradation, or true system failure, so define which one you are measuring.
5. 5Turn the measured ceiling into one number per role: adjust for periodic spikes, subtract a comfortable headroom percentage (safety factor), and call the result the 'red line' for that configuration in that role. Set alert thresholds from it.
6. 6Keep every server in a role on the identical configuration. Then the single red-line number holds for every box, and you can predict what adding or removing 'similar' servers does and when to start sizing the next order. A mixed fleet breaks that arithmetic.
7. 7Plan procurement against the product and engineering roadmaps, and pipeline the purchase of different elements (drives, memory, network switches). Datacenter supply chains are long, so you need to be predictive, not reactive. In the cloud, instances start in minutes, but you still track expected opex and the larger instance-type menu (GPU, FPGA) makes the choice harder, so repeat the red-line test for each candidate type.
8. 8Re-test when dependencies or the application change. Microservice dependencies shift, creating new bottlenecks and masking old ones, so a configuration's role-specific red line is only valid for the workload it was measured on. Put the data in business terms when you request budget: 'pages X percent faster, pageviews and ad revenue up to Y percent.'

The source's own example is a database server answering queries from frontend web servers. Step one is to isolate that role. Step two is to fix one hardware configuration for it and ask how many QPS that configuration can serve before end users feel degradation. Suppose the measured point is Q QPS. You adjust for periodic spikes and subtract a headroom percentage H, which gives a single red-line number R for 'this database configuration in this role.' That number gives you three things: the alert threshold, the expected effect of adding or removing similar database servers (roughly N servers of the same configuration, so about N×R, though system performance does not scale linearly with traffic, so verify by measurement), and the trigger date to start sizing the next order, given long lead times. You take the request to finance as: 'if we get these new database servers, pages are X percent faster and pageviews, with ad revenue, can rise up to Y percent.' The counter-example is the 2014 Oscars tweet outage. A missing architecture component in a data store and the search tier, never exercised by load testing a million people searching for and retweeting one tweet, caused it. The lesson is to test each role's configuration against the extreme scenario, not only the average.

• If a vendor or public benchmark (SPECspeed/SPECrate, Cloud IaaS 2016) is the only evidence for a purchase, then do not buy on it. Run your own role-specific load test, because the benchmark workloads are not representative of web apps.
• If a role has no measured red-line number for its current configuration, then freeze that configuration and measure before adding capacity or changing hardware. Otherwise you cannot predict what adding similar servers will do.
• If a new server for an existing role would differ in configuration, then treat it as a new configuration: re-measure its red line separately and do not assume the old number applies.
• If you are in the cloud and burn rate is climbing, then check for heavyweight instance types and over-provisioning before anything else. Both are named as common causes of a fast burn rate.
• If product or engineering roadmaps or microservice dependencies change, then re-evaluate the hardware choice and the red line. New bottlenecks may appear and old ones may be masked.

• Choosing hardware from SPEC-style benchmark scores. The tell: the numbers look impressive on paper, but production capacity falls short of the prediction because the benchmark workload isn't a web workload.
• Treating the ceiling as a raw measurement and skipping spikes and headroom. The tell: the 'red line' equals the failure load, so alerts fire only when users already suffer or the database has already failed over.
• Defaulting to heavyweight cloud instance types because scaling is easy. The tell: cost grows faster than traffic, with low utilization on large instances (a fast burn rate), and at scale the public cloud becomes very pricey.
• Mixing configurations within a role, or buying without tracking lead times. The tell: you cannot say what adding one more server buys you, and orders arrive after the deadline because drives, memory, or switches weren't pipelined.
• Testing only typical traffic. The tell: a viral or extreme scenario exposes a missing component in the data store or search tier, as with the Oscars tweet outage.

• ▸Role: name the architectural component (e.g., database server behind frontend web servers).
• ▸Performance/availability target: state the end-user performance and DR/failover requirement for this role.
• ▸Hardware/instance configuration: record the exact spec or instance type; every server in this role must match it.
• ▸Selection evidence: list what you measured on this workload; note any benchmark used and why it is or isn't representative.
• ▸Failure definition and measured limit: SLA violation, graceful degradation, or true failure, plus the load at which it occurs (e.g., QPS).
• ▸Spike allowance and headroom %: record the adjustment for periodic spikes and the safety factor subtracted.
• ▸Red-line number: the single resulting figure for this configuration in this role; set alert thresholds from it.
• ▸Procurement lead time and pipeline: list the items (drives, memory, switches), their lead times, and which roadmap milestone drives each.
• ▸Business justification: complete 'pages X percent faster, pageviews/ad revenue up to Y percent' for the budget request.