Designing Data-Intensive Applications
One book, placed in its field — what Martin Kleppmann's book gets right, where it goes further than the rest, and what the rest of the field adds.
Mike West, Editor · Leading With Data · softwareengineeringprofessional.com/guides/profile-designing_data_intensive_applications
Arrow keys or swipe to move · N for notes · F for full screen
Designing Data-Intensive Applications
One book, placed in its field — what Martin Kleppmann's book gets right, where it goes further than the rest, and what the rest of the field adds.
Mike West, Editor · Leading With Data · softwareengineeringprofessional.com/guides/profile-designing_data_intensive_applications
The problem
Reliability erodes when it is left to instinct instead of method.
You were taught the moves piecemeal, never the whole model.
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
The model
- 01Data Model and Encoding Choice
- 02Storage Engine Design
- 03Replication Strategy
- 04Partitioning Strategy
- 05Transaction and Isolation Guarantee Level
- 06Distributed Environment Unreliability
- 07Workload Load
- 08Consensus and Ordering Mechanism
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
Step 1 of 5 · Data Model and Encoding Choice
Data locality is the real tradeoff of document models: great when you read the whole object, costly when you update a fragment.
Misconception
Practitioners believe document (schemaless) models eliminate the pain of schema management.
Reality
Document stores are schema-on-read: the schema still exists, but it moves into your application code as implicit assumptions scattered across every read path. You trade one explicit schema for many implicit ones.
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
Step 2 of 5 · Storage Engine Design
LSM-trees optimize write throughput at the cost of read and latency predictability; B-trees do the reverse.
Misconception
Engineers assume LSM-trees are always faster than B-trees because they write sequentially.
Reality
LSM-trees excel at write throughput but pay compaction overhead that can stall writes under sustained load and make read latency unpredictable; B-trees give steadier, more predictable latency. The right choice depends on whether your bottleneck is write volume or latency variance.
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
Step 3 of 5 · Replication Strategy
Synchronous vs. asynchronous is a durability-vs-availability dial, not a performance detail.
Misconception
Practitioners think adding read replicas gives them consistent reads for free.
Reality
Asynchronous followers lag the leader, so a user can write, then read from a replica and see stale or missing data — replication lag creates real consistency anomalies like reading your own writes disappearing. You must add read-your-writes and monotonic-read guarantees deliberately.
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
Step 4 of 5 · Partitioning Strategy
Hash partitioning trades away efficient range queries for even key distribution.
Misconception
Engineers believe hash partitioning always distributes load evenly and eliminates hotspots.
Reality
Hash partitioning spreads keys evenly but destroys range-query efficiency and does nothing about a single hot key — a celebrity user or viral item still funnels all traffic to one partition. Skew lives in access patterns, not just key distribution.
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
Step 5 of 5 · Transaction and Isolation Guarantee Level
Every isolation level is defined by the specific anomalies it forbids, not by its reassuring name.
Misconception
Practitioners assume that if their database is 'ACID' and set to a named isolation level, concurrency anomalies cannot occur.
Reality
Most databases default to read committed or snapshot isolation, neither of which prevents write skew or lost updates on interdependent reads. The word 'isolation' is marketing until you know exactly which anomalies your chosen level actually excludes.
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
The cheat sheet
- Data Model and Encoding ChoiceData locality is the real tradeoff of document models: great when you read the whole object, costly when you update a fragment.
- Storage Engine DesignLSM-trees optimize write throughput at the cost of read and latency predictability; B-trees do the reverse.
- Replication StrategySynchronous vs. asynchronous is a durability-vs-availability dial, not a performance detail.
- Partitioning StrategyHash partitioning trades away efficient range queries for even key distribution.
- Transaction and Isolation Guarantee LevelEvery isolation level is defined by the specific anomalies it forbids, not by its reassuring name.
- Distributed Environment UnreliabilityYou cannot detect failure directly — you can only observe the absence of a response, which is ambiguous.
- Workload LoadLoad is a set of parameters; name the one that actually drives cost before scaling.
- Consensus and Ordering MechanismLeader election, atomic commit, and uniqueness constraints are all consensus in disguise — solve them with one proven mechanism.
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data
Sources
- Designing Data-Intensive Applications — Martin Kleppmann
Designing Data-Intensive Applications · Mike West, Editor · Leading With Data