Quality, chapter 07 of 10
Load & Performance Testing
Performance budgets tied to SLOs, the test types and when to run each, realistic traffic, capacity headroom, and the measurement trap that hides slow responses.
"It's fast on my machine" is a measurement of one user. Load testing is how you learn what happens with ten thousand, before they arrive on launch day and teach you in production.
Start From a Budget#
A load test without a target just produces graphs. Set the target first, from the SLOs: latency at percentiles ("p95 under 300ms and p99 under 1s at 500 requests per second"), an error rate under load ("fewer than 0.1% at peak"), and a throughput target with headroom, expected peak times a safety factor (2x is a reasonable default), because launches and incidents both arrive as spikes. The test passes or fails against those numbers, in CI where possible. It sits outside the test pyramid: it answers a different question.
Test Types#
Different questions need different shapes of load, and running only one leaves the others to production. The names are k6's, a useful vocabulary whatever the tool:
| Type | Load | Answers |
|---|---|---|
| Smoke | Minimal | Does the script and the system work at all? |
| Average-load | Normal traffic | Do we meet the budget on an ordinary day? |
| Stress | Above normal | How does it degrade, and does it recover? |
| Soak | Normal, for hours | Leaks, connection exhaustion, queues that slowly grow |
| Spike | A sudden surge | Does autoscaling keep up, or does everything fall over first? |
| Breakpoint | Rising until failure | Where is the actual limit, and what breaks first? |
Run smoke tests on every change. Run average-load before every launch and after architecture changes. Run stress, soak and spike on a schedule and before events you can see coming.
Make the Traffic Real#
A test that hammers one endpoint with one payload measures your cache.
- Model real traffic: the mix of endpoints, read/write ratio and payload sizes from production metrics.
- Realistic data volumes. Queries instant on 1,000 rows fall over on 100 million. Seed synthetic data at production scale, never a copy (Test Data & Environments).
- Open load models. Real users don't wait for your slow response before the next arrives. Generate load at an arrival rate, not a fixed number of looping users, or you undercount exactly the slow periods you're looking for.
- Somewhere production-like: same instance sizes, database tier and limits. Results from a smaller environment don't scale linearly.
Coordinated Omission#
The trap that makes most homemade benchmarks lie: when the system stalls, a closed-loop load generator stalls with it and stops sending requests, so the slowest moments are barely recorded. Gil Tene named it coordinated omission. Use tools that schedule requests at a fixed rate and measure from when the request should have been sent.
Tools#
My default is k6: JavaScript scripts, arrival-rate executors, and thresholds that fail the run. Gatling (scenarios as code, strong reporting) and Locust (Python, good for complex user behavior) are fine too. Whichever you use, the scripts live in the repository, reviewed and versioned with the code they test.
Capacity Planning#
- Find the bottleneck, then the next one. Load tests end at the first thing that saturates: connection pools, a lock, a downstream rate limit. Fix it and test again.
- Know your headroom: current peak versus the breakpoint. Revisit it as traffic grows, not after an outage.
- Test the limits you don't control, like provider quotas and third-party rate limits. Your capacity is the smallest of them.
Profile, Don't Guess#
Load tests say that it's slow; profiles say why. Capture CPU and allocation profiles and database query plans during the test, and optimize what the profile shows, not what you suspect.
References#
Reference
- Grafana k6: Test types, smoke, average-load, stress, soak, spike and breakpoint
- k6, load tests as JavaScript with pass/fail thresholds
- Gatling, load testing with scenarios as code
- Locust, load tests written in Python
In practice
- ScyllaDB: On coordinated omission, why closed-loop benchmarks hide slow responses
- AWS Well-Architected: Reliability pillar, testing for capacity and recovery before production does