Backend Scaling and Performance: Part 1
Performance means low latency and high throughput. Latency varies per request and should be measured by percentiles (P50, P90, P99), not averages. Utilization and latency have an exponential relationship—never run systems at 100%. Always measure bottlenecks before optimizing. Key database concerns: N+1 queries, indexing strategy, and connection pooling. Caching patterns (cache-aside, write-through, write-behind) solve expensive operations but require careful invalidation. Vertical scaling is simple but hits hard limits; horizontal scaling avoids limits but introduces distributed systems complexity.
What is Performance?
Latency: The User's Experience
Latency is the total time from when a user clicks a button until they see the result on screen. It includes browser processing, network travel, server processing, database queries, external API calls, and response transmission. Latency is what users perceive as 'fast' or 'slow'.
Latency Varies Per Request
One request might complete in 50ms while another takes 200ms due to cache hits, server load, or network conditions. This variation is critical—averages hide the real story and are misleading for performance analysis.
Percentiles, Not Averages
P50 means 50% of users experience that latency or less. P90 means 90% experience that latency or less (10% experience worse). P99 means 99% experience that latency or less (1% experience worse). Percentiles reveal the distribution; averages hide outliers.
Why P99 and P95 Matter Most
Requests with highest latency (P99, P95) typically involve complex business logic, expensive database queries, and external service calls. These users often represent your most valuable customers (those making purchases or payments), so their experience directly impacts revenue and retention.
Throughput and the Latency-Utilization Curve
Throughput: Requests Per Second
Throughput measures how many requests your system can handle in a given time period, typically expressed as requests per second (RPS) or requests per minute. It answers questions like 'Can we survive a Black Friday traffic spike?'
Latency Increases with Throughput
As throughput increases, latency also increases—slowly at first, then exponentially. A system handling 10 RPS at 150ms latency may see 2 seconds latency at 1,000 RPS. Understanding both metrics together is essential for capacity planning.
Utilization and Exponential Latency Growth
Utilization is the percentage of system capacity in active use (0% = idle, 100% = maxed out). The relationship between utilization and latency is counterintuitive: latency grows linearly until ~80% utilization, then increases exponentially as you approach 100%. This is why you cannot run systems at full capacity.
Traffic Comes in Bursts
Real-world traffic is not linear or predictable. Requests arrive in bursts—high volume at one moment, zero the next. Without headroom (spare capacity), these bursts will push you past 100% utilization and cause system collapse.
Finding and Measuring Bottlenecks
Never Guess, Always Measure
When a system is slow, something specific is causing it—the bottleneck. Engineers often skip measurement and jump to standard solutions (add caching, upgrade database, add servers). This wastes time fixing problems that don't exist. Always measure each component's contribution to latency before optimizing.
Real Example: Logging Was the Culprit
A slow GET /products API was assumed to have a database problem, so caching was added. After measurement, the database query took only 10ms, the cache took 5ms, but synchronous logging to a remote service took 500ms. The real bottleneck was never the database—it was the logging.
Profiling: Measuring CPU Time
Profilers attach to running applications and record which functions execute, how long they take, and their call stacks. Flame graphs visualize this data, with wider bars representing functions that consume more time. However, profilers are better at measuring CPU-bound tasks than I/O-bound tasks (database, API calls, file I/O).
Distributed Tracing: Measuring I/O Latency
Tracing follows a single request through your entire system, recording timestamps at each component (business logic, database query, external API call). It reveals exactly where time is spent. For example, a request might spend 2ms in code but 800ms waiting on a database query.
Database Performance Optimization
N+1 Query Problem
Fetching a list of 20 blog posts, then making 20 separate queries to fetch each post's author, results in 21 total queries. For 1,000 items, you make 1,001 queries. This scales linearly and is wasteful. Solution: use JOIN or bulk fetch (prefetch_related, select_related, includes) to fetch all related data in 1-2 queries regardless of item count.
Indexes: Fast Lookups with Trade-offs
Indexes (typically B-tree structures) maintain a sorted copy of column values with pointers to rows, enabling fast lookups instead of full table scans. A query on 1 million rows might take 4 seconds without an index, 40-100ms with one. However, indexes consume disk space and must be updated on every INSERT, UPDATE, DELETE operation, slowing writes.
Indexing Strategy: Measure Before Adding
Don't index every column. Use distributed tracing to identify slow queries, then use EXPLAIN ANALYZE to see which columns are causing sequential scans. Add indexes only for columns that are frequently searched or joined. Primary keys are indexed by default. Composite indexes (multiple columns) and covering indexes (include extra columns) are advanced techniques.
Connection Pooling: Reuse Database Connections
Each new database connection requires TCP handshake, authentication, encryption negotiation, and memory allocation—expensive overhead. Connection pooling maintains a set of open, reusable connections. When a query finishes, the connection is returned to the pool instead of closed, ready for the next request. This eliminates repeated connection setup costs.
Internal vs. External Connection Pooling
Internal pooling (built into database drivers) maintains a pool per server instance. With horizontal scaling (multiple servers), each server has its own pool, risking exhaustion of the database's connection limit. External pooling (e.g., PgBouncer for PostgreSQL) maintains a single shared pool across all server instances, preventing connection limit exhaustion.
Caching Strategies and Patterns
Caching Basics: Store Expensive Results
Caching stores results of expensive operations (e.g., complex database queries) in fast storage (Redis, Memcached). On subsequent requests, return the cached result instead of re-executing the operation. Example: database query takes 800ms, but returning from cache takes 50ms—16x faster.
Cache Invalidation: The Hard Problem
Cache invalidation is notoriously difficult. When underlying data changes, cached data becomes stale. Two main techniques: (1) Time-based expiration—set a TTL and let cache auto-expire; (2) Event-based invalidation—explicitly delete cache on data updates. Time-based is simpler but risks staleness; event-based is precise but requires remembering to invalidate everywhere.
Cache-Aside Pattern (Lazy Loading)
Most common pattern. On request: check cache first; if hit, return immediately; if miss, query database, store result in cache, return result. On data update: delete cache entry so next request fetches fresh data. Simple and intuitive but risks serving stale data if cache invalidation is forgotten.
Write-Through Pattern
On data update: write to both cache and database simultaneously before returning success. Advantage: no cache misses, always fresh. Disadvantage: write operations are slower because they must update two systems. Good for read-heavy workloads.
Write-Behind Pattern
On data update: write to cache immediately and return success, then asynchronously update database. Advantage: fast writes. Disadvantage: risk of data loss if database write fails—cache and database become inconsistent. Use only when eventual consistency is acceptable.
Local vs. Distributed Caching
Local caching stores data in application server memory (fast, 2-3ms latency). Distributed caching uses external service like Redis (requires network round-trip, ~50ms latency). With multiple servers, local caches become inconsistent. Solution: tiered caching—local cache for hottest data, distributed cache as upstream.
Cache Hit Rate: The Key Metric
Cache hit rate is the percentage of requests successfully served from cache (e.g., 90% hit rate = 90% of requests avoid database). Low hit rate (e.g., 20%) indicates a problem with TTL, cache size, or access patterns. Hit rate depends on TTL length, cache size, and how well you understand user access patterns.
Scaling: Vertical vs. Horizontal
Vertical Scaling: Make One Server More Powerful
Vertical scaling (scaling up) means upgrading a single server's hardware: more CPU cores, more RAM, faster storage (SSD), faster network cards. A server with twice the CPU can handle roughly twice the requests. Simple, no architecture changes, no code changes.
Vertical Scaling: Advantages
Simple and intuitive—no distributed systems complexity. Code doesn't change. No load balancer needed. Economical: a server with 2x power costs less than two servers of half power. Avoids operational overhead of managing multiple machines (security, backup, maintenance).
Vertical Scaling: Hard Limits
No matter how much money you spend, there's a ceiling on server power. Cloud providers offer maximum instances (e.g., 32 cores, 96GB RAM). Once you hit the biggest available machine, you cannot scale vertically further. This is a hard, immovable limit.
Vertical Scaling: Single Point of Failure
One powerful server means one point of failure. If it crashes, your entire service is down—no redundancy. Mitigation requires standby servers or failover logic, adding complexity. Geographic distribution is impossible with one location.
Horizontal Scaling: Add More Servers
Horizontal scaling (scaling out) means adding more instances of the same server to work together. If one server handles 1,000 RPS, five servers handle 5,000 RPS (in theory). No hard limits—keep adding servers as needed. Requires load balancer to distribute traffic.
Horizontal Scaling: Advantages
No hard limits on capacity—scale indefinitely by adding servers. Redundancy: if one server fails, others absorb traffic. Geographic distribution: place servers worldwide to minimize user latency. Flexibility to handle traffic spikes by adding instances dynamically.
Horizontal Scaling: Complexity
Requires load balancer to distribute traffic among servers. Must choose load-balancing algorithm (round-robin, least connections, etc.). Must keep servers in sync (if user updates on server 1, server 2 must know). Network failures between servers cause split-brain scenarios. Requires monitoring to detect failed servers and route around them.
Horizontal Scaling: Distributed Systems Problems
Horizontal scaling introduces distributed systems challenges: consistency (keeping data in sync across servers), availability (handling server failures), partition tolerance (handling network failures). These are fundamental trade-offs—you cannot solve all three perfectly (CAP theorem). Each solution introduces new problems, just different ones.
Notable quotes
Never guess, always measure. Measurement is very important whenever we are talking about performance and scaling. — Instructor
You cannot run your systems at 100% utilization and also expect them to perform well. You need some amount of headroom. — Instructor
There are only two difficult problems in programming: naming things and cache invalidation. — Instructor
Action items
- Audit your current system's latency using percentiles (P50, P90, P99) rather than averages; identify which percentile is causing user complaints
- Implement distributed tracing (e.g., Jaeger, DataDog) to measure where each request spends time across your entire stack
- Review your database queries for N+1 problems; use your ORM's bulk-fetch features (prefetch_related, select_related, includes) or raw JOINs
- Run EXPLAIN ANALYZE on your slow queries to identify missing indexes; add indexes only for columns that are frequently searched or joined
- Implement connection pooling (internal driver-level or external like PgBouncer) if not already in place; monitor connection pool utilization
- Measure your cache hit rate; if below 70%, review your TTL settings, cache size, and access patterns
- Document your current system's utilization baseline; ensure production systems run at 60-80% utilization, not higher
- Profile your application using flame graphs or distributed traces to find the actual bottleneck before implementing standard optimizations