Backend Scaling Part 2: Load Balancers, Databases, CDNs & Serverless

Horizontal scaling requires statelessness enforced through load balancers, read replicas, and sharding. CDNs reduce latency via geographic distribution. Asynchronous processing cuts perceived latency. Microservices and serverless solve organizational and cost problems but introduce complexity. Always measure before optimizing; prefer simple solutions; scale for current problems, not hypothetical millions.

Statelessness: The Foundation of Horizontal Scaling

Statelessness Definition

Statelessness means no server instance holds data exclusive to itself. Any request can be routed to any server and produce identical results. This is the core property enabling horizontal scaling.

Session Management Problem

If server A stores a user's session in memory after authentication, server B cannot verify that user on the next request, causing 401 errors. Solution: store sessions in centralized Redis accessible to all instances.

File Storage Problem

Storing uploaded files on a single server instance makes them inaccessible to other instances. Solution: use centralized object storage like S3 or Cloudflare R2.

Statelessness Checklist

For horizontal scaling: use Redis for in-memory data, centralized databases instead of SQLite files, object storage for files, and managed services for all state. No piece of information should live on a single instance.

Load Balancers: Routing Requests Intelligently

Load Balancer Role

A load balancer sits between clients and multiple server instances, deciding which server receives each request and returning responses. It's the middleman that makes horizontal scaling possible.

Round-Robin Algorithm

Simplest load balancer algorithm: sends requests in rotating order to servers A, B, C, A, B, C. Works well when all requests have similar cost and all servers have equal capacity.

Weighted Round-Robin

Variation of round-robin that accounts for different server capacities. A server with 8 GB RAM and 4 cores receives twice as many requests as servers with 4 GB RAM and 2 cores.

Least Connections Algorithm

Load balancer routes to the server with the fewest active connections. Better than round-robin for mixed request types: expensive requests (2s) stay on one server while lightweight requests (200ms) get routed elsewhere.

Least Response Time Algorithm

Routes requests to servers returning responses fastest. Servers struggling with resources get fewer requests; responsive servers get more.

Health Checks: Detecting Dead Servers

Load balancer sends test requests (GET) every second to all servers. If a server returns non-200 response, it's blacklisted from user traffic but continues receiving test requests. Once healthy again, it's restored to the pool.

Database Scaling: Read Replicas and Sharding

Why Databases Are Hard to Scale

Databases are stateful and hold data that must be consistent across instances. Unlike stateless application servers, you cannot simply duplicate a database—data consistency becomes the primary challenge.

Read Replicas Architecture

One primary database handles all writes; multiple replica databases handle reads only. Replicas are distributed geographically to reduce latency for users in different regions. Approximately 70-90% of requests are reads, so replicas handle most traffic.

Replication Lag Problem

Data travels at light speed through fiber optic cables (~200,000 km/s). From US to India (~20,000 km) introduces ~100-200ms latency. User updates name, then immediately refreshes; read replica hasn't received the update yet, showing stale data.

Solutions for Replication Lag

Route write-related reads to primary database; track replication lag and wait for sync before returning data; add planned latency in frontend before fetching updated data.

Sharding (Partitioning)

Divide a large table (e.g., billions of orders) into multiple physical database instances based on a sharding key (e.g., order date). Each shard holds a subset of data. Reduces query latency and allows independent scaling of each shard.

Modern Distributed Databases

PlanetScale (MySQL), Neon (PostgreSQL), CockroachDB, Yugabyte automatically handle sharding, replication, and distributed transactions. Recommended approach: use managed services rather than rolling your own database infrastructure.

Content Delivery Networks (CDNs): Geographic Latency Reduction

Physics Constraint: Speed of Light

Light travels ~200,000 km/s through fiber optic cables. A request from Tokyo to US East (Virginia) travels ~20,000 km, creating minimum ~100ms round-trip latency. This is a hard physical limit.

CDN Edge Nodes

Place CDN nodes (points of presence) geographically close to users. Tokyo user requests content from Tokyo CDN node instead of US server. Reduces round-trip distance from 20,000 km to ~100 km, cutting latency from 100ms to 2-3ms.

CDN Benefits: Load Distribution

By caching content at CDN nodes, primary server receives ~50% less traffic. Users request from nearest CDN node; only cache misses hit the primary server.

Static Content Caching

Cache JavaScript bundles, CSS, HTML, images, videos, fonts at CDN. Single-page applications (React, Vue) are ideal: deploy bundle to CDN, users fetch from nearest node.

API Response Caching

Cache API responses for content that changes infrequently (product catalogs, blog posts). Use cache invalidation/purging strategies (tag-based) to refresh when data updates.

CDN Security: DDoS Mitigation

Cloudflare and similar CDNs sit in front of servers, absorbing DDoS attacks. Attacker traffic distributed across CDN's massive network instead of hitting origin server. Prevents crashes and financial damage from attack-triggered autoscaling.

Edge Computing: Processing at CDN Layer

Beyond caching, CDN nodes can execute code (e.g., Cloudflare Workers using V8 isolates). Example: authenticate requests at edge, reject unauthorized users in 2-3ms instead of 100ms round-trip to primary server.

Edge Computing Constraints

CDN nodes have limited resources (1GB RAM, 1 CPU core vs primary servers with 8-32GB). Cannot run complex logic or long-running tasks. Ideal for authentication, routing, user customization, validation.

Asynchronous Processing: Reducing Perceived Latency

Synchronous vs Asynchronous Behavior

Synchronous: user waits for response before seeing result (e.g., profile update). Asynchronous: user sees success immediately; background work completes later (e.g., sending email). Not about code execution model but user experience.

Team Invite Example

User invites team member: validate, save to database (100ms), send email via external provider (300ms) = 400ms total. Solution: save to database, return success, push email task to queue. User sees success in 100ms; email sent asynchronously.

Queue-Based Architecture

Producer (main server) pushes task to queue (Redis Queue, RabbitMQ). Consumer/worker picks task from queue, executes it. User doesn't wait; task completes whenever worker processes it.

Ideal Use Cases

Email/notification sending, video/image processing, user data deletion, webhooks, file uploads. Any operation where user doesn't need immediate feedback.

Implementation: Bull MQ

Popular Node.js library using Redis. Handles error handling, rate limiting, retries. Easy to implement background jobs without building queue infrastructure from scratch.

Microservices: Scaling Teams, Not Just Machines

Monolith Definition

Single codebase with all functionality (auth, payments, notifications, orders) as modules. Deployed as one unit. Simple to develop, test, and deploy but becomes problematic at scale.

Monolith Problems at Scale

Deployment dependency: one team's unfinished code blocks another team's deployment. Scaling: cannot scale payment service independently from notification service. Tech stack: cannot use different languages for different modules.

Microservices Definition

Multiple independent services, each with own codebase, database, and deployment cycle. Services communicate via HTTP/gRPC. Solves organizational problems (team autonomy, independent scaling, tech flexibility).

Microservices Trade-offs

Network calls replace function calls (latency, failures). Debugging spans multiple services (need distributed tracing). Data consistency across services (replication lag, eventual consistency). Only justified for large teams (100+ developers).

When to Use Microservices

Large teams with clear boundaries. Independent scaling needs per service. Different technology requirements per service. Not recommended for startups or small teams; complexity outweighs benefits.

Serverless Computing: Pay-Per-Execution Model

Traditional Server Model

Always-on VM with fixed capacity (4GB RAM, 2 CPU cores). Pay 24/7 regardless of traffic. Capacity planning required: underprovision = crashes; overprovision = wasted money.

Autoscaling Limitations

Reactive, not proactive: scales only after detecting high load. Takes seconds to minutes to boot new instances. Set minimum/maximum limits: too low = crashes, too high = unexpected bills.

Serverless Model

No pre-assigned servers. Code (functions) deployed. On request, provider spins up instance, runs code, returns response. Pay only for execution time (milliseconds of CPU). No 24/7 costs.

Cold Start Problem

First request requires spinning up new instance: boot OS, start runtime, load code. Takes milliseconds to seconds depending on runtime. Subsequent requests within few seconds reuse instance (warm start).

Cold Start Mitigation

Use fast runtimes: Cloudflare Workers (V8 isolates, ~5ms), AWS Lambda with Firecracker (~50-100ms). Avoid Java, heavy frameworks. Keep instances warm with periodic pings (but costs money).

Serverless Constraints

Execution timeout (Lambda: 15 minutes max). Limited resources (1GB RAM typical). Stateless only: no persistent connections, no local file storage. No TCP connections to databases.

Ideal Serverless Use Cases

Event-triggered operations: video processing, image resizing, file uploads. Webhooks. Scheduled tasks. Not suitable for: latency-sensitive apps (banking, payments), long-running processes, database-heavy operations.

Serverless Reality

Powerful for specific use cases but not universal server replacement. Industry overhyped. Best used for event-driven workloads, not primary backend. Requires rethinking architecture (statelessness, event-driven design).

Performance Engineering: Principles and Mindset

Always Measure First

Use observability tools (Prometheus, Grafana, New Relic) to measure latency, errors, resource usage. Identify specific bottleneck before optimizing. Premature optimization wastes effort on wrong problems.

Understand Where Time Goes

Trace every request: routing (ms), deserialization (ms), database query (50-100ms), external API call (200-300ms), serialization (ms). Total latency is sum of all components. Fix the largest contributor first.

Prefer Simple Solutions

Vertical scaling before horizontal. Database indexing before caching. Monolith before microservices. Kubernetes only when necessary. Each component adds operational complexity and failure points.

Scale for Current Problems

Build for your current scale with reasonable headroom (10-20% buffer). Don't architect for a million users on day one. Most platforms never reach that scale. Learn bottlenecks through real usage.

Observability from Day One

Exception to 'keep it simple': implement logs, metrics, traces from start. Prevents random crashes, enables quick diagnosis. Pays off immediately as system grows.

Performance is a Mindset

Continuous learning through trial and error. Build systems that handle problems gracefully when they occur. Develop diagnostic skills. No single tutorial teaches this; experience is key.

Notable quotes

The key enabler of horizontal scaling is statelessness—no instance should hold data exclusive to itself. — Instructor
Microservices are not about scaling machines; they're about scaling teams and human organization. — Instructor
Always measure first. Premature optimization is the worst thing you can do when solving performance problems. — Instructor

Action items

  • Implement health checks in your load balancer configuration to detect and blacklist failed server instances.
  • Set up centralized session storage (Redis) and object storage (S3) to enable stateless horizontal scaling.
  • Deploy observability stack (Prometheus, Grafana, or New Relic) to measure latency, errors, and resource usage across all system components.
  • Profile your application to identify bottlenecks: measure database query time, external API latency, and request processing time.
  • For read-heavy applications, configure read replicas in different geographic regions to reduce latency.
  • Implement asynchronous processing for non-critical operations (emails, notifications, video processing) using a queue library like Bull MQ.
  • Evaluate CDN integration for static assets (JavaScript, CSS, images) and cache API responses for infrequently-changing data.
  • Document your current scale and growth projections; only adopt microservices or serverless if justified by specific organizational or technical needs.
Glimpse.
Sriniously
2 hr 18 min video
4 min read
Backend Scaling Part 2: Load Balancers, Databases, CDNs & Serverless
You just saved 2 hr 14 min.
The big takeaway
Horizontal scaling requires statelessness enforced through load balancers, read replicas, and sharding. CDNs reduce latency via geographic distribution. Asynchronous processing cuts perceived latency. Microservices and serverless solve organizational and cost problems but introduce complexity. Always measure before optimizing; prefer simple solutions; scale for current problems, not hypothetical millions.
Statelessness: The Foundation of Horizontal Scaling
Statelessness Definition
Statelessness means no server instance holds data exclusive to itself. Any request can be routed to any server and produce identical results. This is the core property enabling horizontal scaling.
Session Management Problem
If server A stores a user's session in memory after authentication, server B cannot verify that user on the next request, causing 401 errors. Solution: store sessions in centralized Redis accessible to all instances.
File Storage Problem
Storing uploaded files on a single server instance makes them inaccessible to other instances. Solution: use centralized object storage like S3 or Cloudflare R2.
Statelessness Checklist
For horizontal scaling: use Redis for in-memory data, centralized databases instead of SQLite files, object storage for files, and managed services for all state. No piece of information should live on a single instance.
1
Use Redis for sessions and caching
2
Use centralized database (PostgreSQL, RDS)
3
Use object storage (S3, Cloudflare R2)
4
Never store state locally on instances
5
Ensure all instances are interchangeable
Statelessness requirements for horizontal scaling
Load Balancers: Routing Requests Intelligently
Load Balancer Role
A load balancer sits between clients and multiple server instances, deciding which server receives each request and returning responses. It's the middleman that makes horizontal scaling possible.
Round-Robin Algorithm
Simplest load balancer algorithm: sends requests in rotating order to servers A, B, C, A, B, C. Works well when all requests have similar cost and all servers have equal capacity.
1
Request 1 → Server A
2
Request 2 → Server B
3
Request 3 → Server C
4
Request 4 → Server A (cycle repeats)
Round-robin distribution pattern
Weighted Round-Robin
Variation of round-robin that accounts for different server capacities. A server with 8 GB RAM and 4 cores receives twice as many requests as servers with 4 GB RAM and 2 cores.
Least Connections Algorithm
Load balancer routes to the server with the fewest active connections. Better than round-robin for mixed request types: expensive requests (2s) stay on one server while lightweight requests (200ms) get routed elsewhere.
Least Response Time Algorithm
Routes requests to servers returning responses fastest. Servers struggling with resources get fewer requests; responsive servers get more.
Health Checks: Detecting Dead Servers
Load balancer sends test requests (GET) every second to all servers. If a server returns non-200 response, it's blacklisted from user traffic but continues receiving test requests. Once healthy again, it's restored to the pool.
1
Load balancer sends health check every 1 second
2
Server A responds with 200 → stays active
3
Server B fails to respond → blacklisted
4
Server B continues receiving health checks
5
Server B recovers → restored to active pool
Health check mechanism for server monitoring
Database Scaling: Read Replicas and Sharding
Why Databases Are Hard to Scale
Databases are stateful and hold data that must be consistent across instances. Unlike stateless application servers, you cannot simply duplicate a database—data consistency becomes the primary challenge.
Read Replicas Architecture
One primary database handles all writes; multiple replica databases handle reads only. Replicas are distributed geographically to reduce latency for users in different regions. Approximately 70-90% of requests are reads, so replicas handle most traffic.
Primary (writes only)
30 %
Replicas (reads only)
70 %
Typical read/write split in applications
Replication Lag Problem
Data travels at light speed through fiber optic cables (~200,000 km/s). From US to India (~20,000 km) introduces ~100-200ms latency. User updates name, then immediately refreshes; read replica hasn't received the update yet, showing stale data.
100-200ms
Replication lag (US to India)
Physical distance creates consistency delays
Solutions for Replication Lag
Route write-related reads to primary database; track replication lag and wait for sync before returning data; add planned latency in frontend before fetching updated data.
Sharding (Partitioning)
Divide a large table (e.g., billions of orders) into multiple physical database instances based on a sharding key (e.g., order date). Each shard holds a subset of data. Reduces query latency and allows independent scaling of each shard.
1
Orders table: 10 billion rows
2
Shard by date: Jan-Jun → Instance 1
3
Shard by date: Jul-Dec → Instance 2
4
Query latency reduced (5B rows vs 10B)
5
Can scale each shard independently
Sharding reduces data volume per instance
Modern Distributed Databases
PlanetScale (MySQL), Neon (PostgreSQL), CockroachDB, Yugabyte automatically handle sharding, replication, and distributed transactions. Recommended approach: use managed services rather than rolling your own database infrastructure.
Content Delivery Networks (CDNs): Geographic Latency Reduction
Physics Constraint: Speed of Light
Light travels ~200,000 km/s through fiber optic cables. A request from Tokyo to US East (Virginia) travels ~20,000 km, creating minimum ~100ms round-trip latency. This is a hard physical limit.
100ms
Minimum latency Tokyo to US East
Physics limit: cannot be optimized away
CDN Edge Nodes
Place CDN nodes (points of presence) geographically close to users. Tokyo user requests content from Tokyo CDN node instead of US server. Reduces round-trip distance from 20,000 km to ~100 km, cutting latency from 100ms to 2-3ms.
Without CDN
100ms latency
With CDN edge node
2-3ms latency
CDN dramatically reduces geographic latency
CDN Benefits: Load Distribution
By caching content at CDN nodes, primary server receives ~50% less traffic. Users request from nearest CDN node; only cache misses hit the primary server.
Static Content Caching
Cache JavaScript bundles, CSS, HTML, images, videos, fonts at CDN. Single-page applications (React, Vue) are ideal: deploy bundle to CDN, users fetch from nearest node.
API Response Caching
Cache API responses for content that changes infrequently (product catalogs, blog posts). Use cache invalidation/purging strategies (tag-based) to refresh when data updates.
CDN Security: DDoS Mitigation
Cloudflare and similar CDNs sit in front of servers, absorbing DDoS attacks. Attacker traffic distributed across CDN's massive network instead of hitting origin server. Prevents crashes and financial damage from attack-triggered autoscaling.
Edge Computing: Processing at CDN Layer
Beyond caching, CDN nodes can execute code (e.g., Cloudflare Workers using V8 isolates). Example: authenticate requests at edge, reject unauthorized users in 2-3ms instead of 100ms round-trip to primary server.
Edge Computing Constraints
CDN nodes have limited resources (1GB RAM, 1 CPU core vs primary servers with 8-32GB). Cannot run complex logic or long-running tasks. Ideal for authentication, routing, user customization, validation.
Asynchronous Processing: Reducing Perceived Latency
Synchronous vs Asynchronous Behavior
Synchronous: user waits for response before seeing result (e.g., profile update). Asynchronous: user sees success immediately; background work completes later (e.g., sending email). Not about code execution model but user experience.
Team Invite Example
User invites team member: validate, save to database (100ms), send email via external provider (300ms) = 400ms total. Solution: save to database, return success, push email task to queue. User sees success in 100ms; email sent asynchronously.
Synchronous (wait for email)
400ms
Async (queue email)
100ms
Asynchronous processing cuts perceived latency
Queue-Based Architecture
Producer (main server) pushes task to queue (Redis Queue, RabbitMQ). Consumer/worker picks task from queue, executes it. User doesn't wait; task completes whenever worker processes it.
1
Server receives request
2
Performs synchronous work (DB write)
3
Pushes async task to queue
4
Returns success to user
5
Worker picks task from queue
6
Completes async work (send email)
Queue-based asynchronous processing flow
Ideal Use Cases
Email/notification sending, video/image processing, user data deletion, webhooks, file uploads. Any operation where user doesn't need immediate feedback.
Implementation: Bull MQ
Popular Node.js library using Redis. Handles error handling, rate limiting, retries. Easy to implement background jobs without building queue infrastructure from scratch.
Microservices: Scaling Teams, Not Just Machines
Monolith Definition
Single codebase with all functionality (auth, payments, notifications, orders) as modules. Deployed as one unit. Simple to develop, test, and deploy but becomes problematic at scale.
Monolith Problems at Scale
Deployment dependency: one team's unfinished code blocks another team's deployment. Scaling: cannot scale payment service independently from notification service. Tech stack: cannot use different languages for different modules.
Microservices Definition
Multiple independent services, each with own codebase, database, and deployment cycle. Services communicate via HTTP/gRPC. Solves organizational problems (team autonomy, independent scaling, tech flexibility).
Microservices Trade-offs
Network calls replace function calls (latency, failures). Debugging spans multiple services (need distributed tracing). Data consistency across services (replication lag, eventual consistency). Only justified for large teams (100+ developers).
When to Use Microservices
Large teams with clear boundaries. Independent scaling needs per service. Different technology requirements per service. Not recommended for startups or small teams; complexity outweighs benefits.
Serverless Computing: Pay-Per-Execution Model
Traditional Server Model
Always-on VM with fixed capacity (4GB RAM, 2 CPU cores). Pay 24/7 regardless of traffic. Capacity planning required: underprovision = crashes; overprovision = wasted money.
Autoscaling Limitations
Reactive, not proactive: scales only after detecting high load. Takes seconds to minutes to boot new instances. Set minimum/maximum limits: too low = crashes, too high = unexpected bills.
Serverless Model
No pre-assigned servers. Code (functions) deployed. On request, provider spins up instance, runs code, returns response. Pay only for execution time (milliseconds of CPU). No 24/7 costs.
Traditional servers
Pay 24/7 for capacity
Serverless
Pay per millisecond executed
Serverless pricing model
Cold Start Problem
First request requires spinning up new instance: boot OS, start runtime, load code. Takes milliseconds to seconds depending on runtime. Subsequent requests within few seconds reuse instance (warm start).
5-1000ms
Cold start latency (varies by runtime)
Cold starts are serverless' main drawback
Cold Start Mitigation
Use fast runtimes: Cloudflare Workers (V8 isolates, ~5ms), AWS Lambda with Firecracker (~50-100ms). Avoid Java, heavy frameworks. Keep instances warm with periodic pings (but costs money).
Serverless Constraints
Execution timeout (Lambda: 15 minutes max). Limited resources (1GB RAM typical). Stateless only: no persistent connections, no local file storage. No TCP connections to databases.
Ideal Serverless Use Cases
Event-triggered operations: video processing, image resizing, file uploads. Webhooks. Scheduled tasks. Not suitable for: latency-sensitive apps (banking, payments), long-running processes, database-heavy operations.
Serverless Reality
Powerful for specific use cases but not universal server replacement. Industry overhyped. Best used for event-driven workloads, not primary backend. Requires rethinking architecture (statelessness, event-driven design).
Performance Engineering: Principles and Mindset
Always Measure First
Use observability tools (Prometheus, Grafana, New Relic) to measure latency, errors, resource usage. Identify specific bottleneck before optimizing. Premature optimization wastes effort on wrong problems.
Understand Where Time Goes
Trace every request: routing (ms), deserialization (ms), database query (50-100ms), external API call (200-300ms), serialization (ms). Total latency is sum of all components. Fix the largest contributor first.
Prefer Simple Solutions
Vertical scaling before horizontal. Database indexing before caching. Monolith before microservices. Kubernetes only when necessary. Each component adds operational complexity and failure points.
Scale for Current Problems
Build for your current scale with reasonable headroom (10-20% buffer). Don't architect for a million users on day one. Most platforms never reach that scale. Learn bottlenecks through real usage.
Observability from Day One
Exception to 'keep it simple': implement logs, metrics, traces from start. Prevents random crashes, enables quick diagnosis. Pays off immediately as system grows.
Performance is a Mindset
Continuous learning through trial and error. Build systems that handle problems gracefully when they occur. Develop diagnostic skills. No single tutorial teaches this; experience is key.
Worth quoting
"The key enabler of horizontal scaling is statelessness—no instance should hold data exclusive to itself."
— Instructor, at [0:31]
"Microservices are not about scaling machines; they're about scaling teams and human organization."
— Instructor, at [94:18]
"Always measure first. Premature optimization is the worst thing you can do when solving performance problems."
— Instructor, at [130:32]
Try this
Implement health checks in your load balancer configuration to detect and blacklist failed server instances.
Set up centralized session storage (Redis) and object storage (S3) to enable stateless horizontal scaling.
Deploy observability stack (Prometheus, Grafana, or New Relic) to measure latency, errors, and resource usage across all system components.
Profile your application to identify bottlenecks: measure database query time, external API latency, and request processing time.
For read-heavy applications, configure read replicas in different geographic regions to reduce latency.
Implement asynchronous processing for non-critical operations (emails, notifications, video processing) using a queue library like Bull MQ.
Evaluate CDN integration for static assets (JavaScript, CSS, images) and cache API responses for infrequently-changing data.
Document your current scale and growth projections; only adopt microservices or serverless if justified by specific organizational or technical needs.
Made with Glimpse by Wozart
glimpse.wozart.com/v/ps4gpfxn
Share this infographic

More like this