OS Internals: How Programs Run, Memory Works, and Web Servers Handle Requests
When you run a program, the OS creates a process with allocated stack and heap memory. A web server receives requests, spawns threads from a thread pool, and uses stack frames to execute code. The kernel manages thread state via TCBs and PCBs, releasing threads during I/O waits and resuming them when data returns. Understanding this is critical for cloud sizing, containerization, and resource-intensive tasks like video encoding.
Program Execution Basics
From Source Code to Running Process
When you write code and compile it into an executable (e.g., chrome.exe), clicking it sends a message to the OS to load the program into memory. The OS then creates a process—a container holding the program's runtime state. Multiple instances of the same executable create separate processes, each with its own memory allocation.
Essential Process Components
Every process contains stack memory (for function calls and local variables), heap memory (for dynamically allocated objects), and process state (snapshot of execution at a given time). These components together define the runtime behavior of the process.
Memory Layout and Segments
Data Segment: Static Variables
The data segment stores initialized static variables that persist across all instances of a program. For example, a static service fee in a booking application is stored once in the data segment and shared across all requests.
BSS Segment: Uninitialized Statics
The BSS (Block Starting Symbol) segment holds uninitialized static variables. These are allocated space but not yet assigned values, reducing executable file size.
Heap: Dynamic Object Allocation
The heap stores objects created at runtime (e.g., new Booking()). The runtime allocates heap memory when objects are instantiated and garbage collection cleans up unused objects after the stack frame completes.
Stack: Function Execution Flow
The stack holds local variables and executes the program's control flow. A stack frame represents one function or method call. Stack frames can read from and write to heap memory, but heap cannot access stack.
Web Server Request Handling
Request Flow Through Web Server
When a user sends a POST request to a web server (e.g., /booking), the web server receives it and passes it to the application runtime. The runtime invokes the handler method, which creates a stack frame to execute the request logic.
Stack Frame Lifecycle for Single Request
When a request arrives, the OS allocates a stack frame, reads static data from the data segment, creates heap objects (e.g., a Booking object), executes the business logic using both stack and heap, and finally releases the stack frame. The OS then cleans up heap memory via garbage collection.
Concurrency: Multiple Requests and Thread Pools
Thread Pool Management
A web server maintains a thread pool—a collection of reusable threads. When a request arrives, the server assigns a thread from the pool to handle it. Each thread gets its own stack frame, heap objects, and execution context. Threads are returned to the pool after request completion.
Handling Multiple Concurrent Requests
When two requests arrive simultaneously, each gets a separate thread from the thread pool. Thread 1 handles Request 1 with Stack Frame 1, and Thread 2 handles Request 2 with Stack Frame 2. Each stack frame has its own heap objects (e.g., separate Booking objects), but both threads can access shared data like a cache dictionary.
Kernel Role and I/O Management
Thread Release During I/O Calls
When a thread makes an I/O call (e.g., database query), the kernel releases the thread back to the thread pool instead of letting it idle. The kernel then waits for an I/O Completion Port (IOCP) signal indicating the I/O operation finished.
IOCP Signal and Thread Resumption
When the database returns data, it sends an IOCP (Input Output Completion Port) signal to the kernel. The kernel then retrieves a new thread from the pool and resumes execution using the saved thread state. This ensures no thread is idle waiting for I/O.
Process Control Block (PCB)
The PCB stores all process-level metadata: process ID, memory layout (text, data, BSS segments), stack and heap pointers, registers, and references to all thread control blocks (TCBs). The OS uses the PCB to manage the entire process.
Thread Control Block (TCB)
The TCB stores thread-specific state: registers, program counter, stack pointer, thread state (running/waiting/blocked), and the saved stack frame. When a thread is released during I/O, its TCB preserves the execution context so it can resume later.
Compilation and Runtime Memory Initialization
From DLL to Memory: Compilation Pipeline
A developer writes source code and compiles it into a DLL (dynamic link library) containing intermediate language (IL) code. When the runtime loads the DLL, it allocates a PCB, stack, heap, and memory segments in the OS. Each process instance gets separate memory allocations even though they run the same DLL.
Memory Segment Assignment Example
For a simple calculator program: static greeting string goes to data segment, uninitialized statics go to BSS, the Calculator object goes to heap, and local variables (x=10, y=20) plus the add() method execution go to stack. The stack frame accesses heap when needed.
Real-World Applications: Cloud and Modern Computing
EC2 Instance Sizing
Understanding stack and heap memory requirements helps right-size EC2 instances. Overprovisioning wastes money; underprovisioning causes crashes. By analyzing how much memory an application needs at runtime, engineers choose appropriately sized instances.
Containers as Processes
Each containerized application (in Docker, ECS, or Kubernetes) is a separate process from the host OS perspective. It has its own PCB, stack, heap, and memory segments. Container orchestration systems manage these processes just as the OS manages any process.
Lambda and MicroVMs
AWS Lambda functions run in short-lived microVMs. Although AWS abstracts process creation, understanding memory implications is critical. Developers must estimate heap and stack usage to avoid out-of-memory errors and optimize cold start times.
Netflix Video Encoding with cgroups
Netflix encodes raw video into multiple formats (240p, 720p, 1080p, 8K). Each encoding job runs as a separate process. The Linux kernel uses cgroups to allocate specific memory and CPU limits to each encoder process. CPU pinning assigns encoder processes to specific CPU cores, reducing context switches and improving cache locality.
CPU Pinning for Performance
CPU pinning assigns a process to specific CPU cores (e.g., cores 0-5 for one encoder, cores 6-11 for another). This reduces context switching overhead, improves CPU cache locality, and increases throughput during compute-heavy tasks like video encoding.
Key Concepts Summary
Stack vs. Heap Relationship
Stack holds the program's execution flow and local variables; heap holds dynamically allocated objects. Stack can read and write heap, but heap cannot access stack. When a stack frame completes, the OS cleans up associated heap memory.
Process Isolation and Resource Control
The OS isolates each process with its own memory space, PCB, and thread pool. Using cgroups and CPU pinning, administrators enforce resource limits (memory, CPU) per process, ensuring fair allocation and preventing one process from starving others.
Kernel as Orchestrator
The kernel manages thread pools, releases threads during I/O, listens for IOCP signals, restores thread state from TCBs, and schedules CPU time across processes. This orchestration ensures efficient resource utilization and responsiveness.
Notable quotes
Understanding OS fundamentals is essential for engineers at every layer. — Instructor
Stack can access heap, but heap cannot access stack. — Instructor
CPU pinning reduces context switches and improves cache locality during compute-heavy tasks. — Instructor
Action items
- Analyze your application's memory usage (stack and heap) to determine appropriate cloud instance sizes.
- Use cgroups and CPU pinning when deploying resource-intensive workloads to improve performance and efficiency.
- Profile your application to understand how much memory it allocates at runtime and optimize accordingly.
- When designing multi-threaded systems, consider thread pool sizing based on expected concurrent request volume.
- Monitor IOCP signal handling and thread release patterns to identify bottlenecks in I/O-bound operations.