Web Server from Scratch

Writing a small HTTP server is the fastest way to understand failures you normally only see in incident reports: slowloris, header bombs, request smuggling, connection exhaustion, broken keep-alive, and a single blocking handler that freezes every connection.

This chapter is a build process, not a framework tutorial. The goal is a server you can reason about: what it accepts, what it rejects, how it bounds memory, what happens when a client disappears, and how you prove it with tests.

Scope

In scope for v1

  • TCP listener and HTTP/1.1 request parsing.
  • GET/HEAD static files; POST echo.
  • Keep-alive, idle timeouts, header/body limits.
  • One process with a bounded concurrency model.
  • Structured access logs and request IDs.
  • Graceful shutdown.

Out of scope for v1

  • HTTP/2 and HTTP/3.
  • TLS termination (use a reverse proxy).
  • Compression, ranges, WebSockets.
  • Virtual hosts and complex routing.
  • Production-grade caching.

Actors and journeys

Actor Goal What can go wrong
Client Send a request and get a correct response Hung connection, wrong body length, unexpected close
Operator Keep the server responsive under load FD exhaustion, memory growth, blocked event loop
Attacker Exhaust resources or bypass path checks Slowloris, header bomb, traversal, smuggling
Developer Add a route without breaking framing New handler returns the wrong length or leaks a socket

Critical journeys:

  1. Accept a connection, parse one request, respond, and keep the connection usable.
  2. Reject an oversized or malformed request without reading unbounded data.
  3. Shut down without dropping in-flight requests.
  4. Under load, shed work predictably instead of collapsing.

Invariants

  1. A connection has exactly one request in flight at a time (HTTP/1.1 without pipelining support).
  2. Every response is correctly framed: Content-Length or chunked, never both, never neither for a body.
  3. The parser never reads more than the configured header/body limit.
  4. A path is contained in the configured root after decoding and resolution.
  5. A slow client cannot hold a connection forever.
  6. A client disconnect is not an error that crashes the process.
  7. Concurrency is bounded; overflow is rejected or queued with a limit.
  8. Logs contain no raw control characters and no sensitive headers.
  9. Shutdown stops accepting, drains, then exits.

End-to-end flow

1. Startup

start
  → parse config/flags
  → resolve document root (realpath)
  → create listener socket
  → set backlog, reuse-address, optional reuse-port
  → install signal handlers (SIGTERM/SIGINT)
  → start accept loop
  → log listening address and limits

Validate configuration before binding: root exists, limits are positive and compatible, timeout ordering is sane (idle < proxy timeout), and the process has enough file descriptors.

2. Accept

listener
  → accept()
  → if over connection limit: close or reject with 503
  → set socket options (no delay, keepalive)
  → create per-connection state
  → attach idle/header/body timers
  → read request line + headers up to cap

Rules:

  • Accept as fast as possible; do not do blocking work in the accept loop.
  • When at capacity, apply backpressure: stop accepting temporarily or reject with 503 and Retry-After.
  • Treat ECONNRESET/EPIPE as normal client behavior.
  • Never log the raw first line before validating it.

3. Parse the request

bytes
  → find CRLFCRLF within MAX_HEADER_BYTES
  → parse request line: method SP target SP version
  → validate method token and version
  → parse headers: field-name ":" OWS value
  → reject control chars, obs-fold, duplicate conflicting lengths
  → require Host for HTTP/1.1
  → determine body framing:
      Content-Length → exact N bytes
      Transfer-Encoding: chunked → parse chunks
      both → reject 400 (smuggling)
      neither → no body
  → enforce body limit / stream to disk

Detailed failure branches:

  • No CRLFCRLF before cap: 431 Request Header Fields Too Large; close.
  • Malformed request line: 400; close.
  • Unsupported version: 505 HTTP Version Not Supported.
  • Missing Host: 400.
  • Duplicate Content-Length with different values: 400.
  • Content-Length + Transfer-Encoding: 400; do not try to “pick one.”
  • Invalid chunk size: 400; close.
  • Header count/bytes over limit: 431.
  • Body over limit: 413; drain or close without reading the rest.
  • Slow header/body: idle timeout; close.

4. Route and handle

request
  → normalize path (percent-decode once, reject encoded separators if policy)
  → route lookup (exact/prefix)
  → authorize if needed
  → handler:
      static file → open, stat, conditional headers, stream
      echo → read body within limit
      health → constant response
  → build response headers
  → write status + headers + body

Static files:

  • Resolve the real path and verify it is inside the root.
  • Reject .., null bytes, and encoded traversal.
  • Use realpath and a separator-aware containment check.
  • Do not follow symlinks outside the root.
  • Set Content-Type from an allowlist; default to application/octet-stream.
  • Send ETag/Last-Modified; honor If-None-Match/If-Modified-Since with 304.
  • For HEAD, send headers with no body.
  • Stream large files; do not read them into memory.
  • Do not list directories by default.

5. Respond and keep alive

response
  → write status line + headers
  → write body (exact framing)
  → flush
  → if Connection: close or server shutting down: close
  → else reset idle timer and read the next request

Rules:

  • Content-Length must match the bytes written exactly.
  • If the handler throws after headers are sent, close the connection; you cannot change the status.
  • On a write error, close and discard the connection state.
  • Keep-alive timeout must be shorter than the proxy/load-balancer timeout to avoid races.
  • For HEAD, do not write the body even if the handler produced one.

6. Shutdown

SIGTERM
  → stop accepting new connections
  → mark server draining
  → send Connection: close on in-flight responses where possible
  → wait for active requests up to a grace period
  → close idle connections
  → exit 0
  → after grace period: force close, exit 1, log the count

Test shutdown under load. A server that drops requests on deploy will fail every rolling update.

Data structures and limits

Limit Why Starting point
Max header bytes Header bomb 16 KB
Max header count Parser abuse 100
Max request line Method/target abuse 8 KB
Max body Memory/disk safety per route, e.g. 1 MB echo
Max connections FD/memory safety based on ulimit -n
Idle timeout Slowloris/FD exhaustion 5–30 s
Header timeout Slow request line 5–10 s
Body timeout Slow upload route-specific
Write timeout Slow readers 10–30 s
Drain grace Deploy safety 10–30 s

The exact numbers matter less than the existence of the limits and the behavior when they are hit.

Concurrency models

Model How Trade-off
Thread per connection Blocking reads/writes Simple; expensive at 10k+ connections
Event loop + non-blocking State machine per socket Efficient; parsing is harder
Goroutines/virtual threads Blocking style, runtime multiplexing Simpler code; runtime overhead
Process per core + SO_REUSEPORT Kernel distributes accepts Isolation; coordination

For a learning build, start with one thread/task per connection and strict limits. Measure. Move to an event loop only if the concurrency target requires it. A correct blocking server beats a buggy event loop.

Edge cases

Case Expected behavior Notes
Request split across TCP packets Buffer until complete or timeout Never assume one read = one request
Multiple requests in one packet Process sequentially; no pipelining v1 Drain exactly one request
Content-Length: 0 Valid empty body Do not read
Chunked body Parse chunks, enforce total limit Reject malformed chunk sizes
Both length and chunked 400, close Smuggling
Negative/huge length 400 Integer overflow
Duplicate Host 400 or first-value policy Ambiguity
Absolute-form target Reject or normalize per policy Proxy semantics
Encoded traversal (%2e%2e) Decode, then containment check Double-encoding
Null byte in path 400 OS/path truncation
Symlink outside root 403 Containment after realpath
Directory request 404/403, no listing Information leak
HEAD on file Headers, no body Framing
If-None-Match matches 304, no body Correctness
Client disconnects mid-write Close, no crash EPIPE
Slowloris Header timeout, close FD exhaustion
Header bomb 431, close Memory
Body too large 413, close/drain Do not read rest
Too many connections Accept backpressure or 503 FD exhaustion
Handler blocks event loop Other connections stall Offload CPU/blocking work
Response headers too large Truncate? No — close or 500 Do not send malformed
Range request 416/ignore for v1 Document unsupported
HTTP/1.0 client Support or reject clearly Keep-alive semantics differ
Expect: 100-continue Reject or implement Do not hang
Malformed UTF-8 in headers Treat as opaque bytes/latin-1 Do not crash
Log injection Escape control chars Security
Shutdown with in-flight request Drain, then close Deploy safety

Failure modes and recovery

Failure Symptom Detection Mitigation Recovery
No header cap Memory blowup RSS growth Max header bytes/count Restart with limits; add test
Wrong framing Hung/desynced connections Client timeouts Exact Content-Length Close connection; fix handler
No idle timeout FD exhaustion Open sockets Idle/header timeouts Deploy with timeouts
Blocking handler Whole worker stalls Event-loop lag Offload blocking work Add worker pool/queue
Path traversal Data leak Security test Decode + containment Patch, audit logs
Unbounded body OOM/disk full Memory/disk Streaming + limits Reject route, clean temp files
Connection flood Accept loop starvation SYN/accept metrics Backlog, max connections, LB Rate limit upstream
Slow reader Write buffer growth Socket buffer Write timeout Close slow connections
Shutdown drops requests 5xx during deploy Deploy error spike Graceful drain Fix drain; retry deploy
Log volume Disk full Log rate Sampling/rotation Fix logging, rotate

Security and robustness

Threat Defence
Slowloris Header/idle timeouts, connection limits
Header bomb Max header bytes/count
Request smuggling Reject CL + TE, strict chunk parsing, normalize
Path traversal Decode once, resolve, containment check
Memory exhaustion Body limits, streaming, backpressure
Connection exhaustion maxConnections, accept backpressure
Log injection Escape control characters
SSRF via proxy routes Not in v1; if added, validate targets
TLS misconfiguration Terminate at a maintained proxy for v1

Add a request ID and structured access logs: method, path, status, bytes, duration, remote address, and error code. Do not log cookies or authorization headers.

Benchmarking honestly

  • Use wrk/hey/oha with keep-alive and realistic payloads.
  • Measure p50/p95/p99, not just RPS.
  • Run a release build with the same CPU count.
  • Test concurrency levels: 1, 10, 100, 1,000 connections.
  • Watch for coordinated omission in closed-loop generators.
  • Measure TLS/proxy separately.
  • Record the error rate; a fast server that drops requests is not fast.
wrk -t4 -c200 -d30s --latency http://127.0.0.1:8080/hello

Testing strategy

  • Parser unit tests: valid/invalid request lines, duplicate headers, chunked bodies, oversized headers, control characters.
  • Fuzz: truncated input, random bytes, huge values, malformed chunks.
  • Raw socket integration: keep-alive, pipelining, HEAD, Connection: close, partial writes.
  • Security: traversal, smuggling, slow client, header bomb, log injection.
  • Concurrency: many connections, blocking handler, slow reader, shutdown under load.
  • Load: percentiles and error rate at target concurrency.
  • Chaos: kill during write, fill disk, exhaust FDs, send reset.

Milestones and acceptance criteria

Milestone Build Accept when
1. Accept + parse Listener, request line, headers, limits Malformed/oversized requests are rejected without unbounded reads
2. Respond Status, headers, Content-Length, keep-alive Raw socket tests pass for GET/HEAD
3. Static files Root containment, MIME, streaming Traversal and symlink escapes are blocked
4. Robustness Timeouts, connection cap, body limits Slowloris/header bomb/flood tests pass
5. Concurrency Chosen model + backpressure p99 stable at target concurrency
6. Shutdown Drain + signal handling Deploy under load has zero dropped requests
7. Observability Request IDs + structured logs One request can be traced end to end

Review questions

  • What happens if a request arrives in 1-byte TCP segments?
  • How do you reject a header bomb without allocating it?
  • What does your server do with Content-Length and Transfer-Encoding together?
  • How do you prove a path cannot escape the root?
  • What is your behavior at the connection limit?
  • What happens when the client disconnects mid-response?
  • Does shutdown drop in-flight requests?
  • Which handler can block the event loop, and how do you detect it?
  • What do your benchmarks measure, and what do they hide?

Checklist

  • Header/body limits and timeouts are enforced.
  • Framing (Content-Length/chunked) is strict.
  • Keep-alive and Connection: close work.
  • Path traversal and header injection are blocked.
  • Concurrency is bounded; backpressure exists.
  • Client disconnects and write errors do not crash the process.
  • Graceful shutdown drains in-flight requests.
  • Access logs include request id, status, latency, and no secrets.
  • Benchmarks report percentiles with realistic concurrency.
  • Parser fuzzing and raw-socket tests exist.
  • TLS is delegated to a proxy or a maintained library.