Free lesson · Microservices Interview
Centralised Logging at Scale
Centralised Logging at Scale
In a monolith, logs are in one place - one file, one server, one tail -f command. In a microservices architecture with 50 services running 200 pods across 3 availability zones, logs are in 200 different locations. Finding the log entry that explains why a specific request failed requires searching across all 200 pods - which is impractical without a centralised logging system.
Centralised logging collects logs from all services into a single searchable platform where engineers can query, filter, and correlate log entries across the entire architecture. Without it, debugging a production issue requires SSH-ing into individual pods and grepping through local log files - a process that takes minutes for simple issues and hours for cross-service investigations.
At Netflix, centralised logging is the first tool the on-call engineer opens when an alert fires. Before looking at traces, before checking dashboards, the engineer searches the centralised log system for the error that triggered the alert. The speed of this initial search - from alert to relevant log entries - determines how quickly the incident is triaged.
Log Aggregation Architecture
The centralised logging pipeline has three stages: collection, processing, and storage/search.
Collection
Log entries are collected from every pod and shipped to the processing layer. Three collection mechanisms:
Sidecar collector. A logging sidecar (Fluentd, Fluent Bit, Filebeat) runs alongside each service pod. The sidecar reads log files from the service's stdout/stderr (captured by the container runtime) and ships them to the processing layer. This is the Kubernetes-native approach - the container runtime writes stdout to a file, and the sidecar tails the file.
DaemonSet collector. A logging agent runs once per Kubernetes node (as a DaemonSet). The agent reads log files from all containers on the node and ships them to the processing layer. This is more resource-efficient than sidecars (one agent per node instead of one per pod) but provides less isolation (one noisy pod can delay log shipping for all pods on the node).
Direct shipping. The service sends log entries directly to the processing layer via HTTP or gRPC (using an OTel logging SDK or a library like Logstash). No file-based collection. This is the lowest-latency approach (no file I/O) but couples the service's availability to the logging system's availability - if the logging system is down, log shipping fails.
Recommendation: DaemonSet collector (Fluent Bit) for most Kubernetes deployments. Fluent Bit is lightweight (10MB memory), handles all log formats, and can buffer log entries locally if the processing layer is temporarily unavailable. Sidecar collection is justified for services that need custom log processing (parsing, redaction) before shipping.
Processing
The processing layer parses, transforms, and enriches log entries before storage.
Parsing. Raw log lines are parsed into structured fields. A JSON log entry {"level":"error","service":"order-service","traceId":"abc-123","message":"Payment failed"} is parsed into searchable fields: level, service, traceId, message.
Enrichment. The processing layer adds metadata that the service does not include: Kubernetes namespace, pod name, node name, deployment version. This metadata enables queries like "show me all error logs from the order-service in the canary deployment (version v2.4)."
Redaction. Sensitive data (credit card numbers, passwords, API keys, personal identifiable information) must be removed or masked before storage. A processing rule matches patterns (credit card numbers: 16-digit sequences, email addresses, social security numbers) and replaces them with redacted values (xxxx-xxxx-xxxx-1234). Redaction at the processing layer ensures that sensitive data never reaches the storage layer - even if a developer accidentally logs it.
Redaction is not optional - it is a compliance requirement. GDPR, PCI-DSS, and HIPAA all impose restrictions on storing personal or financial data in log systems. A credit card number that appears in a log entry makes the entire log system PCI-DSS scope - requiring the log storage to meet the same security standards as the payment system. Redaction before storage prevents this scope expansion.
A practical redaction pipeline:
- Match patterns: credit card numbers (Luhn-validated 16-digit sequences), email addresses (
\S+@\S+\.\S+), API keys (known prefixes likesk_live_,AKIA) - Replace matched values with redacted placeholders:
[REDACTED_CC],[REDACTED_EMAIL] - Log a counter of redacted entries per service for auditing: "order-service: 47 credit card numbers redacted in last hour"
Filtering. Not all log entries are worth storing. Debug-level logs from a healthy service in production consume storage without providing value. The processing layer can drop debug-level logs from specific services, keeping only info, warn, and error levels. This reduces storage costs by 50-80 percent without losing diagnostically valuable data.
Storage and Search
Two dominant platforms for log storage and search:
ELK Stack (Elasticsearch, Logstash, Kibana). The most widely adopted centralised logging platform. Elasticsearch provides full-text search and field-based filtering. Kibana provides dashboards and search UI. Logstash is the original processing layer (largely replaced by Fluent Bit for collection).
Pros: Powerful search capabilities. Rich Kibana visualisations. Mature ecosystem with extensive documentation. Cons: Elasticsearch is resource-intensive - storing and indexing billions of log entries requires significant memory, CPU, and disk. Cost scales with log volume. Operational complexity of managing Elasticsearch clusters (shard management, index rotation, capacity planning).
Grafana Loki. A log aggregation system designed for cost efficiency. Unlike Elasticsearch (which indexes the full content of every log entry), Loki indexes only the metadata labels (service name, log level, pod name) and stores the log content as compressed chunks. Full-text search requires scanning the chunks - slower than Elasticsearch's inverted index, but significantly cheaper because no full-text indexing is performed.
Pros: 10-50× cheaper than Elasticsearch for the same log volume. Integrates natively with Grafana (same dashboard for metrics and logs). Operationally simpler than Elasticsearch. Cons: Full-text search is slower. Complex queries (regex across millions of log entries) can take seconds instead of milliseconds. Not suitable for use cases that require fast full-text search across high-cardinality fields.
Recommendation: For teams already using Grafana for metrics dashboards, Loki provides a unified observability platform at lower cost. For teams that need fast full-text search across complex log formats (security log analysis, compliance auditing), Elasticsearch is the stronger choice.
Structured Logging - The Non-Negotiable Standard
As covered in Topic 1, structured logging (JSON format) is essential for centralised logging. Every log entry should be a JSON object with standardised fields.
The Standard Log Schema
Every log entry across all services should include:
| Field | Example | Purpose |
|---|---|---|
timestamp |
2025-03-15T10:30:15.234Z |
When the event occurred (ISO 8601, UTC) |
level |
error |
Severity: debug, info, warn, error, fatal |
service |
order-service |
Which service emitted the log |
traceId |
abc-123-def-456 |
Links the log entry to a distributed trace |
spanId |
span-789 |
Links to the specific span within the trace |
message |
Payment failed |
Human-readable description |
error.type |
PaymentDeclinedException |
Exception class (for error logs) |
error.stack |
at OrderService.processPayment(...) |
Stack trace (for error logs) |
Additional context fields should be added per log entry as appropriate: userId, orderId, requestPath, httpMethod, httpStatus, durationMs. These fields enable rich filtering - "show me all error logs for order ord-123" or "show me all logs where durationMs > 1000."
Log Levels in Production
Not all log levels are appropriate for production at full volume:
DEBUG - Detailed internal state, variable values, step-by-step logic. Useful for local development. In production, debug logs generate 10-100× the volume of info logs. Production policy: disabled by default. Enabled per service via a feature flag for time-limited debugging sessions (enable for 30 minutes, then auto-disable).
INFO - Business events: "order created," "payment processed," "user logged in." The primary production log level. Every significant business operation should emit an info-level log entry. Production policy: always enabled.
WARN - Recoverable problems: retry succeeded after failure, cache miss triggered database query, response time exceeded soft threshold. These are not errors (the request succeeded) but signals that something is degrading. Production policy: always enabled.
ERROR - Unrecoverable problems: request failed, exception thrown, downstream service unavailable. Every error log should include the stack trace and enough context to reproduce the issue. Production policy: always enabled. Every error log should be actionable - if an error fires 10,000 times per day and nobody investigates, it is either not an error (reclassify to warn) or a known issue (fix it or suppress it with a filter).
FATAL - Process-ending failures: out of memory, unrecoverable configuration error, database connection permanently failed. The service is shutting down. Production policy: always enabled. Every fatal log should trigger an immediate alert.
Warning: The most common logging mistake in production is leaving DEBUG enabled. A single service with debug logging enabled can generate 10GB of logs per day - overwhelming the log aggregation pipeline, consuming storage budget, and burying important error logs in noise. Debug logging must be disabled by default in production, with a mechanism to enable it temporarily per service when debugging a specific issue.
Correlation IDs - Linking Logs Across Services
The correlation ID (also called trace ID or request ID) is the key that links log entries across services for a single request. Without it, finding the log entries for a specific request across 10 services requires timestamp-based search - which is imprecise (multiple requests may occur at the same millisecond) and error-prone.
How Correlation IDs Flow
- The API Gateway assigns a unique correlation ID to every incoming request (or reads it from the
X-Request-IDheader if the client provides one) - The Gateway includes the correlation ID in the request headers forwarded to internal services:
X-Correlation-ID: abc-123 - Each service reads the correlation ID from the incoming request header
- Each service includes the correlation ID in every log entry it emits for that request
- Each service passes the correlation ID to any downstream service calls it makes
When debugging, the engineer searches the centralised log system for traceId=abc-123 and sees every log entry from every service for that specific request - in chronological order, across the entire call chain.
If OpenTelemetry is in use, the trace ID serves as the correlation ID - no separate correlation mechanism is needed. The OTel SDK automatically injects the trace ID into the logging framework's context (MDC in Java, context in Go), and every log entry includes it.
Log Sampling - Managing Volume at Scale
At high traffic (100,000 RPS), even with debug logging disabled, info-level logs can generate millions of entries per minute. Log sampling reduces the volume while preserving diagnostic value.
Sampling Strategies
Per-level sampling. All error and fatal logs are retained (100 percent). Warning logs are sampled at 50 percent. Info logs are sampled at 10 percent. Debug logs are not emitted in production. This tiered approach preserves the most valuable data (errors) while reducing the volume of routine data (info-level business events).
Per-service sampling. High-traffic services (the Product Service at 50,000 RPS) sample info logs at 1 percent. Low-traffic services (the Admin Service at 100 RPM) retain all info logs. The sampling rate is inversely proportional to the service's traffic volume.
Trace-correlated sampling. If a request's trace is sampled for distributed tracing, all log entries for that request are retained (regardless of the log-level sampling rate). This ensures that sampled traces have complete log context for debugging. If the trace is not sampled, logs are sampled independently.
Dynamic sampling. During an incident (elevated error rate), log sampling automatically reduces or disables - temporarily retaining 100 percent of all logs to maximise diagnostic data during the investigation. After the incident resolves, sampling returns to normal levels. This requires a dynamic configuration mechanism (feature flag, config server) that adjusts sampling rates without redeployment.
Tip: In interviews, mentioning log sampling shows cost-awareness. "At 100,000 RPS, I would sample info-level logs at 10 percent and retain 100 percent of error logs. For sampled traces, all correlated log entries are retained regardless of the sampling rate. This reduces log volume by 90 percent while preserving complete context for every error and every sampled trace."
Log Retention and Rotation
Log storage is not infinite. Retention policies determine how long logs are kept before deletion.
Hot storage (0-7 days). Recent logs in Elasticsearch or Loki. Fast search, full filtering capability. This is the primary debugging window - most production issues are diagnosed within 24 hours.
Warm storage (7-30 days). Older logs moved to cheaper storage (S3-backed Elasticsearch tier, Loki's compacted storage). Searchable but slower. Used for trend analysis and post-incident reviews.
Cold storage (30-365 days). Compressed logs archived to S3 or Glacier. Not searchable without rehydration. Retained for compliance, auditing, and legal requirements.
Index rotation. In Elasticsearch, create a new index daily (logs-order-service-2025-03-15). Delete indexes older than the retention period. This is more efficient than deleting individual log entries (index deletion is an O(1) metadata operation, entry deletion requires scanning and vacuuming).
Logging Anti-Patterns
Logging request and response bodies. Logging the full HTTP request and response body for every request generates enormous log volume and risks logging sensitive data (credit card numbers in payment requests, passwords in login requests). Log only the metadata (method, path, status code, latency) at info level. Log request/response bodies only for error cases, and only after redaction.
Unstructured error messages. "Something went wrong processing the order" is an error log that requires reading the stack trace to understand. {"error.type": "InsufficientStockException", "product_id": "prod-789", "requested": 5, "available": 0} is an error log that can be filtered, aggregated, and understood without reading the stack trace. Make error messages machine-parseable, not just human-readable.
Logging at the wrong level. A service that logs 10,000 error entries per hour for a known, benign condition (a deprecated API endpoint that returns 404) desensitises the team to error logs. If the condition is expected, it is not an error - reclassify it to warn or info. If the condition is unexpected and the team has chosen to ignore it, fix the underlying issue or add a suppression rule. Error logs should be rare enough that each one is worth investigating.
No log context on errors. An error log that says "NullPointerException at line 42" without the request ID, user ID, or business context requires the engineer to reproduce the error to understand it. Every error log should include enough context to diagnose the issue without reproduction: the request that triggered the error, the input data, the state of the entity being processed, and the operation being performed.
Interview: The centralised logging answer: "I would use structured JSON logging with a standard schema across all services. Logs are collected by Fluent Bit DaemonSet agents and shipped to Grafana Loki for cost-efficient storage and search. Every log entry includes the trace ID as the correlation key - I can search for a specific trace ID and see all log entries from all services for that request. Debug logging is disabled by default in production, enabled per-service via feature flag for time-limited debugging. Sensitive data is redacted in the processing pipeline before reaching storage. Info logs are sampled at 10 percent during normal operation and 100 percent during incidents."
What Comes Next
The next lesson covers metrics, SLOs, and alerting - how to define what "healthy" means for each service (SLIs and SLOs), how to set error budgets that balance reliability with feature velocity, and how to design alerts that fire only when they matter (avoiding alert fatigue).