AlgoInsight

Free lesson · System Design Advanced

Prometheus Design Philosophy - Pull-Based Metrics for a Dynamic World

Hard10 min readsystem-designadvanced

Before Prometheus, the dominant monitoring model was push-based: every application instance ran an agent that periodically pushed metrics to a central collection server - Graphite, StatsD, Nagios, Zabbix. This model worked when your infrastructure was a fixed set of servers with stable IP addresses that changed once a quarter when someone racked new hardware. You knew your hosts. You configured your agents. Metrics flowed.

Then containers happened. A service that ran on 3 fixed servers now ran on 15 ephemeral pods that appeared, disappeared, and moved between nodes continuously. In the push model, each new pod needed to know where to push metrics, needed credentials for the metrics backend, and needed to be decommissioned from the metrics system when it died. At Stripe, we experienced this transition firsthand - our push-based monitoring system could not keep up with the churn rate of containerized services. Metrics arrived from pods that had already been terminated. New pods took minutes to register. Monitoring coverage gaps appeared precisely during the most dynamic periods - scaling events and deployments - which were exactly the moments we most needed visibility.

Prometheus inverted the model. Instead of applications pushing metrics to a collector, Prometheus pulls metrics from applications. Prometheus discovers targets through service discovery (Kubernetes API, Consul, DNS, file-based), scrapes each target's metrics endpoint on a configurable interval, and stores the results. When a pod appears, Kubernetes service discovery tells Prometheus about it. When a pod disappears, Prometheus stops scraping it. The monitoring system tracks the infrastructure automatically, with zero per-instance configuration.

Pull vs Push: The Architectural Difference

graph TD
    subgraph "Push-Based Monitoring (StatsD/Graphite)"
        APP1_P["App Instance 1<br/>──────────────────<br/>Runs a StatsD agent.<br/>Must be configured with:<br/>• Metrics server address<br/>• Credentials/API key<br/>• Flush interval<br/>• Which metrics to send<br/><br/>If the metrics server is down,<br/>metrics are lost (UDP) or<br/>the agent blocks/buffers (TCP)."]
        
        APP2_P["App Instance 2<br/>──────────────────<br/>Same agent configuration.<br/>Same credentials.<br/><br/>If this instance dies, the<br/>metrics server doesn't know —<br/>it just stops receiving metrics.<br/>Was the instance healthy but<br/>quiet, or did it crash?<br/>The server cannot distinguish."]
        
        APP1_P --> SERVER_P["Central Metrics Server<br/>──────────────────<br/>Receives metrics from all agents.<br/>Must handle bursts when many<br/>agents flush simultaneously.<br/><br/>Problem: the server has no way<br/>to know its own coverage. Is<br/>it receiving metrics from ALL<br/>instances? Or did 3 instances<br/>die and nobody noticed because<br/>the absence of data is silent?"]
        APP2_P --> SERVER_P
    end
    
    subgraph "Pull-Based Monitoring (Prometheus)"
        PROM["Prometheus Server<br/>──────────────────<br/>Actively scrapes targets on a<br/>schedule (every 15-30 seconds).<br/><br/>KNOWS its target list via service<br/>discovery (Kubernetes API, Consul).<br/><br/>If a scrape fails, Prometheus<br/>records the failure explicitly.<br/>A target that stops responding<br/>triggers an alert. The ABSENCE<br/>of data is a signal, not silence."]
        
        PROM --> APP1_PULL["App Instance 1<br/>──────────────────<br/>Exposes a /metrics HTTP endpoint.<br/>No agent. No push config. No<br/>credentials for the metrics<br/>backend. The app doesn't know<br/>or care who is scraping it.<br/><br/>Standard format: any language,<br/>any framework can expose<br/>Prometheus metrics."]
        
        PROM --> APP2_PULL["App Instance 2<br/>──────────────────<br/>Same /metrics endpoint.<br/>Zero configuration for<br/>monitoring - just expose<br/>the endpoint and Prometheus<br/>finds it automatically."]
        
        PROM --> SD["Service Discovery<br/>──────────────────<br/>Kubernetes API, Consul, DNS,<br/>or file-based discovery.<br/><br/>New pod starts → Prometheus<br/>learns about it automatically.<br/>Pod dies → Prometheus stops<br/>scraping, records the absence."]
    end

This diagram compares push-based and pull-based monitoring architectures. In the push model, each application instance runs an agent configured with the metrics server address, credentials, and flush interval. If an instance dies, the metrics server doesn't know - the absence of data is silent. The server has no way to verify its own coverage. In the pull model, Prometheus actively scrapes targets on a schedule. It knows its target list via service discovery from Kubernetes, Consul, or DNS. Applications expose a simple /metrics HTTP endpoint with zero push configuration. If a scrape fails, Prometheus records the failure explicitly - the absence of data is a signal, not silence. New pods are discovered automatically, and dead pods are detected immediately.

The Dimensional Data Model

Prometheus does not store metrics as flat hierarchical names like server1.cpu.usage. It uses a dimensional data model where every metric is identified by a name and a set of key-value labels. This seemingly small design decision - labels instead of hierarchical paths - has profound consequences for how you query, aggregate, and alert on metrics. It is the reason Prometheus can answer questions that Graphite and StatsD cannot express without restructuring your entire metric naming scheme.

graph TD
    subgraph "Dimensional Data Model"
        FLAT["Flat Naming (Graphite-style)<br/>──────────────────<br/>servers.web01.cpu.usage = 73.2<br/>servers.web02.cpu.usage = 68.1<br/>servers.web03.cpu.usage = 91.4<br/><br/>Problem: to get average CPU<br/>across all web servers, you need<br/>a wildcard: servers.web*.cpu.usage<br/><br/>To filter by region? You need<br/>a naming convention embedded in<br/>the metric path:<br/>us-east.servers.web01.cpu.usage<br/><br/>Rigid. Breaks when you want to<br/>slice data along a dimension<br/>you didn't anticipate in the<br/>naming scheme."]
        
        DIM["Dimensional (Prometheus-style)<br/>──────────────────<br/>cpu_usage{instance='web01',<br/>  region='us-east',<br/>  team='checkout',<br/>  environment='production'} = 73.2<br/><br/>To get avg across all web servers:<br/>avg(cpu_usage)<br/><br/>To filter by region:<br/>cpu_usage{region='us-east'}<br/><br/>To group by team:<br/>avg by (team) (cpu_usage)<br/><br/>Any label combination works<br/>as a query dimension. No<br/>naming convention needed.<br/>Add new labels anytime without<br/>breaking existing queries."]
    end

This diagram contrasts flat metric naming with Prometheus's dimensional data model. In the flat Graphite-style model, metrics are stored as hierarchical paths like servers.web01.cpu.usage. Querying across servers requires wildcards, and filtering by dimensions not in the naming path (like region or team) is impossible without restructuring the metric name. In Prometheus's dimensional model, each metric has a name and key-value labels: cpu_usage with labels for instance, region, team, and environment. Any label can be used as a query dimension: filter by region, group by team, aggregate across instances - all with the same metric, no naming conventions needed. New labels can be added anytime without breaking existing queries.

Important: The four metric types in Prometheus are counter, gauge, histogram, and summary. A counter only goes up (total HTTP requests, total errors) - use rate() to get the per-second rate of change. A gauge goes up and down (current temperature, memory usage, active connections). A histogram samples observations into configurable buckets (request latency distribution) - use histogram_quantile() to calculate percentiles. A summary calculates quantiles client-side (pre-computed p50, p90, p99). In interviews, the key distinction: counters are for events (things that happen), gauges are for states (things that are), and histograms are for distributions (how long or how big things are). Using the wrong type - like a gauge for request count - produces incorrect results when instances restart (the gauge resets, losing the count).

Service Discovery: How Prometheus Finds What to Scrape

The pull model only works if Prometheus knows what to scrape. In a static environment with 10 servers that rarely change, you could hardcode target addresses in the Prometheus configuration file and be done. But in a dynamic environment - Kubernetes clusters where pods appear and disappear every minute, auto-scaling groups that double capacity during traffic spikes, service meshes with sidecar proxies injected into every pod - hardcoded target lists are impossible to maintain. By the time you add a new target to the config, 3 others have been rescheduled to different IPs.

Prometheus solves this with pluggable service discovery mechanisms that integrate directly with the infrastructure layer. Each discovery mechanism watches the infrastructure for changes and dynamically updates Prometheus's target list. No manual configuration per service. No deployment pipeline integration required. When a new pod starts in Kubernetes, Prometheus learns about it within seconds and begins scraping - without anyone editing a configuration file.

graph TD
    subgraph "Prometheus Service Discovery"
        K8S_SD["Kubernetes SD<br/>──────────────────<br/>Prometheus watches the Kubernetes<br/>API for pods, services, endpoints,<br/>and nodes. When a pod with the<br/>annotation 'prometheus.io/scrape:<br/>true' starts, Prometheus adds it<br/>to the scrape target list.<br/><br/>Labels from K8s metadata<br/>(namespace, pod name, service<br/>name, node) are automatically<br/>attached to every scraped metric.<br/><br/>Most common SD in modern infra."]
        
        CONSUL_SD["Consul SD<br/>──────────────────<br/>Prometheus queries Consul's<br/>service catalog for registered<br/>services. Works for non-K8s<br/>environments: VMs, bare metal,<br/>hybrid deployments.<br/><br/>Consul tags become Prometheus<br/>labels for filtering."]
        
        DNS_SD["DNS SD<br/>──────────────────<br/>Prometheus resolves SRV or A<br/>records to discover targets.<br/>Simplest SD - works anywhere<br/>DNS works. Limited metadata<br/>(no labels from DNS)."]
        
        FILE_SD["File SD<br/>──────────────────<br/>Prometheus reads target lists<br/>from JSON/YAML files on disk.<br/>Files can be generated by any<br/>external system (CMDB, Terraform,<br/>custom scripts).<br/><br/>Most flexible - works with any<br/>infrastructure. Least dynamic —<br/>requires file updates for changes."]
        
        K8S_SD --> PROM_SD["Prometheus Scrape Loop<br/>──────────────────<br/>For each discovered target:<br/>1. HTTP GET /metrics every N secs<br/>2. Parse the response (text format)<br/>3. Attach service discovery labels<br/>4. Store in local TSDB<br/><br/>If scrape fails: record up=0<br/>for the target. Alert if down<br/>for more than threshold."]
        CONSUL_SD --> PROM_SD
        DNS_SD --> PROM_SD
        FILE_SD --> PROM_SD
    end

This diagram shows Prometheus's pluggable service discovery. Kubernetes SD watches the K8s API for pods with scrape annotations, automatically attaching namespace, pod, and service labels to metrics - the most common SD in modern infrastructure. Consul SD queries Consul's service catalog for non-Kubernetes environments. DNS SD resolves SRV or A records for simple discovery. File SD reads target lists from JSON or YAML files for maximum flexibility with any infrastructure. All discovery methods feed into the Prometheus scrape loop, which performs HTTP GET on each target's /metrics endpoint every N seconds, parses the response, attaches discovery labels, and stores in the local time-series database. If a scrape fails, Prometheus records up=0 for the target, making absence of data an explicit, alertable signal.

Tip: Kubernetes service discovery with annotations is the zero-config monitoring pattern. Add prometheus.io/scrape: "true" and prometheus.io/port: "8080" as annotations to your pod or service, and Prometheus discovers and scrapes the target automatically. No Prometheus configuration changes needed per service. This means a developer can add monitoring to a new microservice by exposing a /metrics endpoint and adding two annotations - Prometheus picks it up on the next service discovery refresh. This pattern scales to hundreds of services without touching the Prometheus configuration file.

The Production War Story

The monitoring philosophy shift that taught me the most about Prometheus happened during Stripe's migration from StatsD/Graphite to Prometheus in 2016. Our StatsD setup had a fundamental problem that we had normalized: we could not reliably answer the question "is our monitoring actually working?"

In the push model, when a service stopped sending metrics, the graphs in our dashboards simply went flat - showing the last received value. Was the service healthy but idle? Had the agent crashed? Was there a network partition between the agent and the metrics server? The absence of data in a push system is ambiguous. We discovered this the hard way during an incident where a payment processing service had been silently failing for 18 minutes. The service was crashing on startup, but because it never successfully initialized its StatsD agent, no metrics were ever pushed. The dashboard showed the last known good values from before the deployment. We were looking at stale data and thinking everything was fine.

Prometheus eliminated this class of blind spot entirely. When Prometheus scrapes a target and the scrape fails - connection refused, timeout, non-200 response - it records an explicit up{instance="...", job="..."} = 0 metric for that target. A simple alert rule up == 0 for 2m catches any target that stops responding. The absence of data is not silence - it is an alert. When we deployed the same payment service on Prometheus-monitored infrastructure, the crash-on-startup was detected within 30 seconds because the scrape immediately returned up=0. The dashboard showed "target down" instead of stale data. The incident was caught before a single user was affected.

That difference - between a monitoring system that goes silent when things break and a monitoring system that actively tells you when it cannot see a target - is the core argument for pull-based monitoring.

Six Takeaways

  1. Pull-based monitoring inverts the responsibility model. Instead of applications pushing to a collector, Prometheus pulls from applications. The monitoring system tracks infrastructure automatically via service discovery.

  2. Absence of data is a signal, not silence. When a scrape fails, Prometheus records up=0 explicitly. Push-based systems go silent when targets die - Prometheus actively detects and alerts on target failure.

  3. The dimensional data model enables flexible querying. Metrics have names and key-value labels. Any label can be used for filtering, grouping, or aggregation without restructuring the metric name.

  4. Four metric types serve different purposes. Counters for events (use rate()), gauges for states, histograms for distributions (use histogram_quantile()), summaries for pre-computed quantiles.

  5. Service discovery makes monitoring scale with infrastructure. Kubernetes SD, Consul SD, DNS SD, and file SD integrate Prometheus with the infrastructure layer so new targets are monitored automatically.

  6. The /metrics endpoint is the interface contract. Applications expose metrics via HTTP in a standard text format. Any language and framework can implement it. Prometheus does not care what the application is - only that it exposes /metrics.

Interview: "Why does Prometheus use a pull-based model?" is asked in observability interviews. The strong answer: "Pull-based monitoring gives the monitoring system control over its own data collection. Prometheus knows exactly which targets it should be scraping (via service discovery), scrapes them on a predictable schedule, and explicitly records when a scrape fails (up=0). In a push model, the monitoring system passively receives whatever agents send - it has no way to know if a target is healthy-but-quiet or crashed-and-silent. The pull model also simplifies application instrumentation: the app exposes a /metrics endpoint and does not need to know the address, credentials, or protocol of the metrics backend. In dynamic environments like Kubernetes where pods appear and disappear constantly, pull + service discovery provides continuous coverage without per-instance configuration."

Exercises

Exercise: Starter: A team runs 10 microservices monitored by a push-based system (StatsD → Graphite). (a) A new service is deployed but crashes before its StatsD agent initializes. What does the dashboard show? When does the team find out? (b) The same scenario with Prometheus: what does Prometheus record when the scrape fails? What alert fires? (c) A service scales from 3 pods to 8 pods. In the push model, what configuration changes are needed? In the pull model with Kubernetes SD? (d) Explain why pull-based is better suited for containerized, auto-scaling environments.

Exercise: Intermediate: Design the Prometheus metric instrumentation for an HTTP API service. (a) What metric type for total request count? Write the metric name and labels. (b) What metric type for request duration? Why histogram over gauge? (c) What metric type for current active connections? (d) Write the PromQL query for: request rate per second, p99 latency, and error rate (5xx / total). (e) A counter resets to 0 when the pod restarts. How does rate() handle this?

Exercise: Advanced: Compare Prometheus's pull model with OpenTelemetry's push model (OTLP) for a 500-service platform. (a) At what scrape interval and target count does Prometheus's pull model create meaningful load on the targets? Calculate: 500 services × 3 replicas × 1000 metrics each × 15-second scrape interval. (b) OpenTelemetry collectors receive push data and can batch, retry, and buffer. What advantages does this give over pull during network partitions? (c) Design a hybrid architecture using both: Prometheus for infrastructure metrics (pull) and OpenTelemetry for application traces and logs (push). How do they share labels for correlation? (d) Prometheus stores metrics locally. OpenTelemetry exports to backends. Compare operational complexity, data durability, and query flexibility.

Back to System Design Advanced