Prometheus turns operational questions into a disciplined metrics model

01A monitoring system with a narrow center

Prometheus collects and stores time-series samples, evaluates expressions over them and exposes query and rule APIs. Its architecture keeps the core server relatively focused. The server discovers or accepts target information, scrapes HTTP metrics endpoints, writes samples to its local time-series database and evaluates recording and alerting rules. Exporters translate metrics from systems that do not expose the Prometheus format. Alertmanager, maintained as a separate project, receives alerts and handles grouping, silencing and notification routing.

That separation is important when comparing it with an all-purpose observability service. Prometheus is not a log index, trace store or general event database. Exemplars and integrations can connect metrics to other signals, but the native model remains numeric samples over time. This constraint is productive. It encourages teams to represent service health as rates, counts, durations, sizes and states that can be aggregated. It also prevents the core server from pretending that every diagnostic question belongs in one storage engine.

02Metric names tell the subject, labels define the dimensions

A Prometheus time series is identified by a metric name and a set of key-value labels. Samples add a timestamp and a floating-point value. Labels make one metric useful across instances, methods, status codes, regions or other bounded dimensions. They also power aggregation in PromQL. A request counter can be summed across instances, grouped by status family or compared between deployments without creating a separate metric name for every combination.

Every unique label set creates a separate series, which makes label cardinality an architectural concern. User IDs, request IDs, unbounded URLs and arbitrary error text can generate an explosive number of series. The cost appears in memory, storage, query work and human comprehension. The official naming and instrumentation guidance favors stable, meaningful dimensions. If a value is primarily useful for finding one event, it usually belongs in logs or traces. Metrics should answer aggregate operational questions across a controlled set of dimensions.

• Use counters for values that accumulate and derive rates over a time window for activity.
• Use gauges for values that can rise and fall, such as current queue depth or temperature.
• Use histograms or summaries for distributions only after understanding aggregation and bucket tradeoffs.

03Pull collection changes where discovery lives

Prometheus normally scrapes targets over HTTP at configured intervals. The server owns the schedule and records scrape health alongside target metrics. Static configuration works for a few fixed endpoints. Service-discovery integrations can turn information from Kubernetes, cloud providers or other systems into target groups, while relabeling selects targets and rewrites labels before ingestion. This puts a large part of collection policy in the monitoring system instead of requiring every application to know where to push data.

Pull is not a claim that every workload is long lived. The Pushgateway exists for specific short-lived service-level batch jobs, but the official documentation warns against using it as a general substitute for scraping. Pushed series can outlive their producer unless deleted, and the server loses some direct target-health semantics. Exporters, service discovery and carefully chosen gateways solve different problems. A design review should identify who owns target lifecycle and how stale metrics disappear rather than choosing push or pull by slogan.

04PromQL is where stored samples become evidence

PromQL selects time series by metric name and labels, then applies functions, arithmetic and aggregation. An instant vector represents a set of current samples, while a range vector contains samples across a time interval. Functions such as rate interpret counter movement over a window. Aggregation operators can group or collapse labels. Vector matching rules control how series from two expressions line up. These concepts are more important than memorizing a list of functions because subtle type or label mistakes can produce a plausible but incorrect chart.

Good queries preserve only the labels needed for the question. A service-level error ratio might sum request rates across replicas while retaining a service label. A capacity view might compare working-set bytes with a limit by workload. Before placing an expression in a dashboard or alert, inspect the series it returns and test how it behaves when a target disappears, a counter resets or traffic falls to zero. Empty vectors, missing labels and division by zero are semantic cases, not merely visual formatting problems.

05Rules make repeated questions explicit

Recording rules evaluate PromQL at intervals and store the result as a new time series. They reduce repeated computation for expensive or frequently used expressions and establish shared metric definitions. Alerting rules evaluate expressions and create alert instances with labels and annotations. A duration can require a condition to remain active before it fires, which helps distinguish sustained failure from a transient sample. Rule files can be versioned and checked with Prometheus tooling.

An alert expression is not a finished incident system. Alertmanager groups related alerts, removes duplicates, applies silences and inhibition, and routes notifications. The receiving team needs a service owner, severity model, useful annotations and a response procedure. Alerts based on internal implementation details often become noise. Alerts tied to user-visible symptoms, exhausted capacity or failed control loops are easier to act on. Every new rule should explain the decision a human or automation will make when it fires.

06Local storage is a deliberate boundary

The Prometheus server writes samples to a local time-series database organized into blocks, with a write-ahead log protecting recent data. Retention can be constrained by time or size. The storage documentation cautions that the local database is not clustered or replicated. Local autonomy makes a server simple to operate close to a failure domain and lets it continue collecting without a central dependency. It also means that disk durability, capacity and server recovery belong in the deployment design.

For longer retention or a broader query view, Prometheus supports remote write and remote read integrations, and federation can aggregate selected series from other servers. These are building blocks rather than one mandated global architecture. Remote write introduces queues, back pressure, external storage semantics and cost. Federation requires deliberate metric selection. A team should decide which server is authoritative for alert evaluation, how much local history is needed during an outage, and whether the remote system preserves the PromQL behavior its users expect.

07Scaling begins by reducing unnecessary work

Prometheus can handle substantial workloads, but the first scaling measure is often better instrumentation. Remove high-cardinality labels, avoid scraping unused collectors, choose sensible intervals and precompute shared expressions with recording rules. Querying a year of dense raw samples to render a frequently refreshed dashboard is a different workload from evaluating a five-minute error rate. Retention, resolution and query range should reflect the operational question.

Multiple independent Prometheus servers can separate teams, regions, clusters or failure domains. Identical servers can scrape the same targets for high availability, with Alertmanager deduplicating notifications based on label sets. A global query layer or remote storage can sit above them when required. None of these patterns eliminates label governance. A single newly introduced unbounded dimension can affect every replica and downstream store, so metric review belongs in application development rather than being left solely to the observability team.

08A good fit has measurable questions waiting for it

Prometheus is a strong choice when services can expose structured metrics and operators need to ask questions about rates, saturation, errors, latency and state. Its ecosystem, service discovery and alerting pipeline support dynamic infrastructure without making a central vendor endpoint part of every scrape. Apache-2.0 licensing makes the core repository permissive to use, modify and distribute subject to its terms and notices.

It is a weak fit when teams expect to store arbitrary high-cardinality events or search request bodies. It also cannot rescue an organization that has no ownership model for alerts. Start with a small set of service-level indicators and the system resources that explain them. Give every metric a bounded purpose and every alert a recipient. The repository supplies a rigorous data and query model; the monitoring program becomes trustworthy only when those constraints are maintained across instrumented applications.