Every KCNA candidate learns the sentence “Prometheus is the CNCF metrics system that scrapes /metrics endpoints”. The exam then asks questions that sentence does not answer: which metric type a request-duration histogram actually produces, why kubectl top works without Prometheus but a Grafana dashboard does not, what kube-state-metrics exposes that the kubelet does not, which PromQL function you apply to a counter, and why a batch job needs a Pushgateway when everything else is pulled. Those are not trivia. They are the difference between knowing the name of a tool and knowing how it fits into a cluster.
This guide is the deep dive the KCNA curriculum deserves for the one observability tool it names explicitly. Our cloud native architecture guide introduces the three pillars and gives Prometheus a dozen bullet points; this article is the rest of the story. Everything below follows the current Prometheus documentation (Prometheus 3.16, released 4 November 2026), the kube-state-metrics and Metrics Server project documentation, and the Kubernetes v1.37 documentation.
Where This Sits in the KCNA Curriculum
The KCNA curriculum was refreshed in late 2025 and now has four domains: Kubernetes Fundamentals (44%), Container Orchestration (28%), Cloud Native Application Delivery (16%) and Cloud Native Architecture (12%). Observability is a named competency inside Cloud Native Architecture, and Prometheus is the only monitoring project the curriculum calls out by name. In practice the questions spill into three domains:
| Domain | How Prometheus shows up |
|---|---|
| Cloud Native Architecture (Observability) | “Which metric type only increases?”, “Which component handles alert routing?”, “Which project is the CNCF standard for metrics?” |
| Container Orchestration (Troubleshooting) | “Which component must be installed for kubectl top to work?”, “Which exporter reports the number of Pod restarts?” |
| Kubernetes Fundamentals (Administration) | “Which Kubernetes API does the Horizontal Pod Autoscaler read CPU usage from?”, “Where does the kubelet expose container metrics?” |
The exam is multiple choice, so you will never write a scrape config under time pressure. You will be asked to pick the right component, the right metric type, the right function and the right boundary between tools that look similar. That is what this guide drills.
What Prometheus Is (and the Facts the Exam Checks)
Prometheus is an open-source systems monitoring and alerting toolkit. It was created at SoundCloud in 2012, joined the Cloud Native Computing Foundation in 2016 as the foundation’s second hosted project (after Kubernetes), and graduated in 2018. It is written in Go and ships as static binaries. The project’s own description of its main features reads like an exam cheat sheet:
- a multi-dimensional data model with time series identified by a metric name and key/value pairs;
- PromQL, a query language built to exploit that dimensionality;
- no reliance on distributed storage – a single Prometheus server is autonomous;
- collection happens via a pull model over HTTP;
- pushing is supported only via an intermediary gateway;
- targets are found through service discovery or static configuration;
- multiple modes of graphing and dashboarding support.
The ecosystem has several components, most of them optional:
| Component | Role | Exam cue |
|---|---|---|
| Prometheus server | Scrapes targets, stores time series in its local TSDB, evaluates rules, answers PromQL | ”scrapes and stores” |
| Client libraries | Instrument application code (Go, Java, Python, Ruby, .NET, Rust and more) and serve /metrics | ”expose custom application metrics” |
| Exporters | Sidecar or standalone processes that translate third-party systems (Linux kernel, databases, message brokers) into the Prometheus format | ”node_exporter”, “monitor something that cannot be instrumented” |
| Pushgateway | Accepts pushed metrics from short-lived jobs so Prometheus can scrape them later | ”batch job that exits before the next scrape” |
| Alertmanager | Receives alerts from Prometheus servers, deduplicates, groups, routes, silences and inhibits them, and sends notifications | ”route alerts to the on-call channel” |
| Grafana (separate project) | Dashboards on top of Prometheus and other data sources | ”visualise” |
Two boundaries matter for the exam. First, Prometheus handles metrics only. It does not store logs (that is Fluentd, Fluent Bit or Loki territory) or distributed traces (Jaeger, OpenTelemetry). Second, Prometheus ships only a basic expression browser; Grafana draws the dashboards. If a question asks for “the CNCF tool to visualise Prometheus data”, the answer is Grafana, not Prometheus.
The documentation also states when Prometheus does not fit: if you need 100% accuracy, such as per-request billing, Prometheus is the wrong tool, because scraped data is sampled and may be incomplete. The design goal is reliability during an outage, not audit-grade completeness. A question that pairs “billing” with “Prometheus” is testing whether you know this.
The Data Model: What a Time Series Actually Is
Prometheus stores everything as time series: streams of timestamped values that belong to the same metric and the same set of labels. Each series is identified by:
- a metric name, which describes the feature being measured (
http_requests_total); - zero or more labels, key/value pairs that capture the dimensions (
method="POST",handler="/api/tracks").
Each sample in a series is a float64 value (or, for native histograms, a composite histogram value) plus a millisecond-precision timestamp. The notation you will see everywhere is:
<metric name>{<label name>="<label value>", ...}
api_http_requests_total{method="POST", handler="/messages"}
Internally the metric name is itself a label with the reserved name __name__, so the series above is equivalent to {__name__="api_http_requests_total", method="POST", handler="/messages"}. Label names beginning with two underscores are reserved for internal use, which is why the service-discovery metadata you relabel from is called __meta_kubernetes_pod_name and not pod_name.
Rules worth memorising:
- Metric and label names may use any UTF-8 character since Prometheus 3.0 (November 2024), but should stick to
[a-zA-Z_:][a-zA-Z0-9_:]*for compatibility; names outside that set must be quoted in PromQL ({"http.requests.total", method="GET"}). - Colons are reserved for recording rules. Exporters and instrumented code should never emit
job:http_requests:rate5m-style names; that shape tells you a human wrote a rule. - A label with an empty value is equivalent to the label not existing.
- Changing any label value creates a new series. This is what people mean by cardinality: a label such as
user_idorrequest_idcreates one series per distinct value and can take down a server. Labels should carry dimensions with a bounded set of values.
Naming conventions the exam borrows
The naming best-practice page is where many question stems come from:
| Rule | Example |
|---|---|
| Single-word application or domain prefix | prometheus_notifications_total, process_cpu_seconds_total, http_request_duration_seconds |
| One unit and one quantity per metric | never mix seconds and milliseconds, never mix size and duration |
| Base units, not scaled ones | seconds not milliseconds, bytes not megabytes, meters not kilometers |
| Unit suffix in plural | _seconds, _bytes |
Accumulating counts end in _total | http_requests_total, process_cpu_seconds_total |
Pseudo-metrics that carry metadata end in _info | kube_pod_info, node_uname_info (value is always 1, the labels hold the data) |
Unix timestamps end in _timestamp_seconds | data_pipeline_last_record_processed_timestamp_seconds |
If a question shows container_memory_working_set_bytes and asks what it measures, the suffix tells you it is bytes, and the absence of _total tells you it is a gauge-like reading rather than an accumulating count.
The Four Metric Types, Properly
The client libraries offer four core metric types. A subtle but exam-relevant point from the documentation: with the exception of native histograms, the Prometheus server does not yet make use of the type information and flattens everything into untyped float series. The types exist in the instrumentation libraries and the exposition format, and they tell you which PromQL functions are appropriate.
Counter
A counter is a cumulative value that can only increase, or reset to zero when the process restarts. Requests served, tasks completed, errors seen, CPU seconds consumed: all counters. The documentation’s rule is explicit: do not use a counter for anything that can decrease, such as the number of currently running processes; that is a gauge.
Because counters only go up, their raw value is rarely interesting; the rate of change is. That is why rate(), irate() and increase() exist and why they are designed to handle counter resets. The trap the exam sets is a question that shows http_requests_total and asks for the function that gives requests per second; the answer is rate(http_requests_total[5m]), never sum(http_requests_total) on its own.
Gauge
A gauge is a single numerical value that can go up and down: temperature, current memory usage, number of concurrent requests, queue depth, number of running Pods. You read gauges directly, aggregate them with sum, avg, max, and use delta() or deriv() rather than rate() when you care about change. Applying rate() to a gauge is a classic distractor.
Histogram
A histogram records observations (usually request durations or response sizes) by counting them into configurable buckets, and it also keeps the sum of all observed values. The documentation’s one-line summary is that a histogram is “essentially a bucketed counter”.
Histograms come in two forms. A classic histogram with base name http_request_duration_seconds produces several time series:
http_request_duration_seconds_bucket{le="0.1"} 24054 # cumulative: observations <= 0.1 s
http_request_duration_seconds_bucket{le="0.5"} 129389
http_request_duration_seconds_bucket{le="1"} 133988
http_request_duration_seconds_bucket{le="+Inf"} 144320 # equals _count
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320
Three details are testable: the le label is the upper inclusive bound; the buckets are cumulative (the le="0.5" bucket includes everything in le="0.1"); and _bucket{le="+Inf"} always equals _count. Quantiles are calculated on the server, at query time, with histogram_quantile(), which means you can aggregate histograms across instances first and ask for the 95th percentile of the whole fleet.
A native histogram is exposed and ingested as a single composite sample that carries the count, the sum and a dynamic set of exponentially spaced buckets. Native histograms need no bucket configuration at instrumentation time, use far fewer series, are always aggregatable with each other even when resolutions differ, and travel atomically over remote write. They were experimental for several releases and became a stable, opt-in feature in Prometheus 3.8 (November 2025), enabled with the scrape_native_histograms setting. For the exam, “native histogram” means “the newer, more efficient histogram representation”; the classic _bucket/_sum/_count triple is still what most exporters emit.
Summary
A summary also samples observations and also exposes _sum and _count, but instead of buckets it exposes pre-calculated φ-quantiles over a sliding window, as <basename>{quantile="0.99"}. The quantiles are computed in the client, which has two consequences the exam likes: the calculation is comparatively expensive in the instrumented process, and you cannot aggregate quantiles across instances (the average of two 99th percentiles is not the 99th percentile of the combined population). You also cannot ask for a quantile you did not configure up front.
Side by side
| Counter | Gauge | Classic histogram | Native histogram | Summary | |
|---|---|---|---|---|---|
| Direction | Only up (resets to 0 on restart) | Up and down | Buckets are counters | Composite sample | Count/sum are counters |
| Series produced | 1 | 1 | _bucket per bucket + _sum + _count | 1 | one per quantile + _sum + _count |
| Where quantiles are computed | n/a | n/a | Server (histogram_quantile) | Server | Client, at instrumentation time |
| Aggregatable across instances | Yes | Yes | Yes, if bucket bounds match | Yes, always | Count and sum yes, quantiles no |
| Typical function | rate(), increase() | direct, avg_over_time() | histogram_quantile(0.95, sum by (le) (rate(..._bucket[5m]))) | histogram_quantile(0.95, sum(rate(...[5m]))) | read {quantile="0.95"} directly |
| Exam cue | ”only increases”, “requests served" | "current”, “can decrease" | "le label”, “buckets" | "no bucket config”, “fewer series" | "quantile label”, “cannot aggregate” |
Two more conventions: the output of rate() on a counter is gauge-like (a per-second value that can fall), and the output of rate() on a counter histogram is a gauge histogram. If a question says “a histogram that represents the current state of a distribution rather than an accumulation”, it is describing a gauge histogram.
The Exposition Format: What /metrics Actually Returns
Instrumented processes and exporters serve metrics on an HTTP endpoint, by convention /metrics, in a line-oriented text format. A minimal response looks like this:
# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"} 3 1395066363000
# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320
What to know:
# HELPcarries the description and# TYPEthe metric type; both are optional comments, but they are how the type information reaches Prometheus at all.- Each sample line is
name{labels} value [timestamp]; the timestamp is optional and almost always omitted, because Prometheus stamps samples with the scrape time. - The classic Prometheus text format is served with
Content-Type: text/plain; version=0.0.4. - OpenMetrics is the standardised evolution of that format, served as
application/openmetrics-text; version=1.0.0. It adds exemplars (references from a histogram bucket to a trace ID), explicit# EOFtermination and stricter typing. The OpenMetrics specification started as its own CNCF project and is now maintained inside the Prometheus project; OpenMetrics 2.0 exists but is experimental behind theopenmetrics2feature flag. - There is also a protobuf exposition format, currently needed to expose native histograms.
- Since Prometheus 3.0, scrape targets must return a valid
Content-Typeheader. A missing or unsupported content type makes the scrape fail unless afallback_scrape_protocolis configured. This broke a surprising number of hand-written exporters during 3.0 upgrades and is exactly the kind of “what changed in the major version” fact an exam writer enjoys.
The exam does not expect you to write exporters, but it does expect you to recognise that an application “exposing a /metrics endpoint in the Prometheus text format” is instrumented and can be scraped directly, whereas a database or a Linux host needs an exporter.
Pull, Not Push: How Scraping Works
Prometheus pulls. On a schedule (the global scrape_interval, 1 minute by default; scrape_timeout defaults to 10 seconds) it issues an HTTP GET to every target’s metrics path and stores whatever comes back. Each scrape adds two things on top of the target’s own metrics:
- the
upmetric:up{job="api", instance="10.244.1.7:8080"}is1when the scrape succeeded and0when it failed.up == 0is the first alert every team writes. - the
jobandinstancelabels.jobis the scrape job name from the configuration;instanceis thehost:portof the target. If the scraped metrics already carry a label with the same name, Prometheus keeps its own and renames the target’s toexported_job, unlesshonor_labels: trueis set. Other scrape-time series includescrape_duration_secondsandscrape_samples_scraped.
Why pull? The documentation’s reasoning, in exam form: Prometheus can tell the difference between “the target is down” and “the target has nothing to say” (that is what up encodes); you can run a second Prometheus against the same targets for redundancy or testing without touching the applications; and you can hit a /metrics endpoint with a browser or curl to debug instrumentation. The cost is that Prometheus must be able to reach the target, which is why service discovery exists.
The Pushgateway and when it is wrong
Some workloads cannot be scraped because they do not live long enough: a cron job that runs for four seconds will usually exit between two scrapes. The Pushgateway solves exactly this: the job pushes its final metrics to the gateway, and Prometheus scrapes the gateway. The documentation is blunt about its limits, and so is the exam:
- Use it only for service-level batch jobs, not as a general push path for applications.
- Metrics pushed to a gateway persist until deleted, so a job that stops running will keep reporting its last values; Prometheus cannot tell it is gone.
- The gateway becomes a single point of failure and a bottleneck, and the
upmetric now describes the gateway, not the job.
So the answer to “a nightly ETL job finishes before Prometheus can scrape it; what do you use?” is the Pushgateway. The answer to “an application team wants to push metrics instead of being scraped” is to instrument the application and let Prometheus pull, or to send metrics via the OpenTelemetry Collector to Prometheus’ OTLP receiver (see below).
Push paths that do exist
Two more routes carry data into a Prometheus server and are worth knowing so you do not over-apply “Prometheus never accepts pushes”:
- Remote write: another Prometheus (or an agent) streams samples to a receiver that has
--web.enable-remote-write-receiverenabled. This is how fleets ship data to long-term storage. - OTLP receiver: Prometheus can accept OpenTelemetry metrics over OTLP at
/api/v1/otlp/v1/metricswhen theotlpconfiguration block is enabled.
Neither replaces scraping as the primary collection model; both are integration paths.
Service Discovery in Kubernetes
Static target lists die the moment a Deployment scales. Prometheus therefore discovers targets dynamically, and the Kubernetes discovery mechanism (kubernetes_sd_configs) is the one the exam cares about. It talks to the API server (using the Pod’s own ServiceAccount token and CA certificate when Prometheus runs in-cluster) and watches one of six roles:
| Role | Discovers | Typical use |
|---|---|---|
node | One target per Node (kubelet address) | Scrape kubelet, cAdvisor, node-level metrics |
pod | One target per Pod per declared container port | Scrape application Pods directly |
service | One target per Service port | Blackbox-style checks of a Service’s DNS name |
endpoints | One target per Endpoints address (legacy) | Scrape every backend behind a Service |
endpointslice | One target per EndpointSlice endpoint (preferred) | Same as endpoints, scales better |
ingress | One target per Ingress path | Blackbox checks of external entry points |
Discovery produces targets decorated with __meta_kubernetes_* labels (namespace, Pod name, labels, annotations, node name). Relabeling (relabel_configs) then decides which targets to keep, what address and path to scrape, and which metadata to promote into real labels. The widely used prometheus.io/scrape: "true", prometheus.io/port and prometheus.io/path Pod annotations are a convention implemented by relabel rules, not something Kubernetes or Prometheus understands natively. If a question implies the annotation itself makes Prometheus scrape a Pod, the missing piece is the relabel rule that honours it.
The Prometheus Operator and its CRDs
Most clusters do not hand-write scrape configs. The Prometheus Operator (part of the kube-prometheus project) turns configuration into Kubernetes objects, which is a textbook example of the operator pattern and custom resources the KCNA also tests:
| CRD | What it declares |
|---|---|
Prometheus / Alertmanager | The server instances themselves (replicas, retention, storage) |
ServiceMonitor | ”Scrape every Service matching this selector on this port” |
PodMonitor | Same for Pods without a Service |
ScrapeConfig | A raw scrape job as a CRD |
PrometheusRule | Recording and alerting rules |
AlertmanagerConfig | Routing and receivers for a namespace |
The operator watches these objects and regenerates the Prometheus configuration. The kube-prometheus-stack Helm chart bundles the operator, Prometheus, Alertmanager, Grafana, node_exporter, kube-state-metrics and a set of default dashboards and alerts, which is why it is the usual answer to “install a complete monitoring stack on Kubernetes”.
Where Kubernetes Metrics Come From
The exam’s favourite confusion is which component produces which number. The honest answer is that a Kubernetes cluster has at least six metric sources, and a complete monitoring setup scrapes all of them:
| Source | Endpoint | What it reports | Example metric |
|---|---|---|---|
| kubelet | https://<node>:10250/metrics | Kubelet’s own operations | kubelet_running_pods, kubelet_pod_start_duration_seconds |
| cAdvisor (embedded in the kubelet) | /metrics/cadvisor | Per-container resource usage from cgroups | container_cpu_usage_seconds_total, container_memory_working_set_bytes |
| kubelet resource endpoint | /metrics/resource | A slimmed set of CPU/memory metrics per container and Pod | container_cpu_usage_seconds_total, pod_memory_working_set_bytes |
| kube-apiserver | /metrics on 6443 | API request rates, latencies, etcd call latencies, admission timings | apiserver_request_total, apiserver_request_duration_seconds |
| kube-scheduler, kube-controller-manager, etcd, CoreDNS | each component’s /metrics | Scheduling latency, queue depths, fsync latency, DNS request counts | scheduler_pending_pods, etcd_disk_wal_fsync_duration_seconds, coredns_dns_requests_total |
| kube-state-metrics | its own Service on 8080 | The desired and observed state of API objects, not resource usage | kube_deployment_status_replicas_available, kube_pod_container_status_restarts_total, kube_node_status_condition |
| node_exporter (DaemonSet) | :9100/metrics | Host-level Linux metrics | node_cpu_seconds_total, node_filesystem_avail_bytes |
| Your application | its /metrics | Whatever you instrumented | http_requests_total |
Kubernetes system-component metrics follow a stability policy (ALPHA, BETA, STABLE, and deprecated metrics hidden behind --show-hidden-metrics-for-version), mirroring the API lifecycle the community and governance guide covers.
The pairing to memorise: cAdvisor tells you how much CPU a container is using; kube-state-metrics tells you how many replicas a Deployment has available. One is a resource measurement, the other is object state. A question that says “alert when a Deployment has fewer available replicas than desired” is a kube-state-metrics question. A question that says “find the container using the most memory” is a cAdvisor question.
kube-state-metrics vs metrics-server vs cAdvisor
This is the single most reliable Prometheus question on the KCNA, because three components with similar names do three different jobs.
| metrics-server | kube-state-metrics | cAdvisor | |
|---|---|---|---|
| What it is | A Kubernetes add-on that implements the Metrics API (metrics.k8s.io) | An exporter that turns API object state into Prometheus metrics | A library embedded in the kubelet that reads cgroups |
| Data | Latest CPU and memory usage per Pod and Node, in memory only | Replica counts, Pod phases, restart counts, conditions, resource requests and limits | Per-container CPU, memory, network, filesystem usage |
| Who reads it | Horizontal Pod Autoscaler, Vertical Pod Autoscaler, kubectl top | Prometheus (and therefore Grafana, Alertmanager) | Prometheus, via the kubelet |
| History | None – latest value only, collected every 15 s | None itself; Prometheus stores the history | None itself; Prometheus stores the history |
| Protocol | Kubernetes aggregated API; needs the aggregation layer and kubelet webhook authn/authz | Prometheus text format | Prometheus text format |
| What it is not for | The project README says plainly: autoscaling only, do not use it as a monitoring data source | Resource usage | Object state |
Three consequences:
kubectl topworks without Prometheus but fails with “Metrics API not available” when metrics-server is missing. Installing Prometheus does not fix that; installing metrics-server does.- The HPA does not read Prometheus directly. It reads
metrics.k8s.iofor CPU and memory. To scale on a Prometheus metric (queue depth, requests per second) you need an adapter that implements the custom metrics API (custom.metrics.k8s.io) or external metrics API (external.metrics.k8s.io), such as the Prometheus adapter, or an event-driven autoscaler such as KEDA, which our autoscaling guide covers. - Grafana dashboards of Pod restarts or Deployment availability need kube-state-metrics, because neither the kubelet nor metrics-server exposes object state.
If a question lists all three and asks which one Prometheus scrapes for “number of Pods in CrashLoopBackOff”, the answer is kube-state-metrics (kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}), and the Pod status guide explains why that reason lives in a container state rather than a Pod phase.
PromQL at KCNA Depth
PromQL is not on the exam as a programming test, but a handful of constructs appear inside answer options, and recognising them is enough.
Selectors. http_requests_total{job="api", status=~"5.."} selects series by exact match (=), negative match (!=), regular expression (=~) and negative regex (!~). Regex matchers are fully anchored.
Instant vs range vectors. http_requests_total is an instant vector: one sample per matching series at the evaluation time. http_requests_total[5m] is a range vector: the last five minutes of samples per series. Functions such as rate() take a range vector and return an instant vector. A range vector cannot be graphed directly; the Prometheus UI will tell you so.
Counter functions.
rate(http_requests_total[5m]) # per-second average rate over 5 min, handles resets
irate(http_requests_total[5m]) # instantaneous rate from the last two samples
increase(http_requests_total[1h]) # total increase over the window (rate * seconds)
Aggregation.
sum by (status) (rate(http_requests_total[5m])) # requests/s per status code
sum without (instance) (rate(http_requests_total[5m])) # drop the instance dimension
topk(3, rate(container_cpu_usage_seconds_total[5m])) # the three hungriest containers
avg_over_time(node_memory_MemAvailable_bytes[1h]) # gauge over time
Histogram quantiles.
histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))
The sum by (le) is mandatory for classic histograms: you aggregate the buckets across instances first, keeping only le, and then compute the quantile. Dropping le is the standard wrong answer.
Useful one-liners.
up == 0 # scrape failures
absent(up{job="api"}) # the job has no series at all
kube_deployment_status_replicas_available < kube_deployment_spec_replicas
sum(kube_pod_container_status_restarts_total) by (namespace, pod) > 5
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
A recording rule precomputes an expensive expression into a new series on a schedule, using the colon-separated level:metric:operations convention (job:http_requests:rate5m). An alerting rule evaluates an expression and fires when it has been true for the for duration:
groups:
- name: api
rules:
- record: job:http_requests:rate5m
expr: sum by (job) (rate(http_requests_total[5m]))
- alert: HighErrorRate
expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/ sum by (job) (rate(http_requests_total[5m])) > 0.05
for: 10m
labels:
severity: page
annotations:
summary: "{{ $labels.job }} error rate above 5% for 10 minutes"
An alert moves through three states: inactive, pending (expression true, for not yet elapsed) and firing. Prometheus evaluates rules every evaluation_interval (default 1 minute) and sends firing alerts to Alertmanager; Prometheus itself sends no pages.
Alertmanager: Grouping, Inhibition, Silences
Alertmanager (currently v0.35) receives alerts from one or more Prometheus servers and turns a storm of individual alerts into a manageable set of notifications:
| Concept | What it does | Exam cue |
|---|---|---|
| Deduplication | Identical alerts from HA Prometheus replicas become one notification | ”two Prometheus servers, one page” |
| Grouping | Alerts sharing labels (for example cluster, alertname) are batched into a single notification | ”100 Pods down, one message” |
| Routing | A tree of routes matches labels to receivers (email, chat, webhook, paging services) | “severity=page goes to on-call” |
| Inhibition | Suppress alerts when a related, more important alert is already firing | ”do not page about every Pod when the whole node is down” |
| Silences | Mute alerts matching a selector for a time window | ”maintenance window” |
Alertmanager is the component that knows about receivers; Prometheus knows only Alertmanager’s address. A question asking “where do you configure which team gets paged” is an Alertmanager question. A question asking “where do you define the condition that fires” is a Prometheus alerting-rule question.
Storage, Retention and Scaling Out
Each Prometheus server writes to a local time-series database (TSDB): incoming samples go to a write-ahead log and an in-memory head block, then get persisted into immutable two-hour blocks that are later compacted. The default retention is 15 days (--storage.tsdb.retention.time), with an optional size limit. The single-node, local-disk design is deliberate: a Prometheus server has no external dependencies and keeps working when the rest of the platform is on fire.
That design also sets its limits, and the exam phrases them as scenarios:
| Need | Mechanism |
|---|---|
| Keep metrics for a year | remote_write to long-term storage (Thanos, Cortex and other projects, or a vendor’s receiver) |
| Global view across many clusters | Thanos or Cortex query layers over per-cluster Prometheus servers; or federation, where one Prometheus scrapes aggregated series from others |
| High availability of alerting | Run two identical Prometheus servers; Alertmanager deduplicates |
| Shrink the footprint on edge clusters | Agent mode (--agent): scrape and remote-write only, no local querying or rules |
| Scale a single server | Shard scrape jobs across servers by hashing targets, then aggregate above |
Prometheus does not downsample, cluster, or replicate on its own; anything that sounds like “distributed, horizontally scalable, multi-year metrics” is describing a system built around Prometheus, not Prometheus itself.
Prometheus, OpenMetrics and OpenTelemetry
Three names, three roles, often confused:
- Prometheus is a monitoring system: scraper, database, query engine, rule evaluator.
- OpenMetrics is the specification of the exposition format that Prometheus and its exporters speak. It is a format, not a tool.
- OpenTelemetry (OTel) is the CNCF project for instrumentation and collection of traces, metrics and logs, with a vendor-neutral protocol (OTLP) and a Collector that receives, processes and exports telemetry. OTel does not store anything.
How they meet: the OpenTelemetry Collector can scrape Prometheus endpoints (a Prometheus receiver) and can export to Prometheus (a Prometheus exporter or remote write); Prometheus can ingest OTLP directly via its OTLP receiver and can translate OTel histograms into native histograms. A question asking for “the CNCF standard for instrumenting applications across traces, metrics and logs” wants OpenTelemetry; “the CNCF metrics storage and alerting system” wants Prometheus; “the text format on /metrics” wants OpenMetrics or the Prometheus exposition format. The observability guide covers the trace side (Jaeger) and the log side (Fluentd, Fluent Bit, Loki); Prometheus is never the answer for either.
A 30-Minute Lab on kind
Nothing makes the boundaries stick like watching the components. The following assumes kind (or minikube) and Helm. The free KCNA practice environment guide covers the setup.
-
Create a cluster and install the stack.
kind create cluster --name obs helm repo add prometheus-community https://prometheus-community.github.io/helm-charts helm install mon prometheus-community/kube-prometheus-stack -n monitoring --create-namespace kubectl get pods -n monitoringNote what appears: a Prometheus StatefulSet, an Alertmanager StatefulSet, a Grafana Deployment, a
kube-state-metricsDeployment, anode-exporterDaemonSet and the operator. -
See what a target looks like. Port-forward Prometheus and open the Targets page.
kubectl -n monitoring port-forward svc/mon-kube-prometheus-stack-prometheus 9090 # open http://localhost:9090/targetsEach target shows its
job,instance, the discovered__meta_kubernetes_*labels (click “show more”) and the last scrape duration. -
Read raw exposition. Pick a kube-state-metrics Pod and curl its endpoint from inside the cluster.
kubectl -n monitoring run curl --rm -it --image=curlimages/curl --restart=Never -- \ curl -s http://mon-kube-state-metrics.monitoring.svc:8080/metrics | grep -A2 "^# TYPE kube_pod_container_status_restarts_total"You will see
# HELP,# TYPE ... counterand one line per container. -
Confirm what
kubectl topdepends on. On a fresh kind clusterkubectl top nodesfails with “Metrics API not available” even though Prometheus is running, because kube-prometheus-stack does not install metrics-server. Install it and the command works:helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/ helm install metrics-server metrics-server/metrics-server -n kube-system --set args="{--kubelet-insecure-tls}" kubectl top nodes kubectl get apiservice v1beta1.metrics.k8s.ioThat last command shows the aggregated API that metrics-server registers.
-
Run the four canonical queries in the expression browser and note the shape of each result:
rate(container_cpu_usage_seconds_total{container!=""}[5m]) # counter -> gauge-like rate kube_deployment_status_replicas_available # object state, from kube-state-metrics histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m]))) up == 0 -
Break a scrape. Scale
kube-state-metricsto zero and watchup{job="kube-state-metrics"}drop to 0 and theKubeStateMetricsDown-style alert go from inactive to pending to firing on the Alerts page.kubectl -n monitoring scale deploy mon-kube-state-metrics --replicas=0 -
Follow the alert into Alertmanager. Port-forward
svc/mon-kube-prometheus-stack-alertmanageron 9093 and look at the grouped alert, then create a silence for it and watch it disappear from the active list. -
Clean up.
kind delete cluster --name obs.
Twenty minutes of this and the kube-state-metrics vs metrics-server question will never catch you again.
Exam Cue to Answer
| The question says | The answer is |
|---|---|
| ”Only increases, resets on restart” | Counter |
| ”Can go up and down”, “current value” | Gauge |
”le label”, “buckets”, “quantile computed at query time” | Histogram (histogram_quantile) |
“quantile label”, “computed in the client”, “cannot be aggregated” | Summary |
| ”Per-second rate of a counter” | rate() (or irate() for the last two samples) |
| “Scrapes HTTP endpoints on a schedule” | Pull model, scrape_interval |
| ”Short-lived batch job” | Pushgateway |
| ”Translate a database or host into Prometheus metrics” | Exporter (node_exporter for hosts) |
“kubectl top”, “HPA CPU utilisation”, “Metrics API” | metrics-server |
| ”Deployment replicas available”, “Pod restarts”, “object state” | kube-state-metrics |
| ”Container CPU and memory usage” | cAdvisor via the kubelet |
| ”Route alerts, group, silence, inhibit” | Alertmanager |
”Define the alert condition and for duration” | Prometheus alerting rule |
| ”Precompute an expensive query” | Recording rule (colon-separated name) |
| “Dashboards” | Grafana |
| ”Long-term, multi-cluster storage” | remote_write to Thanos, Cortex or another remote store |
| ”Discover Pods and Services automatically” | kubernetes_sd_configs (roles: node, pod, service, endpoints, endpointslice, ingress) |
“ServiceMonitor, PrometheusRule” | Prometheus Operator CRDs |
| ”Vendor-neutral instrumentation for traces, metrics and logs” | OpenTelemetry |
”Text format on /metrics”, “# TYPE” | Prometheus exposition format / OpenMetrics |
| ”100% accurate per-request billing” | Not Prometheus |
Scenario Drill
| # | Scenario | Answer | Why |
|---|---|---|---|
| 1 | An application exposes queue_messages_in_flight. Which type is it? | Gauge | In-flight counts rise and fall |
| 2 | Which expression gives HTTP requests per second from http_requests_total? | rate(http_requests_total[5m]) | Counters need a rate function |
| 3 | A team applies rate() to node_memory_MemAvailable_bytes and gets nonsense. Why? | It is a gauge; use the value directly or delta() | rate() assumes monotonic counters |
| 4 | Which series does a classic histogram rpc_seconds produce? | rpc_seconds_bucket{le=...}, rpc_seconds_sum, rpc_seconds_count | Bucket, sum, count triple |
| 5 | Why can a 95th percentile from a summary not be averaged across ten Pods? | Quantiles are computed per client and are not aggregatable | Use a histogram instead |
| 6 | kubectl top pods returns “Metrics API not available” although Prometheus is running. | Install metrics-server | kubectl top reads metrics.k8s.io, not Prometheus |
| 7 | Alert when a Deployment’s available replicas are below desired. | kube-state-metrics (kube_deployment_status_replicas_available) | Object state, not usage |
| 8 | Find which container on a node uses the most CPU. | cAdvisor metrics via the kubelet (container_cpu_usage_seconds_total) | Resource usage |
| 9 | A nightly job runs 20 seconds; its metrics never appear. | Push to the Pushgateway; Prometheus scrapes the gateway | Job exits between scrapes |
| 10 | Which label tells you a scrape failed? | up == 0 for that job/instance | Synthesised per scrape |
| 11 | Which component sends the page to the on-call engineer? | Alertmanager | Prometheus only evaluates rules |
| 12 | During a node outage, 40 Pod alerts fire alongside one node alert. Which feature hides the 40? | Inhibition | Suppress dependent alerts |
| 13 | Suppress alerts for a planned upgrade window. | Silence in Alertmanager | Time-boxed mute |
| 14 | Metrics must be kept for 13 months across 30 clusters. | remote_write to a long-term store (Thanos, Cortex or similar) | Local TSDB defaults to 15 days, single node |
| 15 | Prometheus should discover every Pod with a metrics port automatically. | kubernetes_sd_configs with role: pod plus relabeling | Service discovery |
| 16 | Declare scrape targets as Kubernetes objects. | ServiceMonitor/PodMonitor via the Prometheus Operator | Operator pattern |
| 17 | The HPA must scale on requests per second from Prometheus. | Custom metrics API adapter (or KEDA), not metrics-server | HPA reads Kubernetes metrics APIs |
| 18 | A new exporter is rejected by Prometheus 3.x with an unsupported content type error. | Return a valid Content-Type (for example text/plain; version=0.0.4) | Mandatory since 3.0 |
| 19 | Which metric name follows conventions: request_latency_ms or http_request_duration_seconds? | http_request_duration_seconds | Prefix, base unit, unit suffix |
| 20 | A series name contains a colon: job:http_requests:rate5m. Where did it come from? | A recording rule | Colons are reserved for rules |
| 21 | Which CNCF project standardises instrumentation for traces, metrics and logs? | OpenTelemetry | Prometheus is metrics only |
| 22 | Which component stores Pod and Node CPU in memory for 15-second autoscaling decisions and nothing else? | metrics-server | Latest values only, autoscaling only |
Mistakes That Cost Marks
- Treating metrics-server as a monitoring system. It is an autoscaling data source with no history. The project’s own README tells you not to use it for monitoring.
- Thinking Prometheus draws dashboards or stores logs. It does neither; Grafana draws, log pipelines store logs.
- Applying
rate()to a gauge orsum()to a raw counter. Match the function to the type. - Forgetting
by (le)inhistogram_quantile. Without it the quantile is computed over an unaggregated mix and the result is wrong or empty. - Believing the
prometheus.io/scrapeannotation is magic. It is honoured by relabel rules (or by aPodMonitor), not by Kubernetes. - Reaching for the Pushgateway for ordinary services. It is for batch jobs; services get scraped.
- Assuming Prometheus clusters or replicates. It does not; HA is two independent servers plus Alertmanager deduplication, and scale-out is remote write or federation.
- Confusing OpenMetrics (a format) with OpenTelemetry (a project).
Practice the Boundaries Until They Are Reflexes
The KCNA rewards candidates who can place a tool at the correct layer in under a minute. Build the reflex with the study guide, the exam format breakdown and the free practice questions, then run the free KCNA sample exam to see how observability questions are phrased. When you want timed, full-length practice, Sailor.sh’s KCNA Certification-Ready Mock Exam Bundle spreads five mock exams across all four domains, with observability items that mix metric types, PromQL fragments and the metrics-server vs kube-state-metrics distinction in the same way the real exam does, each with an explanation of the wrong options. If the Kubestronaut path is your longer goal, the same Prometheus knowledge reappears at CKA depth when you have to debug a cluster with it.
Frequently Asked Questions
Does the KCNA require me to write PromQL?
No. The exam is multiple choice and may show a short PromQL expression inside an answer option, typically rate() on a counter or histogram_quantile() on buckets. Recognising which function belongs to which metric type is enough.
What is the difference between a histogram and a summary in one sentence?
A histogram counts observations into cumulative le buckets and lets the server compute any quantile later and across instances; a summary ships pre-computed quantiles from the client that cannot be aggregated or changed afterwards.
Does Prometheus push or pull?
Pull. It scrapes HTTP endpoints on a schedule. The Pushgateway exists for short-lived batch jobs, and remote write and the OTLP receiver are integration paths, but the primary collection model is pull.
Can the Horizontal Pod Autoscaler use Prometheus metrics?
Not directly. The HPA reads the Kubernetes Metrics API (served by metrics-server) for CPU and memory, and the custom or external metrics APIs for anything else. An adapter exposes Prometheus queries through those APIs; KEDA is the common event-driven alternative.
Is kube-state-metrics part of Kubernetes?
It is a Kubernetes project maintained under the Kubernetes organisation, but it is not installed by default. It runs as a Deployment, watches the API server, and exposes object state as Prometheus metrics.
Why did my old exporter break after upgrading to Prometheus 3?
Since 3.0 a scrape target must return a valid Content-Type header. Set text/plain; version=0.0.4 (or the OpenMetrics content type) in the exporter, or configure fallback_scrape_protocol on the scrape job.
What are native histograms and do I need them for the exam?
They are a more efficient histogram representation that stores count, sum and dynamically sized buckets in a single sample. They became a stable, opt-in feature in Prometheus 3.8. For the KCNA, knowing they exist and why they are more efficient is plenty; classic _bucket histograms are still what you will see on most endpoints.
How long does Prometheus keep data?
15 days by default, on local disk. Longer retention is a configuration change on a single server, or remote write to long-term storage for a fleet.
Conclusion
Prometheus is a small set of ideas applied consistently: a time series is a name plus labels; four metric types tell you which functions apply; targets expose a text format on /metrics; the server pulls, stamps up, job and instance, and stores locally; rules turn series into derived series and alerts; Alertmanager turns alerts into notifications. On Kubernetes the only extra work is knowing who produces which number: cAdvisor for container usage, kube-state-metrics for object state, metrics-server for autoscaling and kubectl top, and the Prometheus Operator’s CRDs to wire it all together. Learn those boundaries and every Prometheus question on the KCNA becomes a matter of reading the cue.