Back to Blog

Prometheus for the KCNA Exam, Explained Properly: The Data Model, Counter vs Gauge vs Histogram vs Summary, the /metrics Exposition Format, Pull-Based Scraping & Service Discovery, PromQL Basics, Recording & Alerting Rules, Alertmanager, and kube-state-metrics vs metrics-server vs cAdvisor

A practitioner's KCNA guide to the one observability tool the curriculum names explicitly — what a Prometheus time series actually is (metric name, labels, float64 or native-histogram samples, UTF-8 names since v3.0), the four metric types and the exam traps around counters and histograms, the text exposition format and why Content-Type is now mandatory, how pull-based scraping, the up metric, job/instance labels and Kubernetes service discovery work, when the Pushgateway is and is not appropriate, every Kubernetes metrics source (kubelet, cAdvisor, kube-apiserver, kube-state-metrics, node_exporter), the kube-state-metrics vs metrics-server distinction the exam loves, PromQL selectors, rate(), sum by() and histogram_quantile(), recording and alerting rules, Alertmanager grouping and silences, remote write and long-term storage, how Prometheus relates to OpenTelemetry and OpenMetrics, a kind-based lab, a 22-row scenario drill and an FAQ.

By Sailor Team , November 12, 2026

Every KCNA candidate learns the sentence “Prometheus is the CNCF metrics system that scrapes /metrics endpoints”. The exam then asks questions that sentence does not answer: which metric type a request-duration histogram actually produces, why kubectl top works without Prometheus but a Grafana dashboard does not, what kube-state-metrics exposes that the kubelet does not, which PromQL function you apply to a counter, and why a batch job needs a Pushgateway when everything else is pulled. Those are not trivia. They are the difference between knowing the name of a tool and knowing how it fits into a cluster.

This guide is the deep dive the KCNA curriculum deserves for the one observability tool it names explicitly. Our cloud native architecture guide introduces the three pillars and gives Prometheus a dozen bullet points; this article is the rest of the story. Everything below follows the current Prometheus documentation (Prometheus 3.16, released 4 November 2026), the kube-state-metrics and Metrics Server project documentation, and the Kubernetes v1.37 documentation.

Where This Sits in the KCNA Curriculum

The KCNA curriculum was refreshed in late 2025 and now has four domains: Kubernetes Fundamentals (44%), Container Orchestration (28%), Cloud Native Application Delivery (16%) and Cloud Native Architecture (12%). Observability is a named competency inside Cloud Native Architecture, and Prometheus is the only monitoring project the curriculum calls out by name. In practice the questions spill into three domains:

DomainHow Prometheus shows up
Cloud Native Architecture (Observability)“Which metric type only increases?”, “Which component handles alert routing?”, “Which project is the CNCF standard for metrics?”
Container Orchestration (Troubleshooting)“Which component must be installed for kubectl top to work?”, “Which exporter reports the number of Pod restarts?”
Kubernetes Fundamentals (Administration)“Which Kubernetes API does the Horizontal Pod Autoscaler read CPU usage from?”, “Where does the kubelet expose container metrics?”

The exam is multiple choice, so you will never write a scrape config under time pressure. You will be asked to pick the right component, the right metric type, the right function and the right boundary between tools that look similar. That is what this guide drills.

What Prometheus Is (and the Facts the Exam Checks)

Prometheus is an open-source systems monitoring and alerting toolkit. It was created at SoundCloud in 2012, joined the Cloud Native Computing Foundation in 2016 as the foundation’s second hosted project (after Kubernetes), and graduated in 2018. It is written in Go and ships as static binaries. The project’s own description of its main features reads like an exam cheat sheet:

  • a multi-dimensional data model with time series identified by a metric name and key/value pairs;
  • PromQL, a query language built to exploit that dimensionality;
  • no reliance on distributed storage – a single Prometheus server is autonomous;
  • collection happens via a pull model over HTTP;
  • pushing is supported only via an intermediary gateway;
  • targets are found through service discovery or static configuration;
  • multiple modes of graphing and dashboarding support.

The ecosystem has several components, most of them optional:

ComponentRoleExam cue
Prometheus serverScrapes targets, stores time series in its local TSDB, evaluates rules, answers PromQL”scrapes and stores”
Client librariesInstrument application code (Go, Java, Python, Ruby, .NET, Rust and more) and serve /metrics”expose custom application metrics”
ExportersSidecar or standalone processes that translate third-party systems (Linux kernel, databases, message brokers) into the Prometheus format”node_exporter”, “monitor something that cannot be instrumented”
PushgatewayAccepts pushed metrics from short-lived jobs so Prometheus can scrape them later”batch job that exits before the next scrape”
AlertmanagerReceives alerts from Prometheus servers, deduplicates, groups, routes, silences and inhibits them, and sends notifications”route alerts to the on-call channel”
Grafana (separate project)Dashboards on top of Prometheus and other data sources”visualise”

Two boundaries matter for the exam. First, Prometheus handles metrics only. It does not store logs (that is Fluentd, Fluent Bit or Loki territory) or distributed traces (Jaeger, OpenTelemetry). Second, Prometheus ships only a basic expression browser; Grafana draws the dashboards. If a question asks for “the CNCF tool to visualise Prometheus data”, the answer is Grafana, not Prometheus.

The documentation also states when Prometheus does not fit: if you need 100% accuracy, such as per-request billing, Prometheus is the wrong tool, because scraped data is sampled and may be incomplete. The design goal is reliability during an outage, not audit-grade completeness. A question that pairs “billing” with “Prometheus” is testing whether you know this.

The Data Model: What a Time Series Actually Is

Prometheus stores everything as time series: streams of timestamped values that belong to the same metric and the same set of labels. Each series is identified by:

  • a metric name, which describes the feature being measured (http_requests_total);
  • zero or more labels, key/value pairs that capture the dimensions (method="POST", handler="/api/tracks").

Each sample in a series is a float64 value (or, for native histograms, a composite histogram value) plus a millisecond-precision timestamp. The notation you will see everywhere is:

<metric name>{<label name>="<label value>", ...}
api_http_requests_total{method="POST", handler="/messages"}

Internally the metric name is itself a label with the reserved name __name__, so the series above is equivalent to {__name__="api_http_requests_total", method="POST", handler="/messages"}. Label names beginning with two underscores are reserved for internal use, which is why the service-discovery metadata you relabel from is called __meta_kubernetes_pod_name and not pod_name.

Rules worth memorising:

  • Metric and label names may use any UTF-8 character since Prometheus 3.0 (November 2024), but should stick to [a-zA-Z_:][a-zA-Z0-9_:]* for compatibility; names outside that set must be quoted in PromQL ({"http.requests.total", method="GET"}).
  • Colons are reserved for recording rules. Exporters and instrumented code should never emit job:http_requests:rate5m-style names; that shape tells you a human wrote a rule.
  • A label with an empty value is equivalent to the label not existing.
  • Changing any label value creates a new series. This is what people mean by cardinality: a label such as user_id or request_id creates one series per distinct value and can take down a server. Labels should carry dimensions with a bounded set of values.

Naming conventions the exam borrows

The naming best-practice page is where many question stems come from:

RuleExample
Single-word application or domain prefixprometheus_notifications_total, process_cpu_seconds_total, http_request_duration_seconds
One unit and one quantity per metricnever mix seconds and milliseconds, never mix size and duration
Base units, not scaled onesseconds not milliseconds, bytes not megabytes, meters not kilometers
Unit suffix in plural_seconds, _bytes
Accumulating counts end in _totalhttp_requests_total, process_cpu_seconds_total
Pseudo-metrics that carry metadata end in _infokube_pod_info, node_uname_info (value is always 1, the labels hold the data)
Unix timestamps end in _timestamp_secondsdata_pipeline_last_record_processed_timestamp_seconds

If a question shows container_memory_working_set_bytes and asks what it measures, the suffix tells you it is bytes, and the absence of _total tells you it is a gauge-like reading rather than an accumulating count.

The Four Metric Types, Properly

The client libraries offer four core metric types. A subtle but exam-relevant point from the documentation: with the exception of native histograms, the Prometheus server does not yet make use of the type information and flattens everything into untyped float series. The types exist in the instrumentation libraries and the exposition format, and they tell you which PromQL functions are appropriate.

Counter

A counter is a cumulative value that can only increase, or reset to zero when the process restarts. Requests served, tasks completed, errors seen, CPU seconds consumed: all counters. The documentation’s rule is explicit: do not use a counter for anything that can decrease, such as the number of currently running processes; that is a gauge.

Because counters only go up, their raw value is rarely interesting; the rate of change is. That is why rate(), irate() and increase() exist and why they are designed to handle counter resets. The trap the exam sets is a question that shows http_requests_total and asks for the function that gives requests per second; the answer is rate(http_requests_total[5m]), never sum(http_requests_total) on its own.

Gauge

A gauge is a single numerical value that can go up and down: temperature, current memory usage, number of concurrent requests, queue depth, number of running Pods. You read gauges directly, aggregate them with sum, avg, max, and use delta() or deriv() rather than rate() when you care about change. Applying rate() to a gauge is a classic distractor.

Histogram

A histogram records observations (usually request durations or response sizes) by counting them into configurable buckets, and it also keeps the sum of all observed values. The documentation’s one-line summary is that a histogram is “essentially a bucketed counter”.

Histograms come in two forms. A classic histogram with base name http_request_duration_seconds produces several time series:

http_request_duration_seconds_bucket{le="0.1"}   24054   # cumulative: observations <= 0.1 s
http_request_duration_seconds_bucket{le="0.5"}   129389
http_request_duration_seconds_bucket{le="1"}     133988
http_request_duration_seconds_bucket{le="+Inf"}  144320  # equals _count
http_request_duration_seconds_sum               53423
http_request_duration_seconds_count             144320

Three details are testable: the le label is the upper inclusive bound; the buckets are cumulative (the le="0.5" bucket includes everything in le="0.1"); and _bucket{le="+Inf"} always equals _count. Quantiles are calculated on the server, at query time, with histogram_quantile(), which means you can aggregate histograms across instances first and ask for the 95th percentile of the whole fleet.

A native histogram is exposed and ingested as a single composite sample that carries the count, the sum and a dynamic set of exponentially spaced buckets. Native histograms need no bucket configuration at instrumentation time, use far fewer series, are always aggregatable with each other even when resolutions differ, and travel atomically over remote write. They were experimental for several releases and became a stable, opt-in feature in Prometheus 3.8 (November 2025), enabled with the scrape_native_histograms setting. For the exam, “native histogram” means “the newer, more efficient histogram representation”; the classic _bucket/_sum/_count triple is still what most exporters emit.

Summary

A summary also samples observations and also exposes _sum and _count, but instead of buckets it exposes pre-calculated φ-quantiles over a sliding window, as <basename>{quantile="0.99"}. The quantiles are computed in the client, which has two consequences the exam likes: the calculation is comparatively expensive in the instrumented process, and you cannot aggregate quantiles across instances (the average of two 99th percentiles is not the 99th percentile of the combined population). You also cannot ask for a quantile you did not configure up front.

Side by side

CounterGaugeClassic histogramNative histogramSummary
DirectionOnly up (resets to 0 on restart)Up and downBuckets are countersComposite sampleCount/sum are counters
Series produced11_bucket per bucket + _sum + _count1one per quantile + _sum + _count
Where quantiles are computedn/an/aServer (histogram_quantile)ServerClient, at instrumentation time
Aggregatable across instancesYesYesYes, if bucket bounds matchYes, alwaysCount and sum yes, quantiles no
Typical functionrate(), increase()direct, avg_over_time()histogram_quantile(0.95, sum by (le) (rate(..._bucket[5m])))histogram_quantile(0.95, sum(rate(...[5m])))read {quantile="0.95"} directly
Exam cue”only increases”, “requests served""current”, “can decrease""le label”, “buckets""no bucket config”, “fewer series""quantile label”, “cannot aggregate”

Two more conventions: the output of rate() on a counter is gauge-like (a per-second value that can fall), and the output of rate() on a counter histogram is a gauge histogram. If a question says “a histogram that represents the current state of a distribution rather than an accumulation”, it is describing a gauge histogram.

The Exposition Format: What /metrics Actually Returns

Instrumented processes and exporters serve metrics on an HTTP endpoint, by convention /metrics, in a line-oriented text format. A minimal response looks like this:

# HELP http_requests_total The total number of HTTP requests.
# TYPE http_requests_total counter
http_requests_total{method="post",code="200"} 1027 1395066363000
http_requests_total{method="post",code="400"}    3 1395066363000

# HELP http_request_duration_seconds A histogram of the request duration.
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.05"} 24054
http_request_duration_seconds_bucket{le="0.1"} 33444
http_request_duration_seconds_bucket{le="+Inf"} 144320
http_request_duration_seconds_sum 53423
http_request_duration_seconds_count 144320

What to know:

  • # HELP carries the description and # TYPE the metric type; both are optional comments, but they are how the type information reaches Prometheus at all.
  • Each sample line is name{labels} value [timestamp]; the timestamp is optional and almost always omitted, because Prometheus stamps samples with the scrape time.
  • The classic Prometheus text format is served with Content-Type: text/plain; version=0.0.4.
  • OpenMetrics is the standardised evolution of that format, served as application/openmetrics-text; version=1.0.0. It adds exemplars (references from a histogram bucket to a trace ID), explicit # EOF termination and stricter typing. The OpenMetrics specification started as its own CNCF project and is now maintained inside the Prometheus project; OpenMetrics 2.0 exists but is experimental behind the openmetrics2 feature flag.
  • There is also a protobuf exposition format, currently needed to expose native histograms.
  • Since Prometheus 3.0, scrape targets must return a valid Content-Type header. A missing or unsupported content type makes the scrape fail unless a fallback_scrape_protocol is configured. This broke a surprising number of hand-written exporters during 3.0 upgrades and is exactly the kind of “what changed in the major version” fact an exam writer enjoys.

The exam does not expect you to write exporters, but it does expect you to recognise that an application “exposing a /metrics endpoint in the Prometheus text format” is instrumented and can be scraped directly, whereas a database or a Linux host needs an exporter.

Pull, Not Push: How Scraping Works

Prometheus pulls. On a schedule (the global scrape_interval, 1 minute by default; scrape_timeout defaults to 10 seconds) it issues an HTTP GET to every target’s metrics path and stores whatever comes back. Each scrape adds two things on top of the target’s own metrics:

  • the up metric: up{job="api", instance="10.244.1.7:8080"} is 1 when the scrape succeeded and 0 when it failed. up == 0 is the first alert every team writes.
  • the job and instance labels. job is the scrape job name from the configuration; instance is the host:port of the target. If the scraped metrics already carry a label with the same name, Prometheus keeps its own and renames the target’s to exported_job, unless honor_labels: true is set. Other scrape-time series include scrape_duration_seconds and scrape_samples_scraped.

Why pull? The documentation’s reasoning, in exam form: Prometheus can tell the difference between “the target is down” and “the target has nothing to say” (that is what up encodes); you can run a second Prometheus against the same targets for redundancy or testing without touching the applications; and you can hit a /metrics endpoint with a browser or curl to debug instrumentation. The cost is that Prometheus must be able to reach the target, which is why service discovery exists.

The Pushgateway and when it is wrong

Some workloads cannot be scraped because they do not live long enough: a cron job that runs for four seconds will usually exit between two scrapes. The Pushgateway solves exactly this: the job pushes its final metrics to the gateway, and Prometheus scrapes the gateway. The documentation is blunt about its limits, and so is the exam:

  • Use it only for service-level batch jobs, not as a general push path for applications.
  • Metrics pushed to a gateway persist until deleted, so a job that stops running will keep reporting its last values; Prometheus cannot tell it is gone.
  • The gateway becomes a single point of failure and a bottleneck, and the up metric now describes the gateway, not the job.

So the answer to “a nightly ETL job finishes before Prometheus can scrape it; what do you use?” is the Pushgateway. The answer to “an application team wants to push metrics instead of being scraped” is to instrument the application and let Prometheus pull, or to send metrics via the OpenTelemetry Collector to Prometheus’ OTLP receiver (see below).

Push paths that do exist

Two more routes carry data into a Prometheus server and are worth knowing so you do not over-apply “Prometheus never accepts pushes”:

  • Remote write: another Prometheus (or an agent) streams samples to a receiver that has --web.enable-remote-write-receiver enabled. This is how fleets ship data to long-term storage.
  • OTLP receiver: Prometheus can accept OpenTelemetry metrics over OTLP at /api/v1/otlp/v1/metrics when the otlp configuration block is enabled.

Neither replaces scraping as the primary collection model; both are integration paths.

Service Discovery in Kubernetes

Static target lists die the moment a Deployment scales. Prometheus therefore discovers targets dynamically, and the Kubernetes discovery mechanism (kubernetes_sd_configs) is the one the exam cares about. It talks to the API server (using the Pod’s own ServiceAccount token and CA certificate when Prometheus runs in-cluster) and watches one of six roles:

RoleDiscoversTypical use
nodeOne target per Node (kubelet address)Scrape kubelet, cAdvisor, node-level metrics
podOne target per Pod per declared container portScrape application Pods directly
serviceOne target per Service portBlackbox-style checks of a Service’s DNS name
endpointsOne target per Endpoints address (legacy)Scrape every backend behind a Service
endpointsliceOne target per EndpointSlice endpoint (preferred)Same as endpoints, scales better
ingressOne target per Ingress pathBlackbox checks of external entry points

Discovery produces targets decorated with __meta_kubernetes_* labels (namespace, Pod name, labels, annotations, node name). Relabeling (relabel_configs) then decides which targets to keep, what address and path to scrape, and which metadata to promote into real labels. The widely used prometheus.io/scrape: "true", prometheus.io/port and prometheus.io/path Pod annotations are a convention implemented by relabel rules, not something Kubernetes or Prometheus understands natively. If a question implies the annotation itself makes Prometheus scrape a Pod, the missing piece is the relabel rule that honours it.

The Prometheus Operator and its CRDs

Most clusters do not hand-write scrape configs. The Prometheus Operator (part of the kube-prometheus project) turns configuration into Kubernetes objects, which is a textbook example of the operator pattern and custom resources the KCNA also tests:

CRDWhat it declares
Prometheus / AlertmanagerThe server instances themselves (replicas, retention, storage)
ServiceMonitor”Scrape every Service matching this selector on this port”
PodMonitorSame for Pods without a Service
ScrapeConfigA raw scrape job as a CRD
PrometheusRuleRecording and alerting rules
AlertmanagerConfigRouting and receivers for a namespace

The operator watches these objects and regenerates the Prometheus configuration. The kube-prometheus-stack Helm chart bundles the operator, Prometheus, Alertmanager, Grafana, node_exporter, kube-state-metrics and a set of default dashboards and alerts, which is why it is the usual answer to “install a complete monitoring stack on Kubernetes”.

Where Kubernetes Metrics Come From

The exam’s favourite confusion is which component produces which number. The honest answer is that a Kubernetes cluster has at least six metric sources, and a complete monitoring setup scrapes all of them:

SourceEndpointWhat it reportsExample metric
kubelethttps://<node>:10250/metricsKubelet’s own operationskubelet_running_pods, kubelet_pod_start_duration_seconds
cAdvisor (embedded in the kubelet)/metrics/cadvisorPer-container resource usage from cgroupscontainer_cpu_usage_seconds_total, container_memory_working_set_bytes
kubelet resource endpoint/metrics/resourceA slimmed set of CPU/memory metrics per container and Podcontainer_cpu_usage_seconds_total, pod_memory_working_set_bytes
kube-apiserver/metrics on 6443API request rates, latencies, etcd call latencies, admission timingsapiserver_request_total, apiserver_request_duration_seconds
kube-scheduler, kube-controller-manager, etcd, CoreDNSeach component’s /metricsScheduling latency, queue depths, fsync latency, DNS request countsscheduler_pending_pods, etcd_disk_wal_fsync_duration_seconds, coredns_dns_requests_total
kube-state-metricsits own Service on 8080The desired and observed state of API objects, not resource usagekube_deployment_status_replicas_available, kube_pod_container_status_restarts_total, kube_node_status_condition
node_exporter (DaemonSet):9100/metricsHost-level Linux metricsnode_cpu_seconds_total, node_filesystem_avail_bytes
Your applicationits /metricsWhatever you instrumentedhttp_requests_total

Kubernetes system-component metrics follow a stability policy (ALPHA, BETA, STABLE, and deprecated metrics hidden behind --show-hidden-metrics-for-version), mirroring the API lifecycle the community and governance guide covers.

The pairing to memorise: cAdvisor tells you how much CPU a container is using; kube-state-metrics tells you how many replicas a Deployment has available. One is a resource measurement, the other is object state. A question that says “alert when a Deployment has fewer available replicas than desired” is a kube-state-metrics question. A question that says “find the container using the most memory” is a cAdvisor question.

kube-state-metrics vs metrics-server vs cAdvisor

This is the single most reliable Prometheus question on the KCNA, because three components with similar names do three different jobs.

metrics-serverkube-state-metricscAdvisor
What it isA Kubernetes add-on that implements the Metrics API (metrics.k8s.io)An exporter that turns API object state into Prometheus metricsA library embedded in the kubelet that reads cgroups
DataLatest CPU and memory usage per Pod and Node, in memory onlyReplica counts, Pod phases, restart counts, conditions, resource requests and limitsPer-container CPU, memory, network, filesystem usage
Who reads itHorizontal Pod Autoscaler, Vertical Pod Autoscaler, kubectl topPrometheus (and therefore Grafana, Alertmanager)Prometheus, via the kubelet
HistoryNone – latest value only, collected every 15 sNone itself; Prometheus stores the historyNone itself; Prometheus stores the history
ProtocolKubernetes aggregated API; needs the aggregation layer and kubelet webhook authn/authzPrometheus text formatPrometheus text format
What it is not forThe project README says plainly: autoscaling only, do not use it as a monitoring data sourceResource usageObject state

Three consequences:

  1. kubectl top works without Prometheus but fails with “Metrics API not available” when metrics-server is missing. Installing Prometheus does not fix that; installing metrics-server does.
  2. The HPA does not read Prometheus directly. It reads metrics.k8s.io for CPU and memory. To scale on a Prometheus metric (queue depth, requests per second) you need an adapter that implements the custom metrics API (custom.metrics.k8s.io) or external metrics API (external.metrics.k8s.io), such as the Prometheus adapter, or an event-driven autoscaler such as KEDA, which our autoscaling guide covers.
  3. Grafana dashboards of Pod restarts or Deployment availability need kube-state-metrics, because neither the kubelet nor metrics-server exposes object state.

If a question lists all three and asks which one Prometheus scrapes for “number of Pods in CrashLoopBackOff”, the answer is kube-state-metrics (kube_pod_container_status_waiting_reason{reason="CrashLoopBackOff"}), and the Pod status guide explains why that reason lives in a container state rather than a Pod phase.

PromQL at KCNA Depth

PromQL is not on the exam as a programming test, but a handful of constructs appear inside answer options, and recognising them is enough.

Selectors. http_requests_total{job="api", status=~"5.."} selects series by exact match (=), negative match (!=), regular expression (=~) and negative regex (!~). Regex matchers are fully anchored.

Instant vs range vectors. http_requests_total is an instant vector: one sample per matching series at the evaluation time. http_requests_total[5m] is a range vector: the last five minutes of samples per series. Functions such as rate() take a range vector and return an instant vector. A range vector cannot be graphed directly; the Prometheus UI will tell you so.

Counter functions.

rate(http_requests_total[5m])        # per-second average rate over 5 min, handles resets
irate(http_requests_total[5m])       # instantaneous rate from the last two samples
increase(http_requests_total[1h])    # total increase over the window (rate * seconds)

Aggregation.

sum by (status) (rate(http_requests_total[5m]))         # requests/s per status code
sum without (instance) (rate(http_requests_total[5m]))  # drop the instance dimension
topk(3, rate(container_cpu_usage_seconds_total[5m]))    # the three hungriest containers
avg_over_time(node_memory_MemAvailable_bytes[1h])       # gauge over time

Histogram quantiles.

histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])))

The sum by (le) is mandatory for classic histograms: you aggregate the buckets across instances first, keeping only le, and then compute the quantile. Dropping le is the standard wrong answer.

Useful one-liners.

up == 0                                                           # scrape failures
absent(up{job="api"})                                             # the job has no series at all
kube_deployment_status_replicas_available < kube_deployment_spec_replicas
sum(kube_pod_container_status_restarts_total) by (namespace, pod) > 5
(1 - node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100

A recording rule precomputes an expensive expression into a new series on a schedule, using the colon-separated level:metric:operations convention (job:http_requests:rate5m). An alerting rule evaluates an expression and fires when it has been true for the for duration:

groups:
- name: api
  rules:
  - record: job:http_requests:rate5m
    expr: sum by (job) (rate(http_requests_total[5m]))
  - alert: HighErrorRate
    expr: sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          / sum by (job) (rate(http_requests_total[5m])) > 0.05
    for: 10m
    labels:
      severity: page
    annotations:
      summary: "{{ $labels.job }} error rate above 5% for 10 minutes"

An alert moves through three states: inactive, pending (expression true, for not yet elapsed) and firing. Prometheus evaluates rules every evaluation_interval (default 1 minute) and sends firing alerts to Alertmanager; Prometheus itself sends no pages.

Alertmanager: Grouping, Inhibition, Silences

Alertmanager (currently v0.35) receives alerts from one or more Prometheus servers and turns a storm of individual alerts into a manageable set of notifications:

ConceptWhat it doesExam cue
DeduplicationIdentical alerts from HA Prometheus replicas become one notification”two Prometheus servers, one page”
GroupingAlerts sharing labels (for example cluster, alertname) are batched into a single notification”100 Pods down, one message”
RoutingA tree of routes matches labels to receivers (email, chat, webhook, paging services)“severity=page goes to on-call”
InhibitionSuppress alerts when a related, more important alert is already firing”do not page about every Pod when the whole node is down”
SilencesMute alerts matching a selector for a time window”maintenance window”

Alertmanager is the component that knows about receivers; Prometheus knows only Alertmanager’s address. A question asking “where do you configure which team gets paged” is an Alertmanager question. A question asking “where do you define the condition that fires” is a Prometheus alerting-rule question.

Storage, Retention and Scaling Out

Each Prometheus server writes to a local time-series database (TSDB): incoming samples go to a write-ahead log and an in-memory head block, then get persisted into immutable two-hour blocks that are later compacted. The default retention is 15 days (--storage.tsdb.retention.time), with an optional size limit. The single-node, local-disk design is deliberate: a Prometheus server has no external dependencies and keeps working when the rest of the platform is on fire.

That design also sets its limits, and the exam phrases them as scenarios:

NeedMechanism
Keep metrics for a yearremote_write to long-term storage (Thanos, Cortex and other projects, or a vendor’s receiver)
Global view across many clustersThanos or Cortex query layers over per-cluster Prometheus servers; or federation, where one Prometheus scrapes aggregated series from others
High availability of alertingRun two identical Prometheus servers; Alertmanager deduplicates
Shrink the footprint on edge clustersAgent mode (--agent): scrape and remote-write only, no local querying or rules
Scale a single serverShard scrape jobs across servers by hashing targets, then aggregate above

Prometheus does not downsample, cluster, or replicate on its own; anything that sounds like “distributed, horizontally scalable, multi-year metrics” is describing a system built around Prometheus, not Prometheus itself.

Prometheus, OpenMetrics and OpenTelemetry

Three names, three roles, often confused:

  • Prometheus is a monitoring system: scraper, database, query engine, rule evaluator.
  • OpenMetrics is the specification of the exposition format that Prometheus and its exporters speak. It is a format, not a tool.
  • OpenTelemetry (OTel) is the CNCF project for instrumentation and collection of traces, metrics and logs, with a vendor-neutral protocol (OTLP) and a Collector that receives, processes and exports telemetry. OTel does not store anything.

How they meet: the OpenTelemetry Collector can scrape Prometheus endpoints (a Prometheus receiver) and can export to Prometheus (a Prometheus exporter or remote write); Prometheus can ingest OTLP directly via its OTLP receiver and can translate OTel histograms into native histograms. A question asking for “the CNCF standard for instrumenting applications across traces, metrics and logs” wants OpenTelemetry; “the CNCF metrics storage and alerting system” wants Prometheus; “the text format on /metrics” wants OpenMetrics or the Prometheus exposition format. The observability guide covers the trace side (Jaeger) and the log side (Fluentd, Fluent Bit, Loki); Prometheus is never the answer for either.

A 30-Minute Lab on kind

Nothing makes the boundaries stick like watching the components. The following assumes kind (or minikube) and Helm. The free KCNA practice environment guide covers the setup.

  1. Create a cluster and install the stack.

    kind create cluster --name obs
    helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
    helm install mon prometheus-community/kube-prometheus-stack -n monitoring --create-namespace
    kubectl get pods -n monitoring

    Note what appears: a Prometheus StatefulSet, an Alertmanager StatefulSet, a Grafana Deployment, a kube-state-metrics Deployment, a node-exporter DaemonSet and the operator.

  2. See what a target looks like. Port-forward Prometheus and open the Targets page.

    kubectl -n monitoring port-forward svc/mon-kube-prometheus-stack-prometheus 9090
    # open http://localhost:9090/targets

    Each target shows its job, instance, the discovered __meta_kubernetes_* labels (click “show more”) and the last scrape duration.

  3. Read raw exposition. Pick a kube-state-metrics Pod and curl its endpoint from inside the cluster.

    kubectl -n monitoring run curl --rm -it --image=curlimages/curl --restart=Never -- \
      curl -s http://mon-kube-state-metrics.monitoring.svc:8080/metrics | grep -A2 "^# TYPE kube_pod_container_status_restarts_total"

    You will see # HELP, # TYPE ... counter and one line per container.

  4. Confirm what kubectl top depends on. On a fresh kind cluster kubectl top nodes fails with “Metrics API not available” even though Prometheus is running, because kube-prometheus-stack does not install metrics-server. Install it and the command works:

    helm repo add metrics-server https://kubernetes-sigs.github.io/metrics-server/
    helm install metrics-server metrics-server/metrics-server -n kube-system --set args="{--kubelet-insecure-tls}"
    kubectl top nodes
    kubectl get apiservice v1beta1.metrics.k8s.io

    That last command shows the aggregated API that metrics-server registers.

  5. Run the four canonical queries in the expression browser and note the shape of each result:

    rate(container_cpu_usage_seconds_total{container!=""}[5m])      # counter -> gauge-like rate
    kube_deployment_status_replicas_available                         # object state, from kube-state-metrics
    histogram_quantile(0.99, sum by (le) (rate(apiserver_request_duration_seconds_bucket{verb!="WATCH"}[5m])))
    up == 0
  6. Break a scrape. Scale kube-state-metrics to zero and watch up{job="kube-state-metrics"} drop to 0 and the KubeStateMetricsDown-style alert go from inactive to pending to firing on the Alerts page.

    kubectl -n monitoring scale deploy mon-kube-state-metrics --replicas=0
  7. Follow the alert into Alertmanager. Port-forward svc/mon-kube-prometheus-stack-alertmanager on 9093 and look at the grouped alert, then create a silence for it and watch it disappear from the active list.

  8. Clean up. kind delete cluster --name obs.

Twenty minutes of this and the kube-state-metrics vs metrics-server question will never catch you again.

Exam Cue to Answer

The question saysThe answer is
”Only increases, resets on restart”Counter
”Can go up and down”, “current value”Gauge
”le label”, “buckets”, “quantile computed at query time”Histogram (histogram_quantile)
“quantile label”, “computed in the client”, “cannot be aggregated”Summary
”Per-second rate of a counter”rate() (or irate() for the last two samples)
“Scrapes HTTP endpoints on a schedule”Pull model, scrape_interval
”Short-lived batch job”Pushgateway
”Translate a database or host into Prometheus metrics”Exporter (node_exporter for hosts)
“kubectl top”, “HPA CPU utilisation”, “Metrics API”metrics-server
”Deployment replicas available”, “Pod restarts”, “object state”kube-state-metrics
”Container CPU and memory usage”cAdvisor via the kubelet
”Route alerts, group, silence, inhibit”Alertmanager
”Define the alert condition and for duration”Prometheus alerting rule
”Precompute an expensive query”Recording rule (colon-separated name)
“Dashboards”Grafana
”Long-term, multi-cluster storage”remote_write to Thanos, Cortex or another remote store
”Discover Pods and Services automatically”kubernetes_sd_configs (roles: node, pod, service, endpoints, endpointslice, ingress)
“ServiceMonitor, PrometheusRule”Prometheus Operator CRDs
”Vendor-neutral instrumentation for traces, metrics and logs”OpenTelemetry
”Text format on /metrics”, “# TYPE”Prometheus exposition format / OpenMetrics
”100% accurate per-request billing”Not Prometheus

Scenario Drill

#ScenarioAnswerWhy
1An application exposes queue_messages_in_flight. Which type is it?GaugeIn-flight counts rise and fall
2Which expression gives HTTP requests per second from http_requests_total?rate(http_requests_total[5m])Counters need a rate function
3A team applies rate() to node_memory_MemAvailable_bytes and gets nonsense. Why?It is a gauge; use the value directly or delta()rate() assumes monotonic counters
4Which series does a classic histogram rpc_seconds produce?rpc_seconds_bucket{le=...}, rpc_seconds_sum, rpc_seconds_countBucket, sum, count triple
5Why can a 95th percentile from a summary not be averaged across ten Pods?Quantiles are computed per client and are not aggregatableUse a histogram instead
6kubectl top pods returns “Metrics API not available” although Prometheus is running.Install metrics-serverkubectl top reads metrics.k8s.io, not Prometheus
7Alert when a Deployment’s available replicas are below desired.kube-state-metrics (kube_deployment_status_replicas_available)Object state, not usage
8Find which container on a node uses the most CPU.cAdvisor metrics via the kubelet (container_cpu_usage_seconds_total)Resource usage
9A nightly job runs 20 seconds; its metrics never appear.Push to the Pushgateway; Prometheus scrapes the gatewayJob exits between scrapes
10Which label tells you a scrape failed?up == 0 for that job/instanceSynthesised per scrape
11Which component sends the page to the on-call engineer?AlertmanagerPrometheus only evaluates rules
12During a node outage, 40 Pod alerts fire alongside one node alert. Which feature hides the 40?InhibitionSuppress dependent alerts
13Suppress alerts for a planned upgrade window.Silence in AlertmanagerTime-boxed mute
14Metrics must be kept for 13 months across 30 clusters.remote_write to a long-term store (Thanos, Cortex or similar)Local TSDB defaults to 15 days, single node
15Prometheus should discover every Pod with a metrics port automatically.kubernetes_sd_configs with role: pod plus relabelingService discovery
16Declare scrape targets as Kubernetes objects.ServiceMonitor/PodMonitor via the Prometheus OperatorOperator pattern
17The HPA must scale on requests per second from Prometheus.Custom metrics API adapter (or KEDA), not metrics-serverHPA reads Kubernetes metrics APIs
18A new exporter is rejected by Prometheus 3.x with an unsupported content type error.Return a valid Content-Type (for example text/plain; version=0.0.4)Mandatory since 3.0
19Which metric name follows conventions: request_latency_ms or http_request_duration_seconds?http_request_duration_secondsPrefix, base unit, unit suffix
20A series name contains a colon: job:http_requests:rate5m. Where did it come from?A recording ruleColons are reserved for rules
21Which CNCF project standardises instrumentation for traces, metrics and logs?OpenTelemetryPrometheus is metrics only
22Which component stores Pod and Node CPU in memory for 15-second autoscaling decisions and nothing else?metrics-serverLatest values only, autoscaling only

Mistakes That Cost Marks

  • Treating metrics-server as a monitoring system. It is an autoscaling data source with no history. The project’s own README tells you not to use it for monitoring.
  • Thinking Prometheus draws dashboards or stores logs. It does neither; Grafana draws, log pipelines store logs.
  • Applying rate() to a gauge or sum() to a raw counter. Match the function to the type.
  • Forgetting by (le) in histogram_quantile. Without it the quantile is computed over an unaggregated mix and the result is wrong or empty.
  • Believing the prometheus.io/scrape annotation is magic. It is honoured by relabel rules (or by a PodMonitor), not by Kubernetes.
  • Reaching for the Pushgateway for ordinary services. It is for batch jobs; services get scraped.
  • Assuming Prometheus clusters or replicates. It does not; HA is two independent servers plus Alertmanager deduplication, and scale-out is remote write or federation.
  • Confusing OpenMetrics (a format) with OpenTelemetry (a project).

Practice the Boundaries Until They Are Reflexes

The KCNA rewards candidates who can place a tool at the correct layer in under a minute. Build the reflex with the study guide, the exam format breakdown and the free practice questions, then run the free KCNA sample exam to see how observability questions are phrased. When you want timed, full-length practice, Sailor.sh’s KCNA Certification-Ready Mock Exam Bundle spreads five mock exams across all four domains, with observability items that mix metric types, PromQL fragments and the metrics-server vs kube-state-metrics distinction in the same way the real exam does, each with an explanation of the wrong options. If the Kubestronaut path is your longer goal, the same Prometheus knowledge reappears at CKA depth when you have to debug a cluster with it.

Frequently Asked Questions

Does the KCNA require me to write PromQL?

No. The exam is multiple choice and may show a short PromQL expression inside an answer option, typically rate() on a counter or histogram_quantile() on buckets. Recognising which function belongs to which metric type is enough.

What is the difference between a histogram and a summary in one sentence?

A histogram counts observations into cumulative le buckets and lets the server compute any quantile later and across instances; a summary ships pre-computed quantiles from the client that cannot be aggregated or changed afterwards.

Does Prometheus push or pull?

Pull. It scrapes HTTP endpoints on a schedule. The Pushgateway exists for short-lived batch jobs, and remote write and the OTLP receiver are integration paths, but the primary collection model is pull.

Can the Horizontal Pod Autoscaler use Prometheus metrics?

Not directly. The HPA reads the Kubernetes Metrics API (served by metrics-server) for CPU and memory, and the custom or external metrics APIs for anything else. An adapter exposes Prometheus queries through those APIs; KEDA is the common event-driven alternative.

Is kube-state-metrics part of Kubernetes?

It is a Kubernetes project maintained under the Kubernetes organisation, but it is not installed by default. It runs as a Deployment, watches the API server, and exposes object state as Prometheus metrics.

Why did my old exporter break after upgrading to Prometheus 3?

Since 3.0 a scrape target must return a valid Content-Type header. Set text/plain; version=0.0.4 (or the OpenMetrics content type) in the exporter, or configure fallback_scrape_protocol on the scrape job.

What are native histograms and do I need them for the exam?

They are a more efficient histogram representation that stores count, sum and dynamically sized buckets in a single sample. They became a stable, opt-in feature in Prometheus 3.8. For the KCNA, knowing they exist and why they are more efficient is plenty; classic _bucket histograms are still what you will see on most endpoints.

How long does Prometheus keep data?

15 days by default, on local disk. Longer retention is a configuration change on a single server, or remote write to long-term storage for a fleet.

Conclusion

Prometheus is a small set of ideas applied consistently: a time series is a name plus labels; four metric types tell you which functions apply; targets expose a text format on /metrics; the server pulls, stamps up, job and instance, and stores locally; rules turn series into derived series and alerts; Alertmanager turns alerts into notifications. On Kubernetes the only extra work is knowing who produces which number: cAdvisor for container usage, kube-state-metrics for object state, metrics-server for autoscaling and kubectl top, and the Prometheus Operator’s CRDs to wire it all together. Learn those boundaries and every Prometheus question on the KCNA becomes a matter of reading the cue.

Limited Time Offer: Get 80% off all Mock Exam Bundles | Sale ends in 7 days. Start learning today.

Claim Now