CodeWithAbdessamad

Monitoring

Monitoring

In today’s cloud-native environments, effective monitoring isn’t just about collecting data—it’s the foundation for proactive system reliability and rapid incident resolution. This section dives into the industry-standard tools that power modern observability pipelines: Prometheus for metric collection and Grafana for visualization. Together, they form a robust, open-source monitoring stack that works seamlessly with Kubernetes.

Prometheus: The Open-Source Monitoring and Alerting Toolkit

Prometheus is a time-series database and monitoring system designed specifically for cloud-native environments. Unlike traditional monitoring tools, it focuses on high-performance metric collection, flexible query language, and native alerting—making it ideal for Kubernetes clusters. Its architecture revolves around three core principles: scraping metrics from targets, storing time-series data, and triggering alerts based on thresholds.

Why Prometheus?

Prometheus excels in Kubernetes due to:

  • Target-based scraping: Metrics are pulled from endpoints rather than pushed, reducing complexity
  • Granular metric resolution: Collects metrics at 15-second intervals by default (configurable)
  • Reliability: Built-in service discovery and health checks
  • Kubernetes-native: Integrates directly with Kubernetes objects via ServiceMonitor and PodMonitor resources

Core Concepts in Action

Before diving into implementation, let’s clarify key Prometheus concepts:

Concept Purpose Example in Kubernetes Context
Target Endpoint scraped for metrics (e.g., a pod, service, or node) kubernetes-pods in ServiceMonitor
Metric Single data point (e.g., httprequeststotal) kubepodcontainerstatusrunning
Scrape Interval Frequency at which metrics are collected 15s (default)
Alert Rule Condition to trigger alerts (e.g., avg(rate(httprequeststotal[5m])) > 1000) Configured in Prometheus Rule files

Deploying Prometheus in Kubernetes

Here’s a minimal, runnable example to deploy Prometheus with a single pod that emits metrics. This uses the official Prometheus Helm chart for simplicity.

First, create a ServiceMonitor to scrape metrics from a simple HTTP server pod:

<code class="language-yaml"># metrics-pod.yaml
<p>apiVersion: v1</p>
<p>kind: Pod</p>
<p>metadata:</p>
<p>  name: metrics-pod</p>
<p>spec:</p>
<p>  containers:</p>
<p>  - name: metrics</p>
<p>    image: prom/prometheus:v2.35.0  # Simplified example</p>
<p>    command: ["/bin/sh"]</p>
<p>    args: ["-c", "echo 'HTTP 200' && sleep 100"]</code>

Next, define a ServiceMonitor to collect metrics from this pod:

<code class="language-yaml"># service-monitor.yaml
<p>apiVersion: monitoring.coreos.com/v1</p>
<p>kind: ServiceMonitor</p>
<p>metadata:</p>
<p>  name: metrics-pod</p>
<p>spec:</p>
<p>  selector:</p>
<p>    matchLabels:</p>
<p>      app: metrics-pod</p>
<p>  endpoints:</p>
<p>  - port: http</p>
<p>    interval: 15s</code>

Finally, deploy Prometheus using the Helm chart:

<code class="language-bash"># Install Prometheus via Helm
<p>helm repo add prometheus-community https://charts.prometheus-community.github.io</p>
<p>helm install prometheus prometheus-community/prometheus -n monitoring --set serviceMonitorSelector[0].matchLabels[app]=metrics-pod</code>

Why this works: The ServiceMonitor tells Prometheus to scrape the metrics-pod every 15 seconds. Prometheus automatically discovers the pod via Kubernetes labels and stores metrics in its time-series database. You can verify metrics at http://:9090/metrics.

Real-World Kubernetes Monitoring

In production, you’d extend this with:

  1. Kube-state-metrics: A Prometheus adapter for Kubernetes objects
  2. Node Exporter: For host-level metrics
  3. Alerting Rules: To notify teams of critical events

Here’s a minimal alert rule for pod restarts (triggers when kubepodcontainerstatusrestarts_total exceeds 5):

<code class="language-yaml"># alert-rules.yaml
<p>groups:</p>
<ul>
<li>name: kubernetes</li>
<p></ul>  rules:</p>
<p>  - alert: PodRestartsExceeded</p>
<p>    expr: kube<em>pod</em>container<em>status</em>restarts_total{job="kube-state-metrics"} > 5</p>
<p>    for: 5m</p>
<p>    labels:</p>
<p>      severity: critical</p>
<p>    annotations:</p>
<p>      summary: "Pod restarts exceeded threshold"</code>

This rule runs in the background and sends Slack/Email alerts when pods restart excessively—critical for detecting misconfigurations or failing services.

Grafana: Visualization and Dashboarding for Observability

Grafana is a powerful open-source platform for visualizing metrics, dashboards, and alerts. It excels at transforming raw Prometheus metrics into intuitive, interactive dashboards—making it the go-to tool for observability teams. Unlike static reporting tools, Grafana enables real-time exploration and drill-downs into your infrastructure.

Why Grafana?

Grafana solves three critical problems in monitoring:

  • Data consolidation: Aggregates metrics from multiple sources (Prometheus, Loki, Datadog)
  • Interactive exploration: Lets users query metrics in real-time without writing SQL
  • Customizable dashboards: Tailored to specific teams (devs, SREs, operations)

Connecting Grafana to Prometheus

The simplest way to start is via the Prometheus data source. Here’s how to set it up:

  1. Deploy Grafana using the official Helm chart:
<code class="language-bash">helm install grafana grafana/grafana -n monitoring --set adminPassword=your<em>secure</em>password</code>

  1. Add Prometheus as a data source:

– Access Grafana at http://:3000

– Go to Configuration > Data Sources

– Click Add data source

– Select Prometheus and enter:

– URL: http://prometheus-service:9090

– Org: Default

– Timeout: 5s

  1. Create a dashboard:

– Click New Dashboard

– Add a Grafana Panel with:

– Type: Time Series

– Query: kubepodcontainerstatusrunning{job="kube-state-metrics"}

Building a Production Kubernetes Dashboard

Here’s a real-world dashboard for Kubernetes monitoring—used by many teams:

  1. Pod Health Dashboard:

– Shows pod status over time with a green/red indicator

– Uses kubepodstatus_phase metric

  1. Resource Utilization:

– CPU and memory usage per node

– Based on kubenodecpuusageseconds_total

  1. Alerts Integration:

– Displays active alerts from Prometheus

– Configured via Alerting > Alert Rules

Example Dashboard Snippet:

<code class="language-yaml"># dashboard.json (simplified)
<p>{</p>
<p>  "title": "Kubernetes Health Overview",</p>
<p>  "panels": [</p>
<p>    {</p>
<p>      "type": "graph",</p>
<p>      "title": "Pod Status",</p>
<p>      "targets": [</p>
<p>        {</p>
<p>          "expr": "kube<em>pod</em>status_phase{phase=\"Running\"}",</p>
<p>          "interval": "5m"</p>
<p>        }</p>
<p>      ]</p>
<p>    },</p>
<p>    {</p>
<p>      "type": "graph",</p>
<p>      "title": "Node CPU Usage",</p>
<p>      "targets": [</p>
<p>        {</p>
<p>          "expr": "rate(kube<em>node</em>cpu<em>usage</em>seconds_total{job=\"node-exporter\"}[5m])"</p>
<p>        }</p>
<p>      ]</p>
<p>    }</p>
<p>  ]</p>
<p>}</code>

This dashboard shows:

  • Pod health: Running vs. terminating pods
  • Node resource usage: Real-time CPU utilization
  • Alerts: Critical alerts appear as red cards on the dashboard

Pro Tips for Effective Grafana Usage

  • Use annotations: Add context to metrics (e.g., environment, region)
  • Leverage alerts: Configure Grafana to trigger Slack/email notifications
  • Save dashboards: Share with teams via Share > Save as Dashboard
  • Customize queries: Use Prometheus Query Language (PQL) for advanced analysis

Summary

Prometheus provides the essential backbone for collecting, storing, and alerting on metrics in Kubernetes environments—while Grafana transforms those metrics into actionable insights through interactive dashboards. Together, they form a powerful, production-ready observability stack that helps teams maintain resilient cloud-native systems. By deploying Prometheus with Kubernetes-native scraping and connecting it to Grafana, you gain real-time visibility into your infrastructure without vendor lock-in. Start small with a single pod and scale to enterprise-grade monitoring as your cluster grows—this combination is the industry standard for observability in Kubernetes. 🌟