Prometheus and Grafana: Monitoring Stack Setup Guide

Enable and start it:

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Verify it's working by visiting http://localhost:9100/metrics. You'll see hundreds of system metrics in Prometheus exposition format. Go back to your Prometheus config and make sure you've added the node scrape job (we did this above). Reload Prometheus to pick up the new target.

Installing and Configuring Grafana

Grafana's installation varies by platform. On Ubuntu, the easiest approach is using the official repository:

sudo apt-get install -y software-properties-common
sudo add-apt-repository "deb https://packages.grafana.com/oss/deb stable main"
sudo apt-get update
sudo apt-get install -y grafana-server

Start Grafana:

sudo systemctl enable grafana-server
sudo systemctl start grafana-server
sudo systemctl status grafana-server

Grafana runs on port 3000. Open http://localhost:3000 in your browser. The default credentials are admin/admin. You'll be prompted to change the password on first login—do that immediately.

Now add Prometheus as a data source. Go to Configuration → Data Sources → Add Data Source. Select Prometheus. Set the URL to http://localhost:9090. Click Save & Test. If everything's connected, you'll see a green message confirming the link works.

Creating Your First Dashboard

With Prometheus feeding data into Grafana, you're ready to build dashboards. Let's start simple: create a panel showing CPU usage.

Click the + icon in the sidebar and select Dashboard. Add a new panel. In the query editor, enter a PromQL query. Here's a useful one for CPU:

100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

This calculates the percentage of CPU not idle over the last 5 minutes. Name the panel "CPU Usage", set the units to percent, and save. You now have a working dashboard panel.

Want to add more panels? Try memory usage:

(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100

Or disk usage:

(1 - (node_filesystem_avail_bytes{fstype!~"tmpfs|fuse.lowerfs|squashfs|vfat"} / node_filesystem_size_bytes{fstype!~"tmpfs|fuse.lowerfs|squashfs|vfat"})) * 100

Grafana has thousands of pre-built dashboards available on Grafana's dashboard repository. You can import them by ID in seconds. Dashboard 1860 (Node Exporter Full) is particularly popular for system monitoring—it's comprehensive and well-maintained.

Configuring Alerts

Metrics alone aren't enough. You need to be notified when something's wrong. Prometheus and Grafana both support alerting, though they work differently.

In Prometheus, create alert rules by adding a rules file. Create /opt/prometheus/rules.yml:

groups:
  - name: system_alerts
    interval: 30s
    rules:
      - alert: HighCPUUsage
        expr: (100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)) > 80
        for: 5m
        annotations:
          summary: "High CPU usage detected on {{ $labels.instance }}"
          description: "CPU usage is {{ $value }}%"
      
      - alert: HighMemoryUsage
        expr: (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value }}%"

Update prometheus.yml to include these rules:

rule_files:
  - "rules.yml"

Reload Prometheus to activate alerts. They'll appear in the Alerts section of the web UI. To actually get notified, you need an Alertmanager instance. For now, focus on getting the alerts firing correctly in the UI.

Using Docker Compose (Optional)

If you prefer containerization, here's a docker-compose.yml that brings everything up in one command:

version: '3.8'
services:
  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9090:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus_data:/prometheus
    command:
      - '--config.file=/etc/prometheus/prometheus.yml'
    restart: always

  node_exporter:
    image: prom/node-exporter:latest
    ports:
      - "9100:9100"
    restart: always

  grafana:
    image: grafana/grafana:latest
    ports:
      - "3000:3000"
    environment:
      GF_SECURITY_ADMIN_PASSWORD: admin
    volumes:
      - grafana_data:/var/lib/grafana
    restart: always

volumes:
  prometheus_data:
  grafana_data:

Run docker-compose up -d and everything starts together. This is great for testing and development. For production, you'll want to customize image versions, environment variables, and storage.

Best Practices for Production

Once you're up and running, here's what separates a hobby setup from a production-grade monitoring stack:

Retention Policy: Prometheus keeps metrics in memory and on disk. By default, it retains 15 days of data. For production, decide based on your needs. High-traffic systems might drop this to 7 days to save disk space. Use --storage.tsdb.retention.time=7d when starting Prometheus.

Remote Storage: Local storage doesn't scale forever. For long-term retention, consider remote storage backends like Thanos, Cortex, or Victoria Metrics. They're designed to handle petabyte-scale metrics.

Scrape Interval: The default 15-second scrape interval is fine for most setups. Lower it to 5 seconds only if you need high granularity and can handle the storage overhead