Most self hosted setups run blind — you find out something's wrong when a user complains, a disk fills up silently, or a service has been down for hours before…
Most self-hosted setups run blind — you find out something's wrong when a user complains, a disk fills up silently, or a service has been down for hours before anyone notices. Prometheus and Grafana together solve this with almost no ongoing maintenance: Prometheus scrapes and stores metrics over time, Grafana turns them into dashboards and alerts, and both are free and self-hostable. This guide sets up monitoring for a Linux server from a cold start to a working dashboard with alerting.
Prerequisites / What You'll Need
- A Linux server (or VM) to run the monitoring stack on
- Docker and Docker Compose installed
- One or more servers you want to monitor (can be the same machine to start)
- 30–40 minutes
Step 1: Understand the Architecture
- Prometheus — pulls ("scrapes") metrics from targets at regular intervals and stores them as time-series data
- Node Exporter — a small agent that runs on each monitored server, exposing CPU, memory, disk, and network metrics for Prometheus to scrape
- Grafana — queries Prometheus and renders the data as dashboards, with alerting on top
The key mental model: Prometheus pulls metrics rather than services pushing them — so you configure Prometheus with a list of targets to scrape, not the other way around.
Step 2: Set Up the Monitoring Stack with Docker Compose
Create a directory and a docker-compose.yml:
services:
prometheus:
image: prom/prometheus:latest
restart: unless-stopped
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
- prometheus_data:/prometheus
ports:
- "9090:9090"
grafana:
image: grafana/grafana:latest
restart: unless-stopped
environment:
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
volumes:
- grafana_data:/var/lib/grafana
ports:
- "3000:3000"
volumes:
prometheus_data:
grafana_data:
Create a .env file alongside it:
GRAFANA_PASSWORD=change_this_to_something_strong
And add .env to .gitignore if this directory is version-controlled.
Step 3: Install Node Exporter on Each Monitored Server
On every server you want metrics from (including this one, if you want to monitor it too):
wget https://github.com/prometheus/node_exporter/releases/download/v1.8.2/node_exporter-1.8.2.linux-amd64.tar.gz
tar xvfz node_exporter-1.8.2.linux-amd64.tar.gz
sudo mv node_exporter-1.8.2.linux-amd64/node_exporter /usr/local/bin/
Create a systemd service so it starts on boot:
sudo tee /etc/systemd/system/node_exporter.service > /dev/null <<EOF
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
EOF
sudo useradd --no-create-home --shell /bin/false node_exporter
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter
Confirm it's exposing metrics:
curl http://localhost:9100/metrics | head
Security note: Node Exporter has no built-in authentication. Never expose port 9100 directly to the internet — restrict it with a firewall rule to only allow your Prometheus server's IP.
sudo ufw allow from <prometheus-server-ip> to any port 9100
Step 4: Configure Prometheus to Scrape Your Targets
Create prometheus.yml in the same directory as your docker-compose.yml:
global:
scrape_interval: 15s
scrape_configs:
- job_name: 'node_exporter'
static_configs:
- targets:
- 'server1-ip:9100'
- 'server2-ip:9100'
labels:
env: 'production'
Each entry in targets is a host:port Prometheus will poll every 15 seconds. Add one line per server you're monitoring.
Step 5: Launch the Stack
docker compose up -d
Check Prometheus sees your targets by visiting http://your-monitoring-server:9090/targets — every target should show State: UP. If one shows DOWN, it's almost always a firewall rule blocking port 9100, or Node Exporter not actually running on that host.
Step 6: Connect Grafana to Prometheus
Log into Grafana at http://your-monitoring-server:3000 (default user admin, password whatever you set in .env).
- Go to Connections → Data sources → Add data source
- Choose Prometheus
- Set the URL to
http://prometheus:9090 (the Docker service name — Compose's internal networking resolves this automatically)
- Click Save & test — you should see a confirmation it connected successfully
Step 7: Import a Ready-Made Dashboard
Rather than building panels from scratch, Grafana's community library has excellent pre-built dashboards:
- Go to Dashboards → New → Import
- Enter dashboard ID 1860 (the widely-used "Node Exporter Full" dashboard)
- Select your Prometheus data source
- Click Import
You'll immediately get CPU, memory, disk I/O, and network graphs for every server you're scraping — no manual panel configuration needed.
Step 8: Set Up a Basic Alert
Alerting is where monitoring actually starts paying off — you want to know about problems before a user tells you. In Grafana:
- Go to Alerting → Alert rules → New alert rule
- Set a query, for example disk space below 10%:
(node_filesystem_avail_bytes{mountpoint="/"} / node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
- Set the evaluation interval (e.g., every 1 minute) and a "for" duration (e.g., 5 minutes, so a brief blip doesn't trigger a false alarm)
- Under Contact points, configure a notification channel — email, Slack webhook, or Discord all work
This one alert alone — low disk space — catches a huge share of real-world outages before they happen.
Common Pitfalls / Troubleshooting
- Target shows
DOWN in Prometheus. Almost always a connectivity issue. SSH into the monitoring server and run curl http://target-ip:9100/metrics directly — if that fails, it's a firewall rule; if it works but Prometheus still shows DOWN, double-check the targets list in prometheus.yml for typos.
- Exposing Node Exporter to the public internet. It leaks detailed system information (running processes, resource usage) with zero authentication by default. Firewall it to only your Prometheus server, never leave
9100 open broadly.
- Grafana dashboard shows "No data." Usually means the data source or dashboard's variable (often labeled
job or instance) doesn't match what's actually in Prometheus — check the dashboard's variable dropdowns at the top of the screen.
- Prometheus disk usage growing unbounded. By default Prometheus retains data for 15 days, but on a long-running install this can still add up. Set explicit retention with
--storage.tsdb.retention.time=30d as a Prometheus command argument if you need a different window.
- Forgetting to change the default Grafana password. The
.env approach above avoids this, but double-check it's not still on admin/admin if you set this up quickly and skipped that step.
Wrapping Up
With this running, you go from finding out about problems reactively to seeing them coming — a disk filling up, memory creeping toward a limit, a service that's been silently restarting. The setup here scales cleanly too: adding a new server to monitor is just installing Node Exporter and adding one line to prometheus.yml.
From here, worthwhile next steps include adding exporters for specific services you run — postgres_exporter for database metrics, or nginx-prometheus-exporter for web server stats — and setting up Alertmanager directly for more advanced routing and deduplication of alerts than Grafana's built-in alerting offers.
Further Reading