Homelab monitoring often begins with a dashboard and ends with dozens of graphs that nobody checks. Start with questions instead: Is the host reachable? Is a disk filling? Did backup stop running? Is memory pressure growing? Which change happened before the service became slow?
Prometheus stores time-series metrics, exporters expose measurements, and Grafana visualizes and queries them. This example builds a small containerized stack for learning. It is not a substitute for backups, application checks, or an alert delivery path.
Keep the first topology small #
Use one monitoring VM with persistent storage. Run Prometheus and Grafana there, then place Node Exporter on Linux targets. Keep exporter ports on a management network and restrict them to the monitoring host; their metrics reveal hostnames, kernel details, filesystems, and resource usage.
Create a working directory with compose.yaml:
1services:
2 prometheus:
3 image: prom/prometheus
4 command:
5 - --config.file=/etc/prometheus/prometheus.yml
6 - --storage.tsdb.path=/prometheus
7 - --storage.tsdb.retention.time=30d
8 ports:
9 - 127.0.0.1:9090:9090
10 volumes:
11 - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
12 - prometheus-data:/prometheus
13 restart: unless-stopped
14
15 grafana:
16 image: grafana/grafana
17 ports:
18 - 127.0.0.1:3000:3000
19 volumes:
20 - grafana-data:/var/lib/grafana
21 restart: unless-stopped
22
23volumes:
24 prometheus-data:
25 grafana-data:
Binding to loopback avoids publishing the web ports on every VM interface. A local reverse proxy can expose them through controlled names and TLS. Pin tested image versions for a maintained deployment; floating tags are shown here only to keep the example version-neutral.
Configure scrape targets #
Prometheus reads YAML configuration. Create prometheus.yml:
1global:
2 scrape_interval: 30s
3 evaluation_interval: 30s
4
5scrape_configs:
6 - job_name: prometheus
7 static_configs:
8 - targets:
9 - localhost:9090
10
11 - job_name: linux
12 static_configs:
13 - targets:
14 - 192.0.2.31:9100
15 - 192.0.2.42:9100
16 labels:
17 site: home
Thirty seconds is frequent enough for ordinary capacity and availability monitoring without pretending that this stack is a high-resolution profiler. Labels should describe stable dimensions such as site or role. Avoid values that grow without bound, such as request IDs, because every unique label set creates another time series.
Validate the compose model and start the services:
1docker compose config
2docker compose up -d
3docker compose ps
Open the Prometheus targets page through a local tunnel or reverse proxy. Every
target should be UP. A failed scrape is useful information; do not disable TLS
verification or firewalls merely to turn the status green.
Install and restrict Node Exporter #
Use the operating system package when it is maintained for the target, or use
the official release with a systemd unit and unprivileged service account. Node
Exporter normally listens on port 9100 and exposes /metrics.
Check locally first:
1curl -fsS http://127.0.0.1:9100/metrics | head
Then allow TCP 9100 only from the monitoring VM. Do not place the exporter directly on the public internet. Enable collectors intentionally; textfile collector scripts in particular must write atomically and avoid untrusted label values.
Add Grafana without losing ownership #
Configure Prometheus as a Grafana data source using the service-to-service URL
reachable inside the Compose network, typically http://prometheus:9090.
Change the initial administrator password immediately and store it in the chosen
secret manager rather than in compose.yaml.
Importing a community dashboard is a useful shortcut, but read its queries. A dashboard can expect different metric names, mount labels, or recording rules. Keep a small home dashboard whose panels answer specific operational questions:
- host availability and scrape duration;
- filesystem free space by mount point;
- memory available and swap activity;
- CPU saturation and load;
- disk I/O latency or queue growth;
- network errors, not merely traffic volume.
Dashboards describe the past. Alerts tell someone to act.
Provision the parts worth keeping #
Clicking through Grafana is fast for exploration, but important data sources and dashboards should be exportable or provisioned from files. Keep dashboard JSON, data-source configuration without secrets, Prometheus configuration, and alert rules beside the deployment definition. That makes a blank-VM recovery possible without screenshots of settings pages.
Do not store the Grafana administrator password or notification credentials in the repository. Inject them through protected environment files or the existing secret-management process, restrict file permissions, and document how recovery retrieves them.
Prometheus configuration can also be checked before reload:
1docker compose exec prometheus \
2 promtool check config /etc/prometheus/prometheus.yml
Validation catches YAML and rule errors, while a post-reload target check catches wrong addresses and firewall policy.
Alert on symptoms with a response #
Begin with a few alerts that have clear actions. A target down for ten minutes, a filesystem predicted to fill soon, sustained memory pressure, and failed backup age are more useful than warning on every brief CPU spike.
A basic rule file could include:
1groups:
2 - name: homelab
3 rules:
4 - alert: TargetDown
5 expr: up == 0
6 for: 10m
7 labels:
8 severity: warning
9 annotations:
10 summary: "{{ $labels.instance }} is not being scraped"
11
12 - alert: FilesystemNearlyFull
13 expr: |
14 node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"}
15 / node_filesystem_size_bytes{fstype!~"tmpfs|overlay"} < 0.10
16 for: 30m
17 labels:
18 severity: warning
19 annotations:
20 summary: "Low free space on {{ $labels.instance }}"
Mount the rule file into Prometheus and reference it with rule_files. Check it
with promtool check rules from the matching Prometheus release before reload.
An alert also needs Alertmanager or another supported delivery integration.
Test the complete path and make sure a resolved notification arrives.
Every alert should link to a brief response: how to confirm it, what can be cleaned or restarted safely, and when to escalate. If nobody can explain what to do, the threshold needs refinement.
Control cardinality before it controls storage #
Prometheus creates a series for every unique metric and label combination. A metric labeled with a username, path, container hash, or request ID can create thousands of series unexpectedly. Watch active series and ingestion rate after adding an exporter. Drop unneeded metrics at scrape time only after confirming that no alert or dashboard depends on them.
Recording rules can precompute expensive, frequently used expressions. They are valuable when queries become slow, not as the first response to an unclear dashboard. Name them consistently and retain the original metrics needed for investigation.
Separate availability from performance #
The up metric reports whether Prometheus scraped a target. It does not report
whether users can complete a transaction. Add a black-box HTTP, DNS, or TCP
probe when the user-facing path matters, and keep an application metric for
domain health such as queue depth or last successful backup time.
These layers answer different questions. Exporter down means host telemetry is missing. HTTP probe down means the external path failed. Application metric stale may mean work stopped even though both infrastructure checks succeed.
Secure the query interfaces #
Prometheus and exporters are designed for trusted networks, not direct public exposure. Restrict them with network policy and place Grafana behind the chosen authenticated proxy. Use read-only viewer roles for ordinary dashboard users.
Metrics can disclose internal addresses, mount paths, software names, and test data. Review what an exporter exposes, and never place passwords, tokens, or full URLs containing credentials in labels.
Plan retention and backup #
Thirty days of metrics is enough to compare recent behavior, but storage usage depends on series count and scrape rate. Monitor Prometheus's own storage and cardinality. Longer retention is not automatically better; durable capacity trends can be recorded at lower resolution or exported elsewhere.
Back up Grafana provisioning, dashboards, and configuration. Prometheus data is often reproducible and may not deserve the same recovery objective, but decide that explicitly. Named Docker volumes live outside the compose file and must be included in whichever backup procedure is chosen.
Monitor the monitor #
A monitoring VM cannot alert while it is powered off or isolated. An external uptime check, router notification, or second tiny observer can cover that blind spot. At minimum, include the monitoring stack in the UPS and restore plans and verify that it starts before dependent alert rules are expected.
Apply updates deliberately: back up Grafana, read release notes, pull the tested images, recreate services, and check targets, queries, dashboards, and alert delivery. A running container is only the first health check.
A useful monitoring stack stays smaller than the environment it explains. Add an exporter or dashboard only when it answers a real question, and remove noisy alerts that teach everyone to ignore them.
Review the stack during ordinary maintenance. Confirm target ownership, remove retired hosts, test one notification, and open the runbook linked from an alert. That short routine keeps dashboards connected to the current lab instead of preserving a beautiful picture of last year's infrastructure. Simple monitoring that prompts action is the useful kind. Always.