Home Monitor openmediavault with Prometheus and Grafana
Post
Cancel

Monitor openmediavault with Prometheus and Grafana

After upgrading my openmediavault instance to version 6 I decided to add some enhanced monitoring of system. While openmediavault offers some basic monitoring out of the box I decided to add a Grafana dashboard for enhanced statistics.

Grafana is basically just a visualization frontend for the collected data, so I needed to add a backend which collects the data. Because Prometheus is one of the best backend for collecting data from various different data sources, I decided to use this as the backend for Grafana.

Installing Grafana along with Prometheus on openmediavault should be done using Docker. Installing Docker on openmediavault is rather easy by following this tutorial. After installing Docker and Portainer I was ready to start with Grafana and Prometheus.

For my monitoring setup I decided to add a new monitoring stack in Portainer using the following configuration:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
version: '3'

volumes:
  prometheus-data:
    driver: local
  grafana-data:
    driver: local  

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - /srv/raid_md0/Config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
  
  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped      

To start Prometheus with a custom configuration I bind-mounted the configuration to a local file in my NAS (/srv/raid_md0/Config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml). The basic configuration of Prometheus was:

1
2
3
4
5
6
7
8
9
10
11
12
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

This is the needed configuration to start Prometheus and Grafana with a minimal setup. Since Prometheus runs in a Docker container, it has limited access to scrape-able resources. But the real strength of Prometheus is to gather data from various other data sources.

To get data from openmediavault itself I decided to use the Node exporter which has been created by the Prometheus team itself. This exporter can be used within Docker itself - you just need to bind-mount the root filesystem of the host into the container. The service configuration of the Node exporter looks like:

1
2
3
4
5
6
7
8
9
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'  

In addition to the metrics of openmediavault (host) itself, I was interested into some metrics of my Docker services. I found a nice Prometheus exporter for Docker metrics called cAdvisor. The service configuration of cAdvisor looks like:

1
2
3
4
5
6
7
8
9
10
11
12
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    restart: unless-stopped

Putting this all together my stack configuration in Portainer looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
version: '3'

volumes:
  prometheus-data:
    driver: local
  grafana-data:
    driver: local  

services:
  prometheus:
    image: prom/prometheus:latest
    container_name: prometheus
    ports:
      - "9090:9090"
    volumes:
      - /srv/dev-disk-by-label-data/Config/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml
      - prometheus-data:/prometheus
    restart: unless-stopped
    command:
      - "--config.file=/etc/prometheus/prometheus.yml"
  
  grafana:
    image: grafana/grafana-oss:latest
    container_name: grafana
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    restart: unless-stopped      
  
  node_exporter:
    image: quay.io/prometheus/node-exporter:latest
    container_name: node_exporter
    command:
      - '--path.rootfs=/host'
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'  
  
  cadvisor:
    image: gcr.io/cadvisor/cadvisor:latest
    container_name: cadvisor
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro
    devices:
      - /dev/kmsg
    restart: unless-stopped

And the configuration for Prometheus (where you need to add the additional exporters too) looks like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
global:
  scrape_interval:     15s # By default, scrape targets every 15 seconds.

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # Override the global default and scrape targets from this job every 5 seconds.
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']

  # Example job for node_exporter
  - job_name: 'node_exporter'
    static_configs:
      - targets: ['node_exporter:9100']

  # Example job for cadvisor
  - job_name: 'cadvisor'
    static_configs:
      - targets: ['cadvisor:8080']

After restarting your Docker stack with the settings above you just need to add Prometheus as your first data source in Grafana by opening the “new datasource” page (e.g. http://nas.lan:3000/datasources/new) and adding the Prometheus instance:

Grafana - Prometheus as data source

The real hard part is to create a dashboard from scratch. Fortunately Grafana offers to import predefined dashboards from their repository. To use a predefined dashboard, you need to head over to the Dashboard repository and search for a dashboard which suits your needs. A good start would be:

To install a predefined dashboard, you just need to head over to the “new dashboard” page (e.g. http://nas.lan:3000/dashboard/new) and choose Import. In the next page you can either paste the url from the dashboard or the corresponding ID. After pressing Load the dashboard will be downloaded from the dashboard repository and installed in your Grafana instance.

Having those two exporters and the dashboards installed, you will have access to the basic metrics for openmediavault itself and your Docker containers.