Monitoring with Prometheus and Grafana

Installing node_exporter

First, we will download the Node Exporter on all machines :

wget https://github.com/prometheus/node_exporter/releases/download/v0.15.2/node_exporter-0.15.2.linux-amd64.tar.gz

Extract the downloaded archive

tar -xf node_exporter-0.15.2.linux-amd64.tar.gz

Move the node_exporter binary to /usr/local/bin:

sudo mv node_exporter-0.15.2.linux-amd64/node_exporter /usr/local/bin

Remove the residual files with:

rm -r node_exporter-0.15.2.linux-amd64*

Next,we will create users and service files for node_exporter.

sudo useradd -rs /bin/false node_exporter

Then, we will create a systemd unit file so that node_exporter can be started at boot.

sudo nano /etc/systemd/system/node_exporter.service

[Unit]
Description=Node Exporter
After=network.target

[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target

Since we have created a new unit file, we must reload the systemd daemon, set the service to always run at boot and start it :

sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Installing Prometheus

The next step is to download and install Prometheus only on the Prometheus Server.

wget https://github.com/prometheus/prometheus/releases/download/v2.1.0/prometheus-2.1.0.linux-amd64.tar.gz

Extract the Prometheus archive :

tar -xf prometheus-2.1.0.linux-amd64.tar.gz

Move the binaries to /usr/local/bin:

sudo mv prometheus-2.1.0.linux-amd64/prometheus prometheus-2.1.0.linux-amd64/promtool /usr/local/bin

Now, we need to create directories for configuration files and other prometheus data.

sudo mkdir /etc/prometheus /var/lib/prometheus

Then, we move the configuration files to the directory we made previously:

sudo mv prometheus-2.1.0.linux-amd64/consoles prometheus-2.1.0.linux-amd64/console_libraries /etc/prometheus

Finally, we can delete the leftover files as we do not need them any more:

rm -r prometheus-2.1.0.linux-amd64*

Configuring Prometheus

After having installed Prometheus, we have to configure Prometheus to let it know about the HTTP endpoints it should monitor. Prometheus uses the YAML format for its configuration.

Go to /etc/hosts and add the following lines, replace x.x.x.x with the machine’s corresponding IP address

x.x.x.x prometheus-target-1
x.x.x.x prometheus-target-2

We will use /etc/prometheus/prometheus.yml as our configuration file

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'prometheus_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9090']
  - job_name: 'node_exporter_metrics'
    scrape_interval: 5s
    static_configs:
      - targets: ['localhost:9100','prometheus-target-1:9100','prometheus-target-2:9100']

In this file, we have defined a default scraping interval of 10 seconds. We have also define two sources of metrics, named prometheus_metrics and node_exporter_metrics. For both of them, we have set the scraping interval to 5 seconds, overriding the default. Then, we have specified the locations where these metrics will be available. Prometheus uses port 9090 and node_exporter uses port 9100 to provide their metrics.

Finally, we will also change the ownership of files that Prometheus will use:

sudo useradd -rs /bin/false prometheussudo 
chown -R prometheus: /etc/prometheus /var/lib/prometheus

Then, we will create a systemd unit file in /etc/systemd/system/prometheus.service with the following contents :

[Unit]
Description=Prometheus
After=network.target

[Service]
User=prometheus
Group=prometheus
Type=simple
ExecStart=/usr/local/bin/prometheus \
    --config.file /etc/prometheus/prometheus.yml \
    --storage.tsdb.path /var/lib/prometheus/ \
    --web.console.templates=/etc/prometheus/consoles \
    --web.console.libraries=/etc/prometheus/console_libraries

[Install]
WantedBy=multi-user.target

Finally, we will reload systemd:

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus

Prometheus provides a web UI for running basic queries located at http://<your_server_IP>:9090/. This is how it looks like in a web browser:

Note: Remember add 9090 port number to your instance security group

Setting up Grafana For Prometheus

First, Install Grafana on our instance which queries our Prometheus server.

wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.0.4_amd64.deb
sudo apt-get install -y adduser libfontconfig
sudo dpkg -i grafana_5.0.4_amd64.deb

Then, Enable the automatic start of Grafana by systemd:

sudo systemctl daemon-reload
 && sudo systemctl enable grafana-server 
 && sudo systemctl start grafana-server.service

Grafana is running now, and we can connect to it at http://your.server.ip:3000. The default user and password is admin / admin.

Note: Remember add 3000 port number to your instance security group

Now you have to create a Prometheus data source:

• Click on the Grafana logo to open the sidebar.
• Click on “Data Sources” in the sidebar.
• Choose “Add New”.
• Select “Prometheus” as the data source.
• Set the Prometheus server URL (in our case: http://localhost:9090/)
• Click “Add” to test the connection and to save the new data source.

Your settings should look like this: