Grafana & Prometheus: Effective Monitoring Tools

Introduction

Monitoring is like giving your system/application a check-up by mimicking how real users interact with it. Instead of waiting for problems to happen, monitoring proactively tests different parts of your system/application to make sure everything is running smoothly. Just like how you might go to the doctor for a regular check-up to catch any health issues early, monitoring helps keep your system healthy by catching potential problems before they affect real users. It's like having a virtual assistant that constantly tests your system's vital signs to ensure it's working as expected. Let's explore how we can set up our monitoring tools by using Prometheus, Blackbox Exporter, and Grafana on the docker environment.

What is Grafana?

Grafana is a data visualization and monitoring tool that allows users to create interactive dashboards and charts to analyze and understand metrics from various data sources, such as databases, cloud services, and monitoring systems like Prometheus.

Usage of Grafana

Grafana is used for:

Creating interactive dashboards.
Visualizing metrics and data from different sources.
Monitoring system performance and health.
Analyzing trends and patterns.
Alerting based on predefined conditions.
Sharing insights with team members.

What is Prometheus?

It's a smart detective that gathers data about how your system is doing by collecting metrics like CPU usage, memory usage, and response times. It is a monitoring and alerting toolkit designed for reliability and scalability, specializing in collecting and analyzing metrics from systems and applications.

Usage of Prometheus

Prometheus is used for:

Collecting metrics from various sources such as applications, services, and systems.
Storing time-series data efficiently.
Querying and analyzing collected metrics.
Creating alerts based on predefined rules.
Visualizing data through integrations with tools like Grafana.
Monitoring system performance and health in real-time.

What is Blackbox Exporter?

This tool is like a spy that goes undercover to check if your system's different parts, like websites or servers, are responding properly to requests. Blackbox Exporter is a monitoring tool that probes endpoints over various protocols (HTTP, TCP, ICMP, etc.) to assess their availability and response times from an external perspective, helping to identify potential issues in a system's components.

Usage of Blackbox Exporter

Blackbox Exporter is used for:

Probing endpoints across different protocols (HTTP, TCP, ICMP, etc.).
Assessing endpoint availability and response times.
Identifying potential issues in system components from an external perspective.
Enhancing monitoring capabilities by complementing internal system metrics with external checks.
Enabling proactive monitoring to detect and address issues before they impact users or services.

Prerequisites

We will integrate this in the docker environment so you should have basic knowledge of docker. A Docker desktop should be installed on your system.

Set-Up

Follow the following steps to set it up.

Step 1. Install Docker Desktop if you have not already installed it. Follow this article to learn how to install Docker Desktop.

Step 2. After successful Installation open your docker desktop application it will turn on your docker engine.

Step 3. Now create a new directory with your desired name like 'synthetic-monitor', Make sure that name is in lowercase and open this directory into your desired IDE like VS Code.

Step 4. Create a new 'blackbox-config.yml' file to provide configuration for blackbox. Update the code of this file as follows.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET

Here http_2xxis the name of the module being defined. It represents a check that expects a successful HTTP response in the 2xx range (e.g., 200 OK),proberSpecifies the type of probe to use for this module. In this case, HTTP indicates that the probe will send HTTP requests to endpoints.
timeout specifies the timeout duration for the probe. In this example, it's set to 5 seconds, meaning the probe will wait for a response for up to 5 seconds before timing out. HTTPThis section contains additional configuration specific to HTTP probes. valid_http_versions Specifies the valid HTTP versions that the probe should accept in responses. Here, both HTTP/1.1 and HTTP/2.0 are allowed. method specifies the HTTP method to use for the probe. In this case, it's set to "GET", indicating that the probe will send HTTP GET requests to endpoints. This configuration essentially defines a module for Blackbox Exporter that checks endpoints using HTTP probes, expecting successful responses in the 2xx range within a specified timeout period, and using the GET method for requests.

Step 5. Create a new 'prometheus.yml' file for Prometheus configuration.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com/
          - https://example2.com
          - https://example3.com/
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Make sure to replace the 'targets' URLs with your actual target URLs.

scrape_interval: This specifies the interval at which Prometheus will scrape metrics from the configured targets. In this case, it's set to every 15 seconds.
targets: This specifies that Prometheus should scrape metrics from the local instance of Prometheus itself, which is typically running on port 9090.
metrics_path: /probe: This specifies the endpoint path where Blackbox Exporter exposes its metrics for scraping by Prometheus.
params: This section specifies parameters to be passed to the Blackbox Exporter probe module.
targets: This lists the URLs of the endpoints that Blackbox Exporter will probe for availability and response times.
relabel_configs: This section specifies how labels should be modified or added to the scraped metrics.
source_labels: [__address__]: This specifies the source label to use for the replacement.
target_label: __param_target: This specifies the target label to be replaced with the source label's value.
source_labels: [__param_target]: This specifies the source label whose value will be used as the replacement instance label.
target_label: instance: This specifies the target label to set with the value from the source label.
target_label: __address__: This specifies the target label to replace with the specified replacement value, which is the address of the Blackbox Exporter.

Step 6. Now create a network in the docker environment by running the following command in the command prompt. If you don't know what is the network in docker explore this article to learn.

docker network create synthetic-tool

Here 'synthetic-tool' is the name of the network.

Output

Synthetic-tool

Step 7. Now deploy the container for the black box exporter by running the following command in the command prompt.

docker run -d --name blackbox-exporter --network synthetic-tool -p 9115:9115 -v <PATH OF WORKING DIRECTORY>/blackbox-config.yml:/etc/blackbox_exporter/config.yml prom/blackbox-exporter

Make sure to replace 'PATH OF WORKING DIRECTORY' with your actual directory path.

Output

DIRECTORY

Here, blackbox-exporter container is deployed you can verify this on your docker desktop or by running the following command.

docker ps

Output

Docker

On docker desktop.

Docker desktop

Step 8. Now deploy the container for Prometheus by running the following commands.

docker run -d --name prometheus --network synthetic-tool  -p 9090:9090 -v <PATH OF WORKING DIRECTORY>/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Make sure to replace 'PATH OF WORKING DIRECTORY' with your actual directory path.

Output

PATH OF WORKING

Here Prometheus container is deployed, you can verify this by using the 'docker ps' command.

Step 9. Now deploy the container for the Grafana dashboard by running the following command in the command prompt.

docker run -d --name grafana --network synthetic-tool -p 3000:3000 grafana/grafana

Output

Grafana dashboard

Step 10. You are all set now open the docker desktop and check that all three containers are running if not then start them. Now navigate to https://localhost:3000/ to view the Grafana dashboard. In Grafana, log in using 'admin' as both the username and password.

Grafana

Step 11. In Grafana, add your data source through the settings.

Settings

Step 12. Select 'Prometheus', and in the connection section, enter your Prometheus URL as 'http://prometheus:9090'

Prometheus

Connection

Step 13. Finally, click on 'Save and Test'.

Save and Test

Step 14. Now follow these sub-steps to create a dashboard.

Click on the "+" on the right sidebar and choose "New dashboard."
Click on "+ Add Visualization".
In the "select data source" section, choose the Prometheus data source.
Select metrics in the metric dropdown. You can start with simple queries like 'probe_duration_seconds' or other metrics provided by Blackbox Exporter.
Click on the 'Run queries'.

Customize your queries based on the metrics you want to visualize. Use the metrics collected by Prometheus, such as those related to the probes configured in Blackbox Exporter. Choose the visualization type for your panel (e.g., Graph, Singlestat, Table). Configure additional settings such as axes, legends, and visualization options. Arrange your panels on the dashboard to create a clear and informative layout. You can add multiple panels to visualize different metrics.

Click on the 'Save ' at the top to save your dashboard. Provide a name for your dashboard and click on save.
You will see the visualization as follows.

Conclusion

Monitoring offers a proactive approach to ensure the smooth functioning of systems and applications by mimicking real user interactions. By integrating tools like Prometheus, Blackbox Exporter, and Grafana within a Docker environment, setting up comprehensive monitoring becomes accessible. This process involves configuring these tools, creating containers, and visualizing data on Grafana dashboards. Through this setup, you can effectively monitor vital signs, detect potential issues early, and maintain the health of your systems, ultimately enhancing user experience and system reliability.