Introduction

Monitoring is like giving your system/application a check-up by mimicking how real users interact with it. Instead of waiting for problems to happen, monitoring proactively tests different parts of your system/application to make sure everything is running smoothly. Just like how you might go to the doctor for a regular check-up to catch any health issues early, monitoring helps keep your system healthy by catching potential problems before they affect real users. It's like having a virtual assistant that constantly tests your system's vital signs to ensure it's working as expected. Let's explore how we can set up our monitoring tools by using Prometheus, Blackbox Exporter, and Grafana on the docker environment.

What is Grafana?

Grafana is a data visualization and monitoring tool that allows users to create interactive dashboards and charts to analyze and understand metrics from various data sources, such as databases, cloud services, and monitoring systems like Prometheus.

Usage of Grafana

Grafana is used for:

What is Prometheus?

It's a smart detective that gathers data about how your system is doing by collecting metrics like CPU usage, memory usage, and response times. It is a monitoring and alerting toolkit designed for reliability and scalability, specializing in collecting and analyzing metrics from systems and applications.

Usage of Prometheus

Prometheus is used for:

What is Blackbox Exporter?

This tool is like a spy that goes undercover to check if your system's different parts, like websites or servers, are responding properly to requests. Blackbox Exporter is a monitoring tool that probes endpoints over various protocols (HTTP, TCP, ICMP, etc.) to assess their availability and response times from an external perspective, helping to identify potential issues in a system's components.

Usage of Blackbox Exporter

Blackbox Exporter is used for:

Prerequisites

We will integrate this in the docker environment so you should have basic knowledge of docker. A Docker desktop should be installed on your system.

Set-Up

Follow the following steps to set it up.

Step 1. Install Docker Desktop if you have not already installed it. Follow this article to learn how to install Docker Desktop.

Step 2. After successful Installation open your docker desktop application it will turn on your docker engine.

Step 3. Now create a new directory with your desired name like 'synthetic-monitor', Make sure that name is in lowercase and open this directory into your desired IDE like VS Code.

Step 4. Create a new 'blackbox-config.yml' file to provide configuration for blackbox. Update the code of this file as follows.

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      method: GET

Here http_2xxis the name of the module being defined. It represents a check that expects a successful HTTP response in the 2xx range (e.g., 200 OK),proberSpecifies the type of probe to use for this module. In this case, HTTP indicates that the probe will send HTTP requests to endpoints.
timeout specifies the timeout duration for the probe. In this example, it's set to 5 seconds, meaning the probe will wait for a response for up to 5 seconds before timing out. HTTPThis section contains additional configuration specific to HTTP probes. valid_http_versions Specifies the valid HTTP versions that the probe should accept in responses. Here, both HTTP/1.1 and HTTP/2.0 are allowed. method specifies the HTTP method to use for the probe. In this case, it's set to "GET", indicating that the probe will send HTTP GET requests to endpoints. This configuration essentially defines a module for Blackbox Exporter that checks endpoints using HTTP probes, expecting successful responses in the 2xx range within a specified timeout period, and using the GET method for requests.

Step 5. Create a new 'prometheus.yml' file for Prometheus configuration.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  - job_name: 'blackbox'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com/
          - https://example2.com
          - https://example3.com/
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Make sure to replace the 'targets' URLs with your actual target URLs.

Step 6. Now create a network in the docker environment by running the following command in the command prompt. If you don't know what is the network in docker explore this article to learn.

docker network create synthetic-tool

Here 'synthetic-tool' is the name of the network.

Output

Synthetic-tool

Step 7. Now deploy the container for the black box exporter by running the following command in the command prompt.

docker run -d --name blackbox-exporter --network synthetic-tool -p 9115:9115 -v <PATH OF WORKING DIRECTORY>/blackbox-config.yml:/etc/blackbox_exporter/config.yml prom/blackbox-exporter

Make sure to replace 'PATH OF WORKING DIRECTORY' with your actual directory path.

Output

DIRECTORY

Here, blackbox-exporter container is deployed you can verify this on your docker desktop or by running the following command.

docker ps

Output

Docker

On docker desktop.

Docker desktop

Step 8. Now deploy the container for Prometheus by running the following commands.

docker run -d --name prometheus --network synthetic-tool  -p 9090:9090 -v <PATH OF WORKING DIRECTORY>/prometheus.yml:/etc/prometheus/prometheus.yml prom/prometheus

Make sure to replace 'PATH OF WORKING DIRECTORY' with your actual directory path.

Output

PATH OF WORKING

Here Prometheus container is deployed, you can verify this by using the 'docker ps' command.

Step 9. Now deploy the container for the Grafana dashboard by running the following command in the command prompt.

docker run -d --name grafana --network synthetic-tool -p 3000:3000 grafana/grafana

Output

Grafana dashboard

Step 10. You are all set now open the docker desktop and check that all three containers are running if not then start them. Now navigate to https://localhost:3000/ to view the Grafana dashboard. In Grafana, log in using 'admin' as both the username and password.

Grafana

Step 11. In Grafana, add your data source through the settings.

Settings

Step 12. Select 'Prometheus', and in the connection section, enter your Prometheus URL as 'http://prometheus:9090'

Prometheus

Connection

Step 13. Finally, click on 'Save and Test'.

Save and Test

Step 14. Now follow these sub-steps to create a dashboard.

Customize your queries based on the metrics you want to visualize. Use the metrics collected by Prometheus, such as those related to the probes configured in Blackbox Exporter. Choose the visualization type for your panel (e.g., Graph, Singlestat, Table). Configure additional settings such as axes, legends, and visualization options. Arrange your panels on the dashboard to create a clear and informative layout. You can add multiple panels to visualize different metrics.

Conclusion

Monitoring offers a proactive approach to ensure the smooth functioning of systems and applications by mimicking real user interactions. By integrating tools like Prometheus, Blackbox Exporter, and Grafana within a Docker environment, setting up comprehensive monitoring becomes accessible. This process involves configuring these tools, creating containers, and visualizing data on Grafana dashboards. Through this setup, you can effectively monitor vital signs, detect potential issues early, and maintain the health of your systems, ultimately enhancing user experience and system reliability.