Telemetry in ASP.NET Core Web API with Prometheus

Introduction

In my previous article, I talked about telemetry and its benefits, and I implemented a telemetry sample using Jaeger.

In this article, I will continue on the same subject, but I will show other tools that can help to enhance the metrics visibility of your Web API. I will start by creating a Web API project, then I'll use Prometheus to expose the metrics and then I'll connect Prometheus with Grafana to have a clear dashboard about the metrics.

Prerequisite

If you want to have an idea about the definition of telemetry, please have a look at the introduction of my previous article.
Visual Studio 2022
Prometheus Nuget packages
Prometheus, you can download it here Download | Prometheus (Version : 2.51.0 / 2024-03-18 for this aricle)
Grafana, you can download it here: Download Grafana | Grafana Labs (Version: Grafana-enterprise-10.4.1 for this article)

Prometheus

Prometheus is an open-source systems monitoring and alerting toolkit [...] it collects and stores its metrics as time series data, i.e. metrics information is stored with the timestamp at which it was recorded, alongside optional key-value pairs called labels. source: Prometheus website

Any application that wants to raise monitoring data in a way that Prometheus understands it, should integrate the Prometheus exporter and expose its metrics through it.

In our case, we will deploy the Prometheus server locally and integrate the nugget packages in the WebApi.

Grafana

Grafana open-source software enables you to query, visualize, alert on, and explore your metrics, logs, and traces wherever they are stored. Grafana OSS provides you with tools to turn your time-series database (TSDB) data into insightful graphs and visualizations. Source : grafana website

In our case, we will use it to visualize the metrics exported by the Web API into Prometheus.

Remarks

Since we are focusing on the metrics management in this article, we will proceed in HTTP mode to avoid having certificate issues during the Prometheus connection So create your web API in HTTP mode
The interval time of Prometheus scrape is configured in the YAML configuration file which we will see further in this article. So if you perform a call in the Api, wait a while before checking the Prometheus side)

Step 1. Create a webApi netcore project

Create a new Web API net core project and follow the instructions. You can call it TelemetryUsingPrometheus.

Web API

Step 2. Add NuGet Packages

Double-click on your TelemetryUsingPrometheus.csproj then add the following packages under the section ItemGroup and they will be installed automatically from Nuget when you build the solution.

<PackageReference Include="prometheus-net" Version="8.2.1" />
<PackageReference Include="prometheus-net.AspNetCore" Version="8.2.1" />

Step 3. Initialize the Prometheus exporter

Once you added the needed packages, you are able to use them in your code.

In your program.cs, you need to indicate that the solution will use the metrics.

//These metrics are called to use prometheus
app.UseMetricServer();
app.UseHttpMetrics();

Step 4. Set the metrics in your code

Prometheus offers several objects of metrics. You can view your metrics in the form of a histogram, gauge, counter, or summary.

So, you need to declare them and then perform the needed updates related to your performance data on every object.

In our case, we will set them from the API call just to have more dynamic interaction. But in real life, you should update the metrics objects from data related to the execution of your code.

using Microsoft.AspNetCore.Mvc;
using Prometheus;

// For more information on enabling Web API for empty projects, visit https://go.microsoft.com/fwlink/?LinkID=397860

namespace TelemetryUsingPrometheus.Controllers
{
    [Route("api/[controller]")]
    [ApiController]
    public class TestTelemetryController : ControllerBase
    {
        Counter _counter;
        Histogram _histogram;
        Gauge _gauge;
        Summary _summary;

        public TestTelemetryController()
        {
            _counter = Metrics.CreateCounter("TestMetricCounter", "will be incremented and published as metrics");
            _histogram = Metrics.CreateHistogram("TestMetricHistogram", "Will observe a value and publish it as Histogram");
            _gauge = Metrics.CreateGauge("TestMetricGauge", "Will observe a value and publish it as Gauge");
            _summary = Metrics.CreateSummary("TestMetricSummary", "Will observe a value and publish it as Summary");
        }


        [HttpPost("IncrementCounter")]
        public void PostCounter(int inc)
        {
            _counter.Inc(inc);
            _counter.Publish();        
        }

        [HttpPost("IncrementHistogram")]
        public void PostHistogram(int inc)
        {
           _histogram.Observe(inc);
           _histogram.Publish();
        }

        [HttpPost("IncrementGauge")]
        public void IncrementGauge(int inc)
        {
            _gauge.Inc(inc);
            _gauge.Publish();
        }

        [HttpPost("DecrementGauge")]
        public void DecrementGauge(int inc)
        {
            _gauge.Dec(inc);
            _gauge.Publish();
        }

        [HttpPost("IncrementSummary")]
        public void PostSummary(int inc)
        {
            _summary.Observe(inc);
            _summary.Publish();
        }       
    }
}

Now you can run your API and then type after your port number /metrics. You can see some generated logs by the Prometheus exporter that you have in your solution.

Prometheus exporter

Step 5. Download and configure Prometheus

Once you download Prometheus, you'll have to the following folder.

Prometheus

Open the file prometheus .yml then add the section related to your service. By default, you'll find the Prometheus job.

Prometheus is in fact observing its own activity. So you can add your service configuration. In our case, the file looks as follows.

# my global config
global:
  scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
          # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "prometheus"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:9090"]
      
        # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: "PrometheusServiceTest"

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
      - targets: ["localhost:5174"]

Once finished, launch the file prometheus.exe, and the Prometheus server will run on the port 9090.

If your configuration is ok, the command should look like this.

Command

Step 6. Launch the service and perform some operations

Now, launch the Web Api and perform some operations using the controller, like incrementing the counter and the gauge.

As said previously, generally the metrics are related to the internal processing of the API, but here we are passing the value in a POST just for test purpose.

Post

Step 7. Visualize the data in Prometheus

Now, you can open Prometheus in your browser using the URL localhost:9090.

then go to Status => targets

Targets

Here, you can find the entry related to your Service and the last scrape time.

Scrape time

in the home screen of Prometheus, you can look for the item that you introduced in the code and check its evolution.

 _counter = Metrics.CreateCounter("TestMetricCounter", "will be incremented and published as metrics");

We can check, for example the Counter graph, TestMetricCounter.

TestMetricCounter

Now we've seen how to check the metrics in Prometheus, let's see how to explore them using another tool, which is Grafana.

Step 8. Download and configure Grafana

In this article, I'm using the version v10.4.1 of Grafana. When you install this version, you will automatically have a server that runs on port 3000.

Ensure also that your Prometheus server is running (prometheus.exe in step 5)

So follow the steps of the installation from the link, and then type in your browser localhost:3000 and you should have the following UI.

Grafana

Go to the connection ==> Data sources.

Data sources

then choose Add new data source.

Add new data

You will have a list of predefined data sources; choose Prometheus, then configure it through the following UI (you need to write explicitly http:// in the url)

Then, finally, scroll down and save, and you should have it added to your data sources list.

Data sources list

Step 9. Check the metrics in Grafana

Now click on the dashboard

On the right top of the screen, choose New ==> choose to Add Visualisation ==> and here you need to choose the Prometheus data source.

Now perform step 6 by generating some metrics from the Web API, let's test the gauge this time.

Configure your dashboard with the metrics of the gauge and you will get them displayed.

The Metric should be the name that you have set in your code.

 _gauge = Metrics.CreateGauge("TestMetricGauge", "Will observe a value and publish it as Gauge");

The label Filters should be Job = the name of your job that you have set in the Prometheus config file : ( - job_name: "PrometheusServiceTest" in the YAML)

PrometheusServiceTest

Here, we have the curve by default related to our metric defined in the code.

Grafana gives also the possibility to import a dashboard using its ID or JSON, which you can get from this link: https://grafana.com/grafana/dashboards/.

We can start testing this one: https://grafana.com/grafana/dashboards/6566-rabbitmq-perftest/ Copy its ID using the button in the UI. It should be 6566.

Now go to your Grafana UI and Import it using the Button New ==> Import, you will get the following UI So Add the ID and then Load.

Grafana UI

If there is no error with the compliance of the dashboard you should get this.

Options

You will have a Global dashboard with several empty panels, and we need to configure them to get value.

Now we will configure just the Histogram Panel since our service manages Histogram in the code.

Histogram Panel

Now we should configure it in the same way as other dashboards by selecting the metric and the label filters (this time the histogram metrics).

Metric

Now run the queries then perform some calls from the swagger (using the histogram POST), you should get some data to show.

Swagger

Click Apply, then go to your main Dashboard, and you will find your Histogram configured. You need to give Prometheus the time to perform a scrape (the configured delay in the YAML file)

Here, it's just an example, but in real life, you need to choose the right dashboard ID where you need most of the metrics, or you can also remove unneeded panels.

And finally, don't forget to save your dashboards.

Unneeded panels

Conclusion

Here we saw another way to manage the telemetry of a net core Web API which is a very recommended concept to enhance the visibility of your services.