Logging With ElasticSearch, Kibana, Serilog Using ASP.NET Core Docker

Configure logging

The next step is to configure the logging-in program. cs.

using System;
using System.Reflection;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.Hosting;
using Serilog;
using Serilog.Sinks.Elasticsearch;

namespace ElasticKibanaLoggingVerify
{
    public class Program
    {
        public static void Main(string[] args)
        {
            var environment = Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT");
            var configuration = new ConfigurationBuilder()
                .AddJsonFile("appsettings.json", optional: false, reloadOnChange: true)
                .AddJsonFile(
                    $"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}.json",
                    optional: true)
                .Build();
            Log.Logger = new LoggerConfiguration()
                .Enrich.FromLogContext()
                .WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(configuration["ElasticConfiguration:Uri"]))
                {
                    AutoRegisterTemplate = true,
                    IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower()}-{DateTime.UtcNow:yyyy-MM}"
                })
                .Enrich.WithProperty("Environment", environment)
                .ReadFrom.Configuration(configuration)
                .CreateLogger();
            CreateHostBuilder(args).Build().Run();
        }

        public static IHostBuilder CreateHostBuilder(string[] args) =>
            Host.CreateDefaultBuilder(args)
                .ConfigureWebHostDefaults(webBuilder =>
                {
                    webBuilder.UseStartup<Startup>();
                }).ConfigureAppConfiguration(configuration =>
                {
                    configuration.AddJsonFile("appsettings.json", optional: false, reloadOnChange: true);
                    configuration.AddJsonFile(
                        $"appsettings.{Environment.GetEnvironmentVariable("ASPNETCORE_ENVIRONMENT")}.json",
                        optional: true);
                })
                .UseSerilog();
    }
}

In the previous article on Serilog, we have seen how important the enrichment and the SinkOptions are.

You can register the ElasticSearch sink in code as follows.

WriteTo.Elasticsearch(new ElasticsearchSinkOptions(new Uri(configuration["ElasticConfiguration:Uri"]))
{
    AutoRegisterTemplate = true,
    IndexFormat = $"{Assembly.GetExecutingAssembly().GetName().Name.ToLower()}-{DateTime.UtcNow:yyyy-MM}"
})

Create a controller to validate the behavior

You can create a controller to verify the logging details in Kibana.

[Route("api/[controller]")]
public class ElasticSearchController : Controller
{ 
    private readonly ILogger<ElasticSearchController> _logger;
    public ElasticSearchController(ILogger<ElasticSearchController> logger)
    {
        _logger = logger;
    }

    // GET: api/values
    [HttpGet]
    public int GetRandomvalue()
    {
        var random = new Random();
        var randomValue = random.Next(0, 100);
        _logger.LogInformation($"Random Value is {randomValue}");
        return randomValue;
    }
}

The above controller is self-explanatory and generates random values between 0 to 100. Thereafter, I'm logging the random value using.

_logger.LogInformation($"Random Value is {randomValue}");

Start logging events to ElasticSearch and configure Kibana

Now, run the Web API application by clicking on F5 and navigating to https://localhost:5001/api/ElasticSearch

Now, let's configure an index in Kibana.

After creating an index, you can filter the message by using.

message: "59"

Logging error to ElasticSearch

Let's add a new HTTP GET method in the ElasticSearch Controller.

[HttpGet("{id}")]
public string ThrowErrorMessage(int id)
{
    try
    {
        if (id <= 0)
            throw new Exception($"id cannot be less than or equal to 0. Value passed is {id}");
        return id.ToString();
    }
    catch (Exception ex)
    {
        _logger.LogError(ex, ex.Message);
    }
    return string.Empty;
}

The above code is quite straightforward and self-explanatory. Let's test the ThrowErrorMessage method by passing id equals to 0.

You can narrow it down to an error log using.

{
  "level": "error"
}

You can filter further by configuring multiple conditions.

{
    "message": "value passed",
    "level": "error"
}

Conclusion

Earlier, setting up the logging was quite a tedious job and thereafter, getting the flat file from servers and identifying/searching for the error was another pain.

With Docker in place, setting up the environment is effortless, and especially with the MicroService architecture, it's much easier to create the ElasticSearch indexes and the data that has been logged can be visualized using Kibana.

Logging can be even more powerful when you perform the same setup in Kubernetes/Azure Kubernetes Service orchestrator with multiple nodes.

There's really no excuse for developers/architects to not incorporate logging using ElasticSearch, Kibana, and Docker.

Cheers!

I hope you like the article. In case, you find the article interesting then kindly like and share it.