Introduction
This blog explains a simple and tested approach on how to integrate .NET Core API with AWS Cloudwatch service using serilog sinks and send log messages directly to the AWS Cloudwatch service where it can be viewed with filter capabilities and troubleshoot your development or production environment problems easily. This blog also gives a bonus approach to capture custom log messages from anywhere in your application irrespective of LogLevel.
Prerequisite
To be able to follow through this blog, you need to have basic knowledge of the following things:
-
Basics of .NET Core API and how it works
-
Basic Knowledge about AWS services
-
Basics of Serilog Logging framework and how it works
Why Do We Need This?
In web application development, almost all developers (or most) would have faced this problem at least once in their experience "Urgh! Code works fine in my local machine but breaks when deployed to Testing/Prod Server". To handle that scenario, we use logging frameworks across our application code to log all important/critical steps with multiple levels such as Trace, Debug, Info, Warning, Error & Fatal. With the use of logging, we can easily troubleshoot/detect why/where our application code breaks and solve the problem immediately. Serilog is such a framework with a variety of logging capabilities.
Instead of storing logs to your database or file system and managing it on your own, we are going one step ahead since AWS provides a cloud based service called Cloudwatch to collect all your application logs and display it in Console UI with filter enabled capabilities. In the current business model, web applications are becoming smart by using more cloud computing services rather building/managing on their own.
In certain cases, we want to log all the actions happening around a critical functionality of our application all the time, for example payment gateway or third party service calling module. Due to this scenario, either you need to flood your AWS cloudwatch storage with information logs OR log INFO level details in the name of ERROR level. Neither way is correct. This blog illustrates an approach to solve this problem by enabling custom logging feature and capture only required details with correct log level.
Okay, enough theory. Let's implement it with a step by step approach.
Create a .NET Core API application (using Visual Studio 2017 or higher version)
This is just an easy and well-known step to start. From Visual Studio, create a new .Net Core Web application and select API with Controllers type which will create a runnable API project with controller having CRUD HTTP methods. This is enough to demonstrate the scope of this blog but in real time, you need to apply the following steps in your existing Core API service.
Add necessary NuGet Libraries
Following are the required Nuget packages which should be added in order to store logs into AWS cloudwatch.
-
AWSSDK.Core - This SDK establishes connection to AWS services and makes the difficult and time consuming deployment process into an easy task.
-
Serilog - This library provides all the necessary functions to capture logs from your application and write it in the desired target.
-
Serilog.Sinks.AwsCloudwatch - This library helps us to create a connection between .net core application and AWS cloudwatch service
AWS SDK Installation
Once AWS SDK library is installed in a .Net core application, you will be able to open AWS Toolkit under View menu. After opening it for the first time, you need to provide your AWS account access key details to login to your account. This will create a default profile which will be used for all your deployments or access to direct AWS services automatically.

If you want to create and use a different profile other than default, you can easily do that from AWS Toolkit and set AWS_PROFILE environment variable to your profile name. Setting AWS_PROFILE environment variable to your profile lets AWS SDK pick up required profile information from .NET credentials file stored in AWS SDK and connect it to AWS service. Please refer to this link for more information here.
Setup Logger for .NET Core API
Once AWS account setup is done through SDK, we need to add required configurations in Startup.cs file to start capturing log information and to let .NET Core API know that we are going to send log messages to AWS Cloudwatch. Add the below lines of code to your Startup.cs file.
- public Startup(IConfiguration configuration)
- {
- Configuration = configuration;
- int.TryParse(Configuration.GetSection("Serilog").GetSection("LogLevel").Value, out int level);
- int.TryParse(Configuration.GetSection("Serilog").GetSection("RententionPolicy").Value, out int retentionConfig);
- var logLevel = typeof(LogEventLevel).IsEnumDefined(level) ?
- (LogEventLevel)level : LogEventLevel.Error;
- var retentionPolicy = typeof(LogGroupRetentionPolicy).IsEnumDefined(retentionConfig) ?
- (LogGroupRetentionPolicy)retentionConfig : LogGroupRetentionPolicy.OneWeek;
- var region = RegionEndpoint.GetBySystemName(Configuration.GetSection("Serilog").GetSection("Region").Value);
- var levelSwitch = new LoggingLevelSwitch();
- levelSwitch.MinimumLevel = logLevel;
- // customer formatter
- var formatter = new CustomLogFormatter();
- var options = new CloudWatchSinkOptions
- {
- // the name of the CloudWatch Log group from config
- LogGroupName = Configuration.GetSection("Serilog").GetSection("LogGroup").Value,
- // the main formatter of the log event
- TextFormatter = formatter,
- // other defaults
- MinimumLogEventLevel = logLevel,
- BatchSizeLimit = 100,
- QueueSizeLimit = 10000,
- Period = TimeSpan.FromSeconds(10),
- CreateLogGroup = true,
- LogStreamNameProvider = new DefaultLogStreamProvider(),
- RetryAttempts = 5,
- LogGroupRetentionPolicy = retentionPolicy
- };
- // setup AWS CloudWatch client
- var client = new AmazonCloudWatchLogsClient(region);
- // Attach the sink to the logger configuration
- Log.Logger = new LoggerConfiguration()
- .WriteTo.Logger(l1 => l1
- .MinimumLevel.ControlledBy(levelSwitch)
- .WriteTo.AmazonCloudWatch(options, client))
- .CreateLogger();
- }


Join the conversation! Your thoughts help the community grow.