Benchmark Your Code Like dotNetDave!

Benchmarking is the process of measuring and baselining the performance of your code. It helps identify bottlenecks in comparing the performance of different algorithms or approaches that target the same set of problems and choosing the one that has optimal time and memory consumption. There are many ways to code the same thing in .NET, so how do you know which one is more performant? There can be big differences that not only affect performance but memory too. I recently did a poll asking developers if they benchmark their code. The majority said they do not or do not know what benchmarking is. This article is to help get you jumpstarted in your benchmark journey! Setting up your benchmark tests might take time, but they are worth it. I have found many issues in benchmark tests that do not show up in unit tests. Unit tests are run once while benchmark tests can run millions of times for a single test.

Sadly, I have never worked for a company that benchmarked their code, unless I wrote it on my own time. Benchmarking the code that you put in the cloud is critically important since, for most of the services, you are charged for the length of time that the code is executing.

Benchmarking of code should always be done before it’s released!

For many years for the work for my code performance book and OSS assemblies, I have been using BenchmarkDotNet. I highly recommend using this NuGet package. Microsoft even uses it to benchmark .NET! In previously released versions of Spargine, I have had an assembly to make it easier to use BenchmarkDotNet, but for my work moving Spargine to .NET 7, I have made many changes to these classes to make it even easier to benchmark your code, more efficiently. The information in this article is from DotNetTips.Spargine.Benchmarking assembly and NuGet package. The code and NuGet packages can be found below.

GitHub: https://bit.ly/SpargineCode
NuGet: http://bit.ly/dotNetDaveNuGet

Benchmarking Your Code with Spargine

To view the benchmark reports for Spargine, go here: https://bit.ly/Spargine6BenchmarkReports

Overview

The DotNetTips.Spargine.Benchmarking assembly makes setting up BenchmarkDotNet easier and features classes and methods that pre-generate real-world data that you can use in your benchmark tests. First, let me show you what a benchmark test method looks like for AppendBytes() from the StringBuilder.

[Benchmark(Description = nameof(StringBuilderExtensions.AppendBytes))]
public void AppendBytes() 
{
    var sb = new StringBuilder();
    sb.AppendBytes(this.GetByteArray(1));
    this.Consume(sb.ToString());
}

The required attribute for a benchmark test is [Benchmark]. I always add a description since it will be used in the reports. In much of my code, I also use [BenchmarkCategory] which is also used in the reports and allows you to run a suite of tests for a given category. The output of this test will look something like this,

Benchmarking Your Code with Spargine

All the columns above are configured in the DotNetTips.Spargine.Benchmarking.Benchmark class. Below are the main classes that make up this assembly.

Benchmarking Your Code with Spargine

Benchmark.cs

The Benchmark abstract class is the core type to run benchmark tests. This is where the information in the reports is set up along with many helper methods constants and properties. All the data is either constants or loaded when the class is created so it does not interfere with the benchmark test timing.

Constants

Below are the constants defined that I use for many of my benchmark tests.

LowerCaseString	ProperCaseString	String10Characters01
String10Characters02	String15Characters01	String15Characters02
TestEmailLowerCase	TestEmailMixedCase	UpperCaseString

Properties

Here are the properties that are preloaded with data.

Base64String	Returns a base 64 string.
Coordinate01 & Coordinate02	Returns DotNetTips.Spargine.Tester.Models.ValueTypes.Coordinate for use in comparison benchmark tests.
CoordinateProper01 & CoordinateProper02	Returns DotNetTips.Spargine.Tester.Models.ValueTypes.CoordinateProper for use in comparison benchmark tests.
JsonTestDataPersonProper	Returns the JSON string for DotNetTips.Spargine.Tester.Models.RefTypes.PersonProper.
JsonTestDataPersonRecord	Returns the JSON string for DotNetTips.Spargine.Tester.Models.RefTypes.PersonRecord.
LanuchDebugger	Set this to true if you want to launch the BenchmarkDotNet debugger.
LongTestString	Returns a very long string that can be used for benchmark tests for RegEx and more.
PersonProperRef01 & PersonProperRef02	Returns DotNetTips.Spargine.Tester.Models.RefTypes.PersonProper for use in comparison benchmark tests.
PersonRecord01 & PersonRecord02	Returns DotNetTips.Spargine.Tester.Models.RefTypes.PersonRecord for use in comparison benchmark tests.
PersonRef01 & PersonRef02	Returns DotNetTips.Spargine.Tester.Models.RefTypes.Person for use in comparison benchmark tests.
PersonVal01 & PersonVal02	Returns DotNetTips.Spargine.Tester.Models.ValueTypes.Person for use in comparison benchmark tests.
StringEmpty	Returns string.Empty.
StringNull	Returns a null string.
StringToTrim	Returns the same text as LongTestString, with added spaces before and after for tests for string.Trim.
TestGuid	Returns a GUID.
XmlTestDataPersonProper	Returns the XML string for DotNetTips.Spargine.Tester.Models.RefTypes.PersonProper.
XmlTestDataPersonRecord	Returns the XML string for DotNetTips.Spargine.Tester.Models.RefTypes.PersonRecord.

Methods

Here is a list of the methods for Benchmark with descriptions.

Cleanup (virtual)	Code to clean up the data. Override this method to run the cleanup after the benchmark test is completed.
Consume<T>	Use to simulate the use of a type. I add this to the end of every benchmark test.
ConsumeAsync<T>	Same as Consume<T> for async benchmark test.
GetByteArray()	Returns a byte array in the size requested (in Kb).
GetStringArray()	Returns a string array of words using a specified count and the minimum and maximum length of the words can be set.
GlobalCleanup	This is called when the benchmark tests are completed.
GlobalSetup	This is called before the benchmark tests are run. It preloads data for the properties.
Setup (virtual)	This method preloads the data for Benchmark. Call this method from your benchmark classes before any setup that you do.

Benchmark is the core type used throughout this assembly and I use it to benchmark the code in Spargine as shown below.

public class TypeHelperBenchmark: Benchmark

When running your benchmark tests using this library, they will run better as Admin.

Setting Up Reports

There are many ways to set up the reports that BenchmarkDotNet will create. This is done in the Spargine Benchmark class. Benchmark will set up the output of most of the common reports that include HTML, GitHub markdown, JSON, RP plots, and more. Furthermore, it will set up columns for the reports like the Max, Minimum, Namespace, and more. Here is a list of all the attributes used to set this up in Benchmark.

[AllStatisticsColumn]	[AsciiDocExporter]	[Atlassian]
[BaselineColumn]	[CategoriesColumn]	[ConfidenceIntervalErrorColumn]
[CsvExporter]	[CsvMeasurementsExporter]	[Default]
[DisassemblyDiagnoser]	[EvaluateOverhead]	[Full]
[GcServer(true)]	[GitHub]	[HtmlExporter]
[IterationsColumn]	[JsonExporter]	[KurtosisColumn]
[LogicalGroupColumn]	[MarkdownExporter]	[MaxColumn]
[MemoryDiagnoser]	[MinColumn]	[MValueColumn]
[NamespaceColumn]	[Orderer(SummaryOrderPolicy.Method)]	[PlainExporter]
[RankColumn]	[RPlotExporter]	[SkewnessColumn]
[StackOverflow]	[StatisticalTestColumn]

The reports I use the most are HTML and GitHub markdown. For more information on these attributes, check out the documentation here: https://benchmarkdotnet.org/articles/overview.html

CollectionsBenchmark.cs

The CollectionsBenchmark class holds and loads a few different collection types that can be used for benchmarking. The collection types use the models from DotNetTips.Spargine.Tester assembly. While working on the latest edition of my code performance book, I noticed that some of the time was from the generation of collections or other values. The changes I have made ensure that this is minimized as much as possible by pre-generating the collections and then returning a clone of it.

Constructor

The constructor requires a maxCount that is used when loading the collections. This number is also used for the MaxCount property.

Properties

MaxCount

This is used to set the max count of the number of items returned in a collection.

Methods

Below are the methods and their descriptions.

GetCoordinateProperValArray()	Returns a pre-generated array of DotNetTips.Spargine.Tester.Models.ValueTypes.CoordinateProper based on the value of the Count property.
GetCoordinateProperValList()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.ValueTypes.CoordinateProper based on the value of the Count property.
GetCoordinateValArray()	Returns a pre-generated array of DotNetTips.Spargine.Tester.Models.ValueTypes.Coordinate based on the value of the Count property.
GetCoordinateValList()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.ValueTypes.Coordinate based on the value of the Count property.
GetPeopleRefToInsert()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.RefTypes.PersonProper for use in collection add or insert methods.
GetPeopleValToInsert()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.ValTypes.PersonProper for use in collection add or insert methods.
GetPersonProperRefArray()	Returns a pre-generated array of DotNetTips.Spargine.Tester.Models.RefTypes.PersonProper based on the value of the Count property.
GetPersonProperRefDictionary()	Returns a pre-generated dictionary of DotNetTips.Spargine.Tester.Models.RefTypes.PersonProper based on the value of the Count property.
GetPersonProperRefList()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.RefTypes.PersonProper based on the value of the Count property.
GetPersonRecordArray()	Returns a pre-generated array of DotNetTips.Spargine.Tester.Models.RefTypes.PersonRecord based on the value of the Count property.
GetPersonRecordList()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.RefTypes.PersonRecord based on the value of the Count property.
GetPersonRefArray()	Returns a pre-generated array of DotNetTips.Spargine.Tester.Models.RefTypes.Person based on the value of the Count property.
GetPersonRefList()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.RefTypes.Person based on the value of the Count property.
GetPersonValArray()	Returns a pre-generated array of DotNetTips.Spargine.Tester.Models.ValTypes.Person based on the value of the Count property.
GetPersonValList()	Returns a pre-generated collection of DotNetTips.Spargine.Tester.Models.ValTypes.Person based on the value of the Count property.
LoadCoordinateCollections(), LoadCoordinateProperCollections(), LoadPersonCollections(), LoadPersonProperCollections(), LoadPersonRecordCollections()	These methods are called from Setup() to pre-generate collections.
Setup()	This method pre-loads the collections. It’s called by the other benchmark classes.

All the methods that generate an ICoordinate, IPerson, or PersonRecord collection, return a clone of the pre-generated collection. If you want just half of the collection, use CollectionSize.Half when calling the method.

LargeCollectionBenchmark.cs

This is the main type (inherits CollectionsBenchmark) that I use whenever I need to benchmark code using collections. It will run benchmarks with the following collection counts: 10, 25, 50, 100, 250, 500, 1000, and 2500. I use this class with many of the benchmark tests for Spargine as shown below.

public class ListExtensionsCollectionBenchmark: LargeCollectionBenchmark

SmallCounterBenchmark.cs

This class (inherits CollectionsBenchmark) sets up running a benchmark test with the following collection counts: 2, 5, 10, 20, 25, 50, 75, 100, and 250. I use it for some of the tests for Spargine as shown below.

public class StringBuilderHelperCounterBenchmark: SmallCounterBenchmark

Examples

Here are just a few examples of how I use this assembly when benchmarking code for Spargine and my code performance book. In most of the benchmarking tests that I write I use the Consumer class in BenchmarkDotNet that will consume the type that you are testing, giving a more realistic result.

[Benchmark(Description = nameof(XmlSerialization.Deserialize)]
[BenchmarkCategory(Categories.XML)]
public void Deserialize01() 
{
    var result = XmlSerialization.Deserialize < PersonProper > (base.XmlTestDataPersonProper);
    this.Consume(result);
}
    
[Benchmark(Description = "WriteAsync")]
[BenchmarkCategory(Categories.Async)]
public async Task WriteAsync() 
{
    var channel = new ChannelQueue < PersonProper > ();
    var people = this.GetPersonProperRefArray();
    
    for (var peopleCount = 0; peopleCount < people.Length; peopleCount++) 
    {
        await channel.WriteAsync(people[peopleCount]).ConfigureAwait(false);
    }
    
    this.Consume(channel.Count);
}
        
[Benchmark(Description = "foreach()")]
[Benchmark(Description = nameof(StringBuilderExtensions.AppendKeyValue))]
public void AppendKeyValue1() 
{
    var sb = new StringBuilder();
    var stringArray = base.GetStringArray(count: 10, wordMinLength: 15, wordMaxLength: 20);

    for (var index = 0; index < stringArray.Length; index++) 
    {
       var testString = stringArray[index];
       sb.AppendKeyValue(testString, testString);
    }

    this.Consume(sb.ToString());
}

Creating Your Own Benchmark Classes

Creating your own benchmark classes with Spargine is easy. Simply inherit the Benchmark type for tests that do not test collections as shown below.

public class GeneralBenchmark: Benchmark

If setup or cleanup is needed, ensure to overload the Setup() or Cleanup() methods as shown below.

public override void Setup() 
{
    base.Setup();
    this._personList = this.GetPersonProperRefList();
}

public override void Cleanup() 
{
    base.Cleanup();
    DirectoryHelper.DeleteDirectory(this._tempPath, retries: 5);
    DirectoryHelper.DeleteDirectory(this._sourcePath, retries: 5);
}

Make sure to call base.Setup() or base.Cleanup() before you add your code.

If you need to benchmark collections, then create your type by inheriting either LargeCollectionsBenchmark or SmallCollectionsBenchmark as shown below.

public class ListExtensionsCollectionBenchmark: LargeCollectionsBenchmark

Setup() and Cleanup() can also be overridden as shown above.

These base classes will automatically set up all the reports, columns, and more. All you need to do is run your benchmark. Here is how I set it up in my benchmark projects.

var config = DefaultConfig.Instance
    .AddJob(Job.Default.WithRuntime(CoreRuntime.Core70))
    .AddJob(Job.Default.WithRuntime(CoreRuntime.Core60));
 config = config.WithOption(ConfigOptions.DisableOptimizationsValidator, true)
    .WithOption(ConfigOptions.StopOnFirstError, true);
_ = BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).RunAll(config);

Summary

If you use BenchmarkDotNet, I hope you will find this assembly useful. If you would like anything added, please do a pull request, or submit an issue. If you have any comments or suggestions, please make them below. To see how I benchmark Spargine, go to: https://bit.ly/Spargine6BenchmarkTests.

Happy benchmarking!