Image Analysis In ASP.NET Core 5.0 Using Azure Cognitive Service

Background

Azure’s computer vision services give a wide range of options to do image analysis. These AI services enable you to discover the content and analyze images and videos in real time. Using computer vision, which is a part of Azure cognitive services, we can do image processing to label content with objects, moderate content, identify objects in image, extract text, generate details descriptions of the image, orientations, people movements and so on.

The word “cognitive” means mental action or the process of acquiring knowledge and understanding through thoughts, experience, and the senses. (Copied from Wikipedia). Thinking, Visualizing, Knowing, Learning, Judging, and discovering, deciding etc. are the actions of cognitive.

Source code

Azure Cognitive Services

Azure Cognitive Services are cloud-based AI services which are developed by Microsoft to enable cognitive intelligence into any application through REST APIs or client library SDKs. This means, we can build AI applications without knowing Artificial intelligence and Data science algorithms or skills. In another way, these services are ready-made AI services to use in applications of any platform or language and enable AI and cognitive features in your solutions. There are various cognitive services which are mainly divided into five categories: vision, speech, language, decision, and search. More details here.

Computer Vision

Computer vison, which is part of the Azure cognitive service vision category, provides advanced processing of images and videos with advance algorithms to extract text, do OCR, visual features from image like objects, faces, adult content, and auto-generate image descriptions. Additionally, this cognitive service analyzes videos and identify the movement and presence of object.

In this article, I will do image analyzes using computer vision in asp.net core solution.

Features of Image analysis of Computer vision

Detect Object: Using this service, we can identify the object in the image. The service will return the bounding box coordinate of each object. The detection operation returns the coordinates of the objects like animals and person from the image which can further extend to create relation among those objects.

Tagging
This service can identify and tag visual features like scenes, living things, furniture, tools, gadgets, and so on.
Face Detection
This service can identify and detect the face and some properties like age, gender, face gestures from image.
Detect Brands
There are numerous commercial brands and logs are stored in the database and this service can detect famous brands.
Image Description
This service returns the comprehensive description of an entire image in natural language. It evaluates the image and provides complete information.
Color Scheme detection
this service detects image color scheme like black/white or color image, also provide information on the accent and dominant colors.
Image categorization
This service can be used for image categorization using category taxonomy to entire image.
Image Type detection
This is another feature of this service that can detect the characteristics of images. For example, the image is a clip art or line drawing.
Thumbnail Generation
This cognitive service creates high-quality thumbnail based on the analysis of the image and area of interest. It crops the image based on area of interest and generates the thumbnail of different aspect ratio compared to the original image.
Domain-Specific content detection
Using this service, we can identify the domain of image content like celebrities and landmarks.
Area of Interest
This feature returns the area of interest of the image by analyzing the contents. This will give the bounding box coordinates of the area of interest.
Detect Moderate Content
This service can be used to detect the adult content in an image. This feature can be useful to know the confidence of moderate content and set a threshold for flagging.

Note

Image categorization and tagging is only supported in English language.

Now we will create a real project to implementing image analysis using computer vision.

Prerequisites

Azure subscription (you can create trial one for free)
Visual Studio (I am using Visual Studio 2019, however, it will be same for other versions as well)
.NET 5.0 (ASP.NET core 5.0)

Note

I will be doing this tutorial with asp.net core 5.0

Create Computer Vision Cognitive Service in Azure

Let us first create Azure Computer vision API service from Azure Portal.

We can search with Cognitive service or computer vision as illustrated below. In this demonstration, we will be using computer vision, an azure cognitive service.

Create a new cognitive service, then it will display the marketplace. Simply we can search computer vision and select it.

Then, we will get option to create this service.

Create computer vision by filling the necessary information as portrayed.

There is free tier as well for development which you will get in pricing tier, however, in my case, I have already utilized free in another demo. Therefore, I will choose standard one.

Subsequently, we will get option to choose virtual network option, based on your convenience, select one option.

Finally, we will review and create as shown:

Free tier has following quota,

With free tier, we can do 5000 calls per 30 days.

Most importantly, we need to get subscription key and endpoint for accessing this computer vision service/API. We can navigate to keys and endpoint tab get the required information as illustrated.

To run the service from asp.net core, we basically need this two information: Key (also called subscription key) and Endpoint URL. Therefore, we can keep this in some note or can login again in azure portal and collect the information.

Then again, let us create asp.net core 5.0 project to implement computer vision.

Implementing Image analysis using Client library in asp.net core 5.0

Quickly, we will create a new asp.net core 5.0 razor pages project with Visual Studio 2019.

Open Visual studio >> Create a new Project.

Please Note: Here I am selecting ASP.NET Core with Razor pages (not with MVC).

Then we will get option for project name and location. You can give any name and choose location.

Then, we will get option to select the target framework. Here, we will select .NET 5.0 which is a current version.

We have successfully created the ASP.NET core 5.0 web application with Razor pages.

Once the solution is created, then quickly, you can build and run the solution.

We need to add client library from NuGet packages for accessing computer vision service.

Microsoft.Azure.CognitiveServices.Vision.ComputerVision

We can add this package by right clicking on the project, selecting Manage NuGet Packages and browser with the above name as depicted.

After installing NuGet package, I will create service for computer vision. Create a folder name Service and add Interface and classes as shown.

We will add Dto (data transfer object) for image analysis result.

ImageAnalysisViewModel.cs

using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
namespace computervision.aspcore.Services.Dto
{
public class ImageAnalysisViewModel
{
public ImageAnalysis imageAnalysisResult { get; set; }
}
}

We will follow the folder/files structure as shown above. Then, we have added interface for computer vision service.

IComputerVisionService.cs

using computervision.aspcore.Services.Dto;
using System.IO;
using System.Threading.Tasks;
namespace computervision.aspcore.Services
{
public interface ICompuerVisionService
{
Task<ImageAnalysisViewModel> AnalyzeImageUrl(string imageUrl);
}
}

Then, in service class, we will first do authentication of client using key and endpoint URL. This service class is implementation of the above Interface.

Authentication code for client.

// Add your Computer Vision subscription key and endpoint
private string subscriptionKey = "PASTE_YOUR_COMPUTER_VISION_SUBSCRIPTION_KEY_HERE";
private string endpoint = " PASTE_YOUR_COMPUTER_VISION_ENDPOINT_HERE";
/*
* AUTHENTICATE
* Creates a Computer Vision client used by each example.
*/
public ComputerVisionClient Authenticate() {
ComputerVisionClient client = new ComputerVisionClient(new ApiKeyServiceClientCredentials(subscriptionKey)) {
Endpoint = endpoint
};
return client;
}

We have these subscription key and endpoints URL from Azure portal as shown in earlier steps.

Subsequently, we will use the client authentication to analyze the image. The following code will be used for analyzing the image.

Function for Analyzing the image from URL as shown,

public async Task < ImageAnalysisViewModel > AnalyzeImageUrl(string imageUrl) {
try {
// Creating a list that defines the features to be extracted from the image.
ComputerVisionClient client = Authenticate();
List < VisualFeatureTypes ? > features = new List < VisualFeatureTypes ? > () {
VisualFeatureTypes.Categories, VisualFeatureTypes.Description,
VisualFeatureTypes.Faces, VisualFeatureTypes.ImageType,
VisualFeatureTypes.Tags, VisualFeatureTypes.Adult,
VisualFeatureTypes.Color, VisualFeatureTypes.Brands,
VisualFeatureTypes.Objects
};
ImageAnalysis results;
using(Stream imageStream = File.OpenRead(imageUrl)) {
results = await client.AnalyzeImageInStreamAsync(imageStream, visualFeatures: features);
//imageStream.Close();
}
ImageAnalysisViewModel imageAnalysis = new ImageAnalysisViewModel();
imageAnalysis.imageAnalysisResult = results;
return imageAnalysis;
} catch (System.Exception ex) {
// add your log capture code
throw;
}
}

Let me explain the above function. Firstly, I am doing authentication, then defining the feature which will be sent to analysis service of computer vision as shown,

ComputerVisionClient client = Authenticate();
List < VisualFeatureTypes ? > features = new List < VisualFeatureTypes ? > () {
VisualFeatureTypes.Categories, VisualFeatureTypes.Description,
VisualFeatureTypes.Faces, VisualFeatureTypes.ImageType,
VisualFeatureTypes.Tags, VisualFeatureTypes.Adult,
VisualFeatureTypes.Color, VisualFeatureTypes.Brands,
VisualFeatureTypes.Objects
};

In this above visual feature, I am selecting categories, descriptions, faces, Image type, tags, adult, color, brands, and objects. In other words, the response of the client analysis will return the above features using computer vision cognitive services.

Complete code of service class, CompterVisionService.cs

using computervision.aspcore.Services.Dto;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;
using System.Collections.Generic;
using System.IO;
using System.Threading.Tasks;
namespace computervision.aspcore.Services {
public class CompterVisionService: ICompuerVisionService {
// Add your Computer Vision subscription key and endpoint
private string subscriptionKey = "05b6ca4476c447eeb8c841e16119d101";
private string endpoint = "https://autotagingdemo.cognitiveservices.azure.com/";
/*
* AUTHENTICATE
* Creates a Computer Vision client used by each example.
*/
public ComputerVisionClient Authenticate() {
ComputerVisionClient client = new ComputerVisionClient(new ApiKeyServiceClientCredentials(subscriptionKey)) {
Endpoint = endpoint
};
return client;
}
public async Task < ImageAnalysisViewModel > AnalyzeImageUrl(string imageUrl) {
try {
// Creating a list that defines the features to be extracted from the image.
ComputerVisionClient client = Authenticate();
List < VisualFeatureTypes ? > features = new List < VisualFeatureTypes ? > () {
VisualFeatureTypes.Categories, VisualFeatureTypes.Description,
VisualFeatureTypes.Faces, VisualFeatureTypes.ImageType,
VisualFeatureTypes.Tags, VisualFeatureTypes.Adult,
VisualFeatureTypes.Color, VisualFeatureTypes.Brands,
VisualFeatureTypes.Objects
};
ImageAnalysis results;
using(Stream imageStream = File.OpenRead(imageUrl)) {
results = await client.AnalyzeImageInStreamAsync(imageStream, visualFeatures: features);
//imageStream.Close();
}
ImageAnalysisViewModel imageAnalysis = new ImageAnalysisViewModel();
imageAnalysis.imageAnalysisResult = results;
return imageAnalysis;
} catch (System.Exception ex) {
// add your log capture code
throw;
}
}
}
}

This is how we have completed the image analysis function for computer vision cognitive services.

Now, I will create file upload functionality to upload file and send to computer vision service for analyzing it.

To upload files in asp.net core please check this link

Upload Single or Multiple Files in ASP.NET Core Razor Pages with Insights

We will add file upload model index.cshtml.cs for file upload as shown,

public class FileUpload {
[Required]
[Display(Name = "File")]
public IFormFile FormFile {
get;
set;
}
public string SuccessMessage {
get;
set;
}
}

Then, we will do dependency injection of ComputerVisionService through interface in Index.cshtml.cs page constructor.

private readonly ILogger < IndexModel > _logger;
private readonly ICompuerVisionService _compuerVisionService;
private readonly IWebHostEnvironment _hostEnvironment;
public IndexModel(ILogger < IndexModel > logger, IWebHostEnvironment hostEnvironment) {
_logger = logger;
_compuerVisionService = new CompterVisionService();
this._hostEnvironment = hostEnvironment;
}

Eventually, we will use this service in upload method to upload and do image analysis same time. The upload method is as shown below,

public async Task < IActionResult > OnPostUpload(FileUpload fileUpload) {
string fullPath = _hostEnvironment.WebRootPath + "/UploadImages/";
if (!Directory.Exists(fullPath)) {
Directory.CreateDirectory(fullPath);
}
var formFile = fileUpload.FormFile;
if (formFile.Length > 0) {
var filePath = Path.Combine(fullPath, formFile.FileName);
ViewData["ImageUrl"] = formFile.FileName;
using(var stream = new FileStream(filePath, FileMode.Create)) {
await formFile.CopyToAsync(stream);
}
// using service to analyze the image
var imageAnalysis = await _compuerVisionService.AnalyzeImageUrl(filePath);
if (imageAnalysis.imageAnalysisResult != null) ViewData["ImageAnalysisViewModel"] = imageAnalysis;
}
ViewData["SuccessMessage"] = fileUpload.FormFile.FileName.ToString() + " file uploaded!!";
return Page();
}

Let me explain this method. This method is straight forward, first, I am creating folder for uploading the file. Subsequently, I am using method AnalyzeImageUrl from CompterVisionService which will return analysis result using computer vision cognitive service, explained earlier.

Here the complete code for index.cshtml.cs

using computervision.aspcore.Services;
using Microsoft.AspNetCore.Hosting;
using Microsoft.AspNetCore.Http;
using Microsoft.AspNetCore.Mvc;
using Microsoft.AspNetCore.Mvc.RazorPages;
using Microsoft.Extensions.Logging;
using System.ComponentModel.DataAnnotations;
using System.IO;
using System.Threading.Tasks;
namespace computervision.aspcore.Pages
{
public class IndexModel : PageModel
{
private readonly ILogger<IndexModel> _logger;
private readonly ICompuerVisionService _compuerVisionService;
private readonly IWebHostEnvironment _hostEnvironment;
public IndexModel(ILogger<IndexModel> logger, IWebHostEnvironment hostEnvironment)
{
_logger = logger;
_compuerVisionService = new CompterVisionService();
this._hostEnvironment = hostEnvironment;
}
[BindProperty]
public FileUpload fileUpload { get; set; }
public void OnGet()
{
ViewData["SuccessMessage"] = "";
}
public async Task<IActionResult> OnPostUpload(FileUpload fileUpload)
{
string fullPath = _hostEnvironment.WebRootPath + "/UploadImages/";
if (!Directory.Exists(fullPath))
{
Directory.CreateDirectory(fullPath);
}
var formFile = fileUpload.FormFile;
if (formFile.Length > 0)
{
var filePath = Path.Combine(fullPath, formFile.FileName);
ViewData["ImageUrl"] = formFile.FileName;
using (var stream = new FileStream(filePath, FileMode.Create))
{
await formFile.CopyToAsync(stream);
}
// using service to analyze the image
var imageAnalysis =await _compuerVisionService.AnalyzeImageUrl(filePath);
if (imageAnalysis.imageAnalysisResult != null)
ViewData["ImageAnalysisViewModel"] = imageAnalysis;
}
ViewData["SuccessMessage"] = fileUpload.FormFile.FileName.ToString() + " file uploaded!!";
return Page();
}
public class FileUpload
{
[Required]
[Display(Name = "File")]
public IFormFile FormFile { get; set; }
public string SuccessMessage { get; set; }
}
}
}

Similarly, the complete code of razor page, index.cshtml

@page
@model IndexModel
@using computervision.aspcore.Services.Dto;
@{
ViewData["Title"] = "Home page";
}
<div class="text-center">
<h4>Rijsat.com | Image Analysis using cognive service in asp.net core 5.0</h4>
<p>Learn about <a href="https://docs.microsoft.com/aspnet/core">building Web apps with ASP.NET Core</a>.</p>
</div>
<form enctype="multipart/form-data" method="post">
@*File Upload part*@
<div class="form-group">
<label class="file">
<input type="file" asp-for="fileUpload.FormFile" multiple aria-label="File browser example">
<span class="file-custom"></span>
</label>
<input asp-page-handler="Upload" class="btn btn-primary" type="submit" value="Upload">
</div>
@{
if (ViewData["SuccessMessage"] != null)
{
<span class="badge badge-success"> @ViewData["SuccessMessage"]</span>
}
}
@*Displaying Analysis results*@
<div class="row">
<div class="col-sm-6">
@if (ViewData["ImageUrl"] != null)
{
var imageUrl = Convert.ToString(ViewData["ImageUrl"]);
<img src="UploadImages/@imageUrl" alt="rijsat.com" class="img-thumbnail" />
}
</div>
<div class="col-sm-6 small">
<p>
@if (ViewData["ImageAnalysisViewModel"] != null)
{
var imageAnalysisView = (ImageAnalysisViewModel)ViewData["ImageAnalysisViewModel"];
<h5>Description</h5>
foreach (var caption in imageAnalysisView.imageAnalysisResult.Description.Captions)
{
<span class="text-info"> @caption.Text: with confidence @caption.Confidence</span><br />
}
<h5>Category:</h5>
foreach (var category in imageAnalysisView.imageAnalysisResult.Categories)
{
<span class="text-success"> @category.Name with confidence: @category.Score</span><br />
}
<h5>Tags</h5>
foreach (var tag in imageAnalysisView.imageAnalysisResult.Tags)
{
<span class="badge badge-success">@tag.Name - @tag.Confidence</span>
}
<h5>Object Detection</h5>
foreach (var obj in imageAnalysisView.imageAnalysisResult.Objects)
{
<span class="badge badge-warning">@obj.ObjectProperty with Confidence @obj.Confidence</span>
}
<h5>Face Detection</h5>
foreach (var face in imageAnalysisView.imageAnalysisResult.Faces)
{
<span class="badge badge-info">
A @face.Gender of age @face.Age at location @face.FaceRectangle.Left, @face.FaceRectangle.Top, @face.FaceRectangle.Width, @face.FaceRectangle.Top, @face.FaceRectangle.Height
</span>
}
<h5>Image color scheme</h5>
<span class="badge badge-secondary">Is black and white?:@imageAnalysisView.imageAnalysisResult.Color.IsBWImg</span>
<span class="badge badge-secondary">Accent color:@imageAnalysisView.imageAnalysisResult.Color.AccentColor</span>
<span class="badge badge-secondary">Dominant background color:@imageAnalysisView.imageAnalysisResult.Color.DominantColorBackground</span>
<span class="badge badge-secondary">Dominant foreground color:@imageAnalysisView.imageAnalysisResult.Color.DominantColorForeground</span>
<h5>Image Type</h5>
<span class="badge badge-secondary">Clip Art Type: @imageAnalysisView.imageAnalysisResult.ImageType.ClipArtType</span>
<span class="badge badge-secondary">Line Drawing Type: @imageAnalysisView.imageAnalysisResult.ImageType.LineDrawingType</span>
}
</p>
</div>
</div>
</form>

Let me explain this razor page. There are mainly two parts, file upload and analysis results which I have commented in the page.

Now, our image analysis solution is ready to use. Azure cognitive service has made it comfortable to develop AI solutions without having deep knowledge on Artificial Intelligence or machine learning. Using these services, we can include AI features in our applications effortlessly. Additionally, there are numerous other services that can be implemented in any applications regardless of framework, platform, or language to build AI features.

Since our application is ready to do check with some images and analyze with results.

I have uploaded the complete solution to GitHub, please download and use it.

Source Code

Let me run this solution.

Now, I will select a picture and do analysis.

Let me elaborate on the analysis result.

In my AnalyzeImageUrl method of computer vision service, I have requested following features:

Categories
Description
Faces
Image Type
Tags
Adult
Color
Brands
Objects

This is done by defining the features parameters as defined in that method,

List < VisualFeatureTypes ? > features = new List < VisualFeatureTypes ? > () {
VisualFeatureTypes.Categories, VisualFeatureTypes.Description,
VisualFeatureTypes.Faces, VisualFeatureTypes.ImageType,
VisualFeatureTypes.Tags, VisualFeatureTypes.Adult,
VisualFeatureTypes.Color, VisualFeatureTypes.Brands,
VisualFeatureTypes.Objects
};

Based on these features, the analysis result of my solution,

Every finding in this response is with confidence factor which shows the percentage of confidence of data or output.

Now, let try the analysis of another complex image.

The analysis of this is also done with many useful information and tags.

This is how we can do image analysis in any application using computer vision.

Source Code

Conclusion

Including AI features in any application has become quite simple using azure cognitive services without having much knowledge of AI or machine learning. In this article, I have explained image analysis with its feature and computer vision service. Furthermore, I have portrayed how to create a cognitive service, computer vision from the Azure portal. Additionally, I have implemented the service in asp.net core 5.0 using visual studio with examples. Finally, I have done some image analysis and explored the analysis results.

Source Code

2 Comments

Join the conversation! Your thoughts help the community grow.

Narayan Nyaupane5y ago
Great work. Thank you so much. I have a few requests for you. I want to build an application analysis of medical images. Using this service we only analyze general-purpose images. Can you please help me using custom vision like this.
- Rijwan Ansari5y ago
  Sure. Please let me know more details for medical images analysis. You can share some samples and also expected outcomes.