Post
Ask Question

Image Classification In ASP.NET Core Using ML.NET

Problem

 
Image classification is a common case in many business scenarios. For these cases, you can either use pre-trained models or train your own model to classify images specific to your custom domain.
 

DataSet

 
There are two data sources: the tsv file and the image files. The tsv file contains two columns: the first one is defined as ImagePath and the second one is the Label corresponding to the image. As you can observe, the file does not have a header row, and looks like this:
  1. broccoli.jpg    broccoli  
  2. broccoli.png    broccoli  
  3. canoe2.jpg  canoe  
  4. canoe3.jpg  canoe  
  5. canoe4.jpg  canoe  
  6. coffeepot.jpg   coffeepot  
  7. coffeepot2.jpg  coffeepot  
  8. coffeepot3.jpg  coffeepot  
  9. coffeepot4.jpg  coffeepot  
  10. pizza.jpg   pizza  
  11. pizza2.jpg  pizza  
  12. pizza3.jpg  pizza  
  13. teddy1.jpg  teddy bear  
  14. teddy2.jpg  teddy bear  
  15. teddy3.jpg  teddy bear  
  16. teddy4.jpg  teddy bear  
  17. teddy6.jpg  teddy bear  
  18. toaster.jpg toaster  
  19. toaster2.png    toaster  
  20. toaster3.jpg    toaster  

The training and testing images are located in the assets folders. These images belong to Wikimedia Commons.

  • Wikimedia Commons, the free media repository. Retrieved 10:48, October 17, 2018 from:
  • https://commons.wikimedia.org/wiki/Pizza
  • https://commons.wikimedia.org/wiki/Coffee_pot
  • https://commons.wikimedia.org/wiki/Toaster
  • https://commons.wikimedia.org/wiki/Category:Canoes
  • https://commons.wikimedia.org/wiki/Teddy_bear

Pre-trained model

 
There are multiple models which are pre-trained for classifying images. In this case, we will use a model based on an Inception topology, and trained with images from Image.Net. This model can be downloaded from here, but it's also available here.
 

Solution

 
The class library project ImageClassification.Score can be used to classify sample images based on the pre-trained Inception-5h TensorFlow model.

Again, note that this sample only uses/consumes a pre-trained TensorFlow model with ML.NET API. Therefore, it does not train any ML.NET model. Currently, TensorFlow is only supported in ML.NET for scoring/predicting with existing TensorFlow trained models.
 

Code Walkthrough

 
There are two projects in the solution -ImageClassification.Score is responsible for loading the model in TensorFlow format, and then classify images; hile ImageClassification.Web is responsible for consuming the ImageClassification.Score project to get the image from the user and show prediction results to the user.
 

ML.NET: Model Scoring 

 
The first step is to load the data using TextLoader,
 
vardata=mlContext.Data.ReadFromTextFile<ImageNetData>(dataLocation, hasHeader: true);

The image file used to load images has two columns: the first one is defined as ImagePath and the second one is the Labelcorresponding to the image.

It is important to highlight that the label in the ImageNetData class is not really used when scoring with the TensorFlow model. It is used when testing the predictions so you can compare the actual label of each sample data with the predicted label provided by the TensorFlow model.

  1. broccoli.jpg    broccoli  
  2. bucket.png    bucket   
  3. canoe.jpg    canoe   
  4. snail.jpg    snail   
  5. teddy1.jpg    teddy bear   

As you can observe, the file does not have a header row.

The Inception model has several default parameters you need to pass in.

  1. public struct ImageNetSettings {
  2.      public const int imageHeight = 224;
  3.      public const int imageWidth = 224;
  4.      public const float mean = 117;
  5.      public const bool channelsLast = true;
  6. }   

The second step is to define the estimator pipeline. Usually, when dealing with deep neural networks, you must adapt the images to the format expected by the network. This is the reason images are resized and then transformed (mainly, pixel values are normalized across all R,G,B channels).

  1. var pipeline = mlContext.Transforms.LoadImages(outputColumnName: "input", imageFolder: imagesFolder, inputColumnName: nameof(ImageNetData.ImagePath))  
  2.     .Append(mlContext.Transforms.ResizeImages(outputColumnName: "input", imageWidth: ImageNetSettings.imageWidth, imageHeight: ImageNetSettings.imageHeight, inputColumnName: "input"))  
  3.     .Append(mlContext.Transforms.ExtractPixels(outputColumnName: "input", interleavePixelColors: ImageNetSettings.channelsLast, offsetImage: ImageNetSettings.mean))  
  4.     .Append(mlContext.Model.LoadTensorFlowModel(modelLocation)  
  5.         .ScoreTensorFlowModel(outputColumnNames: new[] { "softmax2" }, inputColumnNames: new[] { "input" },  
  6.             addBatchDimensionInput:true));  

You also need to check the neural network and check the names of the input/output nodes. In order to inspect the model, you can use tools like Netron, which is automatically installed with Visual Studio Tools for AI. These names are used later in the definition of the estimation pipe: in the case of the inception network, the input tensor is named 'input' and the output is named 'softmax2'

Image Classification In ASP.NET Core Using ML.NET

Finally, we extract the prediction engine after fitting the estimator pipeline. The prediction engine receives as a parameter an object of type ImageNetData (containing 2 properties: ImagePath and Label) and then returns an object of type ImagePrediction.

  1. ITransformer model = pipeline.Fit(data);     
  2. var predictionEngine = mlContext.Model.CreatePredictionEngine<ImageNetData, ImageNetPrediction>(model);          
When obtaining the prediction, we get an array of floats in the property PredictedLabels. Each position in the array is assigned to a label, so for example, if the model has 5 different labels, the array will be length = 5. Each position in the array represents the label's probability in that position; the sum of all array values (probabilities) is equal to one. Then, you need to select the biggest value (probability), and check which is the assigned label to that position.
 

ImageClassification.Score

 
Download this from here and include this project into your Visual Studio solution.
 

ImageClassification.Web

 
Let's start to build ASP.NET Core application which will consume the Tensor Flow pre-trained model using ML.NET API. 
  • Open Visual Studio>New Projects>ASP.NET Core
  • Set Project name as ImageClassification.Web
  • Click Create button>MVC Web Application
  • A new Project will be added to your solution.
  • Open Nuget Manager Console and write Install-Package Microsoft.ML
  • Right-click on the project and Add refrence>Select ImageClassification.Score>Press Ok
  • Download assets from here and include in project
  • Here is how your project will look,

    Image Classification In ASP.NET Core Using ML.NET

  • Now we need to consume the model. Create a new Controller or Open HomeController and add the following snippet.
    1. using System;  
    2. using System.Collections.Generic;  
    3. using System.Diagnostics;  
    4. using System.Linq;  
    5. using System.Threading.Tasks;  
    6. using Microsoft.AspNetCore.Mvc;  
    7. using ImageClassification.Web.Models;  
    8. using System.IO;  
    9. using ImageClassification.ModelScorer;  
    10. using Microsoft.AspNetCore.Http;  
    11. using ImageClassification.ImageDataStructures;  
    12.   
    13. namespace ImageClassification.Web.Controllers  
    14. {  
    15.     public class HomeController : Controller  
    16.     {  
    17.         public IActionResult Index()  
    18.         {  
    19.             return View();  
    20.         }  
    21.         [HttpPost]  
    22.         public async Task<IActionResult> Index(IFormFile image)  
    23.         {  
    24.             string assetsRelativePath = @"../../../assets";  
    25.             string assetsPath = GetAbsolutePath(assetsRelativePath);  
    26.   
    27.             var tagsTsv = Path.Combine(assetsPath, "inputs", "images", "tags.tsv");  
    28.             var imagesFolder = Path.Combine(assetsPath, "inputs", "images");  
    29.             //Path of Tensor Flow pre trained model
    30.             var inceptionPb = Path.Combine(assetsPath, "inputs", "inception", "tensorflow_inception_graph.pb");  
    31.             var labelsTxt = Path.Combine(assetsPath, "inputs", "inception", "imagenet_comp_graph_label_strings.txt");  
    32.   
    33.             try  
    34.             {  
    35.                 var filePath = Path.GetTempFileName();  
    36.                 //Save image to temp
    37.                 using(var ms=new FileStream(filePath,FileMode.Create))  
    38.                 {  
    39.                     await image.CopyToAsync(ms);  
    40.                 }
    41.                 //Create Tensor Flow model for image classification and prediction using ML.NET API
    42.                 var modelScorer = new TFModelScorer(tagsTsv, imagesFolder, inceptionPb, labelsTxt);  
    43.                 //Check image prediction
    44.                 var prediction=(ImageNetDataProbability)modelScorer.Score(new ImageNetData() { ImagePath=filePath,Label=""});
    45.                 //Store results to ViewBag to show in View  
    46.                 ViewBag.PredictedLabel = prediction.PredictedLabel;  
    47.                 ViewBag.Probability = prediction.Probability;  
    48.             }  
    49.             catch  
    50.             {  
    51.                 return View();  
    52.             }  
    53.             return View();  
    54.         }  
    55.         public static string GetAbsolutePath(string relativePath)  
    56.         {  
    57.             FileInfo _dataRoot = new FileInfo(typeof(Program).Assembly.Location);  
    58.             string assemblyFolderPath = _dataRoot.Directory.FullName;  
    59.             string fullPath = Path.Combine(assemblyFolderPath, relativePath);  
    60.             return fullPath;  
    61.         }  
    62.     }  
    63. }  
  • Now we need to create an index view which will be used to upload image to server and show image classification or prediction result from server. Please add following snippet to your view.
    1. @{  
    2.     ViewData["Title"] = "Home Page";  
    3. }  
    4. <h2>Image Classification in ASP.NET Core Using ML.NET with TensorFlow</h2>  
    5. <hr />  
    6. <div class="row">  
    7.     <div class="col-md-6">  
    8.         <form method="post" enctype="multipart/form-data">  
    9.             <input type="file" name="image" />  
    10.             <button class="btn btn-sm btn-primary" type="submit">Classify</button>  
    11.         </form>  
    12.     </div>  
    13.     <div class="col-md-6">  
    14.         @if (ViewBag.Probability != null)  
    15.         {  
    16.             <h3>Prediction</h3>  
    17.             <h6>Predicted Label: @ViewBag.PredictedLabel</h6>  
    18.             <h6>Probability: @ViewBag.Probability</h6>  
    19.         }  
    20.   
    21.     </div>  
    22. </div>  
  • Now try to build and run the ImageClassification.Web application

Demo

 
This sample ASP.NET Core application takes image form user and classify them and show the result to the user. 
 
Image Classification In ASP.NET Core Using ML.NET
 

Conclusion

 
So, in this article, we tried to build an ASP.NET Core MVC Web Application which consumes Tensor Flow pre-trained model for Image Classification or Image prediction. In this article, we used ML.NET API to consume the Tensor Flow Deep Learning Model.
 
Note
Please don't forget to add the Tensor Flow pre-trained model here into asset>inputs>inceptions folder. The source code doesn't include this due to huge size.
 
For more detailed information please see here.
 
You can also download project source code from here.


Similar Articles
Finchship
We Provide Web, Desktop and Mobile Apps Solution