Emotion Recognition In Xamarin.Forms With Microsoft Cognitive Services

Introduction

Microsoft Cognitive Services offer awesome APIs and services for developers to create more intelligent applications. You can add interesting features, like people's emotions and video detection, face, speech, and vision recognition and speech and language understanding into your application.

The Emotion API takes a facial expression in an image stream as an input, and returns confidence levels across a set of emotions for each face in the image, like anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise, in a facial expression
Microsoft Cognitive Services
Where we can implement it

There is no single unifying definition or theory of emotion. Nevertheless, several characteristics of an emotional reaction are frequently mentioned. Nowadays, feedback is more important for product promotion and review of the article and store, etc. Manual entry of emotion maybe wrong, so we automate the feedback and review.

Microsoft Cognitive Services

In this article, we will learn how to implement Emotion Recognition in Xamarin.Forms with Microsoft Cognitive Services.

Getting Started with the Emotion API

The Emotion API can detect anger, contempt, disgust, fear, happiness, neutral, sadness, and surprise, as a facial expression, as well as, it returns an emotion result for a facial expression. The Emotion API also returns a bounding box for detected faces using the Face API. If a user has already called the Face API, he/she can submit the face rectangle as an optional input. Note that an API key must be obtained to use the Emotion API.

You can follow the below steps for obtaining a free API key on Microsoft.com

Step 1

Click here for generating Subscriber Key, and click on "Create".

Microsoft Cognitive Services
Step 2

Agree to the terms and Microsoft privacy statement,  select the country, and click on "Next".

Microsoft Cognitive Services

Step 3

Login with Microsoft /LinkedIn /Facebook /GitHub for creating your application.

Microsoft Cognitive Services

Step 4

You should see the keys and the below information available for your subscriptions.

Microsoft Cognitive Services
We will use Emotion API key in the following Xamarin application. Before that, let's test the API key.

Test your Emotion Key

Navigate to the Emotion API online console application and provide the subscription key and image url. If your provided key is valid, you will get its JSon data as output or relevant error.

Microsoft Cognitive Services

Create New Xamarin.Forms Application

You can refer to my previous article for setting up and creating new Xamarin.Forms application.

If you have already installed VS on your machine, open Visual Studio >> Create New Xamarin Forms application like below.

Microsoft Cognitive Services

Solution will be created with all the platform and PCL projects.

Microsoft Cognitive Services

In this solution, we are going to create demo application for Article review automate based on the emotion.

Camera in Xamarin. Forms

We can use Media.Plugin Nuget package for Capture image or select image gallery from all the mobile platform (iOS,Android,Windows)

Right Click on Solution > Type “xam.plugin” in the search box > Select all the project (PCL, iOS, Android and UWP) > Click on Install

Microsoft Cognitive Services

Emotion Nuget Package

This client library is a thin C# client wrapper for the Microsoft Emotion API.The easiest way to use this client library is to get microsoft.projectoxford.emotion package from nuget .

Right Click on Solution > Type “Microsoft Emotion” in the search box > Select all the project (PCL, iOS, Android and UWP) > Click on Install

Microsoft Cognitive Services

UI Design

After successfully install above two nuget package. Let start UI design from PCL project

In PCL Project > Open MainPage.Xaml page and design following xaml code

  1.     <?xml version="1.0" encoding="utf-8" ?>  
  2.     <ContentPage xmlns="http://xamarin.com/schemas/2014/forms"  
  3.                  xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"  
  4.                  xmlns:local="clr-namespace:DevEnvExeEmotion"  
  5.                  x:Class="DevEnvExeEmotion.MainPage">  
  6.         <StackLayout Margin="20">  
  7.             <StackLayout Orientation="Horizontal" Padding="30" HorizontalOptions="CenterAndExpand" >  
  8.                 <StackLayout x:Name="emotion">  
  9.                 <Label Text="Reader is :"  FontSize="Large"/>  
  10.                 <Label x:Name="emotionResultLabel" FontSize="Large" />  
  11.                 </StackLayout>  
  12.                 <Button Text="Share Your FeedBack" Clicked="OnFeedBackClicked"></Button>  
  13.             </StackLayout>  
  14.             <Image x:Name="image" Source="article.png" />  
  15.             <ActivityIndicator x:Name="activityIndicator" />  
  16.         </StackLayout>  
  17.       
  18.     </ContentPage>  
  19. In mainPage.xaml.cs file add your logic.   
  20.     public partial class MainPage : ContentPage  
  21.         {  
  22.             EmotionServiceClient emotionClient;  
  23.             MediaFile photo;  
  24.       
  25.             public MainPage()  
  26.             {  
  27.                 InitializeComponent();  
  28.                 emotion.IsVisible = false;  
  29.                 emotionClient = new EmotionServiceClient("7f495be5faf643adbeead444d5b79a13");  
  30.             }  
  31. For capture image, write the following code   
  32.          public async Task CaptureImage()  
  33.             {  
  34.                 await CrossMedia.Current.Initialize();  
  35.                 emotion.IsVisible = false;  
  36.                 // Take photo  
  37.                 if (CrossMedia.Current.IsCameraAvailable || CrossMedia.Current.IsTakePhotoSupported)  
  38.                 {  
  39.                     photo = await CrossMedia.Current.TakePhotoAsync(new StoreCameraMediaOptions  
  40.                     {  
  41.                         SaveToAlbum = false  
  42.                           
  43.                     });  
  44.       
  45.                 }  
  46.                 else  
  47.                 {  
  48.                     await DisplayAlert("No Camera""Camera unavailable.""OK");  
  49.                 }  
  50.             }  
  51. For Recognize emotion, write the following code   
  52.     public async Task Recognizeemotion()  
  53.             {  
  54.                 try  
  55.                 {  
  56.                     if (photo != null)  
  57.                     {  
  58.                         using (var photoStream = photo.GetStream())  
  59.                         {  
  60.       
  61.                             Emotion[] emotionResult = await emotionClient.RecognizeAsync(photoStream);  
  62.                             if (emotionResult.Any())  
  63.                             {  
  64.                                 // Emotions detected are happiness, sadness, surprise, anger, fear, contempt, disgust, or neutral.  
  65.                                 emotionResultLabel.Text = emotionResult.FirstOrDefault().Scores.ToRankedList().FirstOrDefault().Key;  
  66.                                 emotion.IsVisible = true;  
  67.                             }  
  68.                             photo.Dispose();  
  69.                         }  
  70.                     }  
  71.                 }  
  72.                 catch (Exception ex)  
  73.                 {  
  74.                     Debug.WriteLine(ex.Message);  
  75.                 }  
  76.             }  
  77. While click on Feedback button, call the above two methods   
  78.       async void OnFeedBackClicked(object sender, EventArgs e)  
  79.             {  
  80.                 //capture image  
  81.                 await CaptureImage();  
  82.                 ((Button)sender).IsEnabled = false;  
  83.                 // Recognize emotion  
  84.                 await Recognizeemotion();  
  85.                 ((Button)sender).IsEnabled = true;  
  86.             }  

Run the Application

Select the iOS /Android /Windows and install into the device and click on ‘Share your Feedback ‘button



The device should be camera so after click on the button, capture your face for Recognize emotion.

Microsoft Cognitive Services
Summary

In this article, you learned how to generate Emotion API key and implement Emotion Recognition in Xamarin.Forms with Microsoft Cognitive Services.

If you have any questions/ feedback/ issues, please write in the comment box.