Getting Started With Bing Emotion API

In this tutorial, we will learn how to use Bing Emotion API. In our previous example of the cognitive service demo, we had combined two cognitive services, Bing Speech API and Bing Emotion API, so as to find images using speech to text (STT). To extend this demo, we will be using Bing Emotion API which will detect the emotions of a person from the image that we had searched using Bing Image Search API. I would highly recommend you to go through the previous two parts before starting with this demo.

To register for Bing Emotion API, follow these steps.

  1. Navigate to https://azure.microsoft.com/en-us/try/cognitive-services/
  2. Navigate to the "Vision" tab and select Bing Emotion API. Then, click "Get API Key".

    Azure

  3. Select "I agree" and your region, as shown in the below image.

    Azure

  4. Login with Microsoft account to get your API key.

    Azure

  5. Save your key.

    Azure

Now, from Part 2, we already have the URL of the searched image using Bing Image Search API. So, we will use the same URL to get the emotion of a person using Bing Emotion API. We will simply create one method named as GetEmotion which takes String URL of the image which we already have from part 2.

  1. private async Task GetEmotion(string imageUri)  
  2.         {  
  3.             var client = new HttpClient();  
  4.             var queryString = HttpUtility.ParseQueryString(string.Empty);  
  5.   
  6.             // Request headers  
  7.             client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key""Your_key_Bing_Emotion_API");  
  8.   
  9.             // Request parameters  
  10.             var uri = "https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize?" + queryString;  
  11.             EmotionRequest request = new EmotionRequest();  
  12.             request.url = imageUri;  
  13.             byte[] byteData = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(request));  
  14.   
  15.             using (var content = new ByteArrayContent(byteData))  
  16.             {  
  17.                 content.Headers.ContentType = new MediaTypeHeaderValue("application/json");  
  18.                 var response = await client.PostAsync(uri, content);  
  19.                 var json = await response.Content.ReadAsStringAsync();  
  20.                 if (response.IsSuccessStatusCode)  
  21.                 {  
  22.                     List<EmotionResponse> emotionResponse =  
  23.                         JsonConvert.DeserializeObject<List<EmotionResponse>>(json);  
  24.   
  25.                     if (emotionResponse != null && emotionResponse.Count > 0)  
  26.                     {  
  27.                         var scores = emotionResponse[0].scores;  
  28.                         Dictionary<string, double> dScores = new Dictionary<string, double>();  
  29.                         dScores.Add("anger", scores.anger);  
  30.                         dScores.Add("contempt", scores.contempt);  
  31.                         dScores.Add("disgust", scores.disgust);  
  32.                         dScores.Add("fear", scores.fear);  
  33.                         dScores.Add("happiness", scores.happiness);  
  34.                         dScores.Add("neutral", scores.neutral);  
  35.                         dScores.Add("sadness", scores.sadness);  
  36.                         dScores.Add("surprise", scores.surprise);  
  37.                         var highestScore = dScores.Values.OrderByDescending(score => score).First();  
  38.                         //probably a more elegant way to do this.  
  39.                         var highestEmotion = dScores.Keys.First(key => dScores[key] == highestScore);  
  40.   
  41.                         await Application.Current.Dispatcher.BeginInvoke(  
  42.                             DispatcherPriority.Normal,  
  43.                             new Action(  
  44.                                 () =>  
  45.                                 {  
  46.                                     this.MySpeechSentiment.Text = $"Emotion: {highestEmotion},";  
  47.                                     this.MySpeechSentimentConfidence.Text =  
  48.                                         $"confidence: {Convert.ToInt16(highestScore * 100)}%";  
  49.                                 }));  
  50.                           
  51.                     }  
  52.                     else  
  53.                     {  
  54.                         this.MySpeechSentiment.Text = $"I'm not able to get the emotion, sorry.";  
  55.                     }  
  56.                 }  
  57.                 else  
  58.                 {  
  59.                     await Application.Current.Dispatcher.BeginInvoke(  
  60.                         DispatcherPriority.Normal,  
  61.                         new Action(() => { this.MySpeechSentiment.Text = "Could not get emotion from this image";  
  62.                         }));  
  63.                       
  64.                 }  
  65.             }  
  66.   
  67.         }  

This Method will make POST call to Bing Emotion API with required parameters and API will return the JSON result. To store this JSON result, we will create one class and name it EmotionResponse. Again, it is optional; one can directly bind the JSON result.

First, download the part 2 from here, then follow the steps.

  1. Replace the design of MainWindow.xaml with the following code.
    1. <Grid>  
    2.         <Grid.RowDefinitions>  
    3.             <RowDefinition Height="Auto"></RowDefinition>  
    4.             <RowDefinition Height="Auto"></RowDefinition>  
    5.             <RowDefinition Height="Auto"></RowDefinition>  
    6.             <RowDefinition Height="Auto"></RowDefinition>  
    7.             <RowDefinition Height="*"></RowDefinition>  
    8.         </Grid.RowDefinitions>  
    9.         <Grid.ColumnDefinitions>  
    10.             <ColumnDefinition Width="Auto"></ColumnDefinition>  
    11.             <ColumnDefinition Width="*"></ColumnDefinition>  
    12.             <ColumnDefinition Width="3*"></ColumnDefinition>  
    13.         </Grid.ColumnDefinitions>  
    14.         <TextBlock FontSize="24" Grid.Row="0" Grid.Column="0" Grid.ColumnSpan="3" Margin="4">Bing Speech,Image Search and Emotion Demo</TextBlock>  
    15.         <Button x:Name="button" HorizontalAlignment="Center" VerticalAlignment="Center" Width="75" Click="button_Click"  
    16.                 Grid.Row="1" Grid.Column="0" Grid.RowSpan="3" Margin="4" Height="75">  
    17.             <Button.Content>  
    18.                 <StackPanel Orientation="Vertical">  
    19.                     <TextBlock FontSize="16">Speak</TextBlock>  
    20.                 </StackPanel>  
    21.             </Button.Content>  
    22.         </Button>  
    23.   
    24.         <TextBlock x:Name="status" TextWrapping="Wrap" Text="Not Listening" VerticalAlignment="Center" FontSize="16" Visibility="Collapsed"  
    25.                    Grid.Row="1" Grid.Column="1" HorizontalAlignment="Center" Margin="4"/>  
    26.   
    27.         <StackPanel Orientation="Horizontal" Grid.Row="1" Grid.Column="1" Grid.ColumnSpan="2">  
    28.             <TextBlock x:Name="MySpeechResponse" FontSize="20" Margin="4" TextWrapping="Wrap" VerticalAlignment="Center" />  
    29.             <TextBlock x:Name="MySpeechResponseConfidence" FontSize="12" Margin="4" TextWrapping="Wrap" VerticalAlignment="Center" />  
    30.         </StackPanel>  
    31.         <StackPanel Grid.Row="3" Grid.Column="1" Grid.ColumnSpan="2" Orientation="Horizontal">  
    32.             <TextBlock x:Name="MySpeechSentiment" Margin="4" FontSize="16" TextWrapping="Wrap" VerticalAlignment="Center" />  
    33.             <TextBlock x:Name="MySpeechSentimentConfidence" FontSize="12" Margin="4" TextWrapping="Wrap" VerticalAlignment="Center" />  
    34.         </StackPanel>  
    35.   
    36.         <Image x:Name="searchImage" Margin="4" Stretch="Uniform"  
    37.                Grid.Row="4" Grid.Column="0" Grid.ColumnSpan="3"/>  
    38.   
    39.         <StackPanel Orientation="Vertical" HorizontalAlignment="Center" VerticalAlignment="Center" x:Name="RecordingBar" Visibility="Collapsed"  
    40.                     Grid.Row="0" Grid.Column="0" Grid.RowSpan="5" Grid.ColumnSpan="3">  
    41.   
    42.             <ProgressBar HorizontalAlignment="Left" Width="207" Margin="0,16,0,0" IsIndeterminate="True" />  
    43.         </StackPanel>  
    44.   
    45.     </Grid>  
  1. Replace the code of MainWindow.xaml.cs with the code given below.
    1. namespace BingSpeechImageEmotionDemo  
    2. {  
    3.     using Newtonsoft.Json;  
    4.     using System.Windows.Threading;  
    5.     using System.Net.Http;  
    6.     using System.Net.Http.Headers;  
    7.     using System.Web;  
    8.     using Microsoft.CognitiveServices.SpeechRecognition;  
    9.      
    10.       
    11.   
    12.     /// <summary>  
    13.     /// Interaction logic for MainWindow.xaml  
    14.     /// </summary>  
    15.     ///   
    16.     public partial class MainWindow : Window  
    17.     {  
    18.         private MicrophoneRecognitionClient micClient;  
    19.         public MainWindow()  
    20.         {  
    21.             InitializeComponent();  
    22.             this.micClient = SpeechRecognitionServiceFactory.CreateMicrophoneClient(  
    23.                 SpeechRecognitionMode.ShortPhrase,"en-US""Your_Key_Bing_Speech_API");  
    24.             this.micClient.OnMicrophoneStatus += MicClient_OnMicrophoneStatus;  
    25.             this.micClient.OnResponseReceived += MicClient_OnResponseReceived;  
    26.         }  
    27.   
    28.   
    29.         private void MicClient_OnMicrophoneStatus(object sender, MicrophoneEventArgs e)  
    30.         {  
    31.             Application.Current.Dispatcher.BeginInvoke(  
    32.                 DispatcherPriority.Normal,  
    33.                 new Action(  
    34.                     () =>  
    35.                     {  
    36.                         if (e.Recording)  
    37.                         {  
    38.                             this.status.Text = "Listening";  
    39.                             this.RecordingBar.Visibility = Visibility.Visible;  
    40.                         }  
    41.                         else  
    42.                         {  
    43.                             this.status.Text = "Not Listening";  
    44.                             this.RecordingBar.Visibility = Visibility.Collapsed;  
    45.                         }  
    46.   
    47.                     }));  
    48.         }  
    49.         private async void MicClient_OnResponseReceived(object sender, SpeechResponseEventArgs e)  
    50.         {  
    51.             if(e.PhraseResponse.Results.Length>0)  
    52.             {  
    53.   
    54.                await Application.Current.Dispatcher.BeginInvoke(DispatcherPriority.Normal,  
    55.                      
    56.                    new Action(  
    57.                        () =>  
    58.                        {  
    59.                            this.MySpeechResponse.Text = $"'{e.PhraseResponse.Results[0].DisplayText}',";  
    60.                            this.MySpeechResponseConfidence.Text = $"confidence: {e.PhraseResponse.Results[0].Confidence}";  
    61.   
    62.                        }));  
    63.                 this.SearchImage(e.PhraseResponse.Results[0].DisplayText);  
    64.             }  
    65.               
    66.         }  
    67.         private async void SearchImage(string phraseToSearch)  
    68.         {  
    69.             var client = new HttpClient();  
    70.             var queryString = HttpUtility.ParseQueryString(string.Empty);  
    71.   
    72.             // Request headers  
    73.             client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key""Your_Key_Bing_Search_API");  
    74.   
    75.             // Request parameters  
    76.             queryString["q"] = phraseToSearch;  
    77.             queryString["count"] = "1";  
    78.             queryString["offset"] = "0";  
    79.             queryString["mkt"] = "en-us";  
    80.             queryString["safeSearch"] = "Moderate";  
    81.             var uri = "https://api.cognitive.microsoft.com/bing/v5.0/images/search?" + queryString;  
    82.   
    83.             var response = await client.GetAsync(uri);  
    84.             var json = await response.Content.ReadAsStringAsync();  
    85.             // MessageBox.Show(json.ToString());  
    86.             BingImageSearchResponse bingImageSearchResponse = JsonConvert.DeserializeObject<BingImageSearchResponse>(json);  
    87.             var uriSource = new Uri(bingImageSearchResponse.value[0].contentUrl, UriKind.Absolute);  
    88.   
    89.             await Application.Current.Dispatcher.BeginInvoke(  
    90.                 DispatcherPriority.Normal, new Action(() =>  
    91.                 {  
    92.                     this.searchImage.Source = new BitmapImage(uriSource);  
    93.   
    94.                 }));  
    95.               
    96.             await GetEmotion(uriSource.ToString());  
    97.         }  
    98.         private async Task GetEmotion(string imageUri)  
    99.         {  
    100.             var client = new HttpClient();  
    101.             var queryString = HttpUtility.ParseQueryString(string.Empty);  
    102.   
    103.             // Request headers  
    104.             client.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key""Your_Key_Emotion_API");  
    105.   
    106.             // Request parameters  
    107.             var uri = "https://westus.api.cognitive.microsoft.com/emotion/v1.0/recognize?" + queryString;  
    108.             EmotionRequest request = new EmotionRequest();  
    109.             request.url = imageUri;  
    110.             byte[] byteData = Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(request));  
    111.   
    112.             using (var content = new ByteArrayContent(byteData))  
    113.             {  
    114.                 content.Headers.ContentType = new MediaTypeHeaderValue("application/json");  
    115.                 var response = await client.PostAsync(uri, content);  
    116.                 var json = await response.Content.ReadAsStringAsync();  
    117.                 if (response.IsSuccessStatusCode)  
    118.                 {  
    119.                     List<EmotionResponse> emotionResponse =  
    120.                         JsonConvert.DeserializeObject<List<EmotionResponse>>(json);  
    121.   
    122.                     if (emotionResponse != null && emotionResponse.Count > 0)  
    123.                     {  
    124.                         var scores = emotionResponse[0].scores;  
    125.                         Dictionary<string, double> dScores = new Dictionary<string, double>();  
    126.                         dScores.Add("anger", scores.anger);  
    127.                         dScores.Add("contempt", scores.contempt);  
    128.                         dScores.Add("disgust", scores.disgust);  
    129.                         dScores.Add("fear", scores.fear);  
    130.                         dScores.Add("happiness", scores.happiness);  
    131.                         dScores.Add("neutral", scores.neutral);  
    132.                         dScores.Add("sadness", scores.sadness);  
    133.                         dScores.Add("surprise", scores.surprise);  
    134.                         var highestScore = dScores.Values.OrderByDescending(score => score).First();  
    135.                         //probably a more elegant way to do this.  
    136.                         var highestEmotion = dScores.Keys.First(key => dScores[key] == highestScore);  
    137.   
    138.                         await Application.Current.Dispatcher.BeginInvoke(  
    139.                             DispatcherPriority.Normal,  
    140.                             new Action(  
    141.                                 () =>  
    142.                                 {  
    143.                                     this.MySpeechSentiment.Text = $"Emotion: {highestEmotion},";  
    144.                                     this.MySpeechSentimentConfidence.Text =  
    145.                                         $"confidence: {Convert.ToInt16(highestScore * 100)}%";  
    146.                                 }));  
    147.                     //    await  
    148.                     //        this.Speak(  
    149.                     //            $"I'm  {Convert.ToInt16(highestScore * 100)}% sure that this person's emotion is {highestEmotion}");  
    150.                     }  
    151.                     else  
    152.                     {  
    153.                         //await  
    154.                         //    this.Speak(  
    155.                         //        $"I'm not able to get the emotion, sorry.");  
    156.                     }  
    157.                 }  
    158.                 else  
    159.                 {  
    160.                     await Application.Current.Dispatcher.BeginInvoke(  
    161.                         DispatcherPriority.Normal,  
    162.                         new Action(() => { this.MySpeechSentiment.Text = "Could not get emotion from this image"; }));  
    163.                     //await  
    164.                     //    this.Speak(  
    165.                     //        $"Could not get emotion from this image.");  
    166.                 }  
    167.             }  
    168.   
    169.         }  
    170.         private  void button_Click(object sender, RoutedEventArgs e)  
    171.         {  
    172.             this.MySpeechSentiment.Visibility = Visibility.Visible;  
    173.             this.MySpeechSentimentConfidence.Visibility = Visibility.Visible;  
    174.             this.MySpeechSentiment.Text = string.Empty;  
    175.             this.MySpeechSentimentConfidence.Text = string.Empty;  
    176.             this.MySpeechResponse.Text = string.Empty;  
    177.             this.MySpeechResponseConfidence.Text = string.Empty;  
    178.             this.searchImage.Source = null;  
    179.             this.micClient.StartMicAndRecognition();  
    180.             
    181.         }  
    182.   
    183.     }  
  1. Also, add two classes - EmotionRequest and EmotionResponse - to store the JSON response of Emotion API.
    1. public class EmotionRequest  
    2. {  
    3.     public string url {  
    4.         get;  
    5.         set;  
    6.     }  
    7. }  
    8. usingCollections.Generic;  
    9. using Newtonsoft.Json;  
    10.   
    11. public class FaceRectangle {  
    12.     public int height {  
    13.         get;  
    14.         set;  
    15.     }  
    16.     public int left {  
    17.         get;  
    18.         set;  
    19.     }  
    20.     public int top {  
    21.         get;  
    22.         set;  
    23.     }  
    24.     public int width {  
    25.         get;  
    26.         set;  
    27.     }  
    28. }  
    29.   
    30. public class Scores {  
    31.     public double anger {  
    32.         get;  
    33.         set;  
    34.     }  
    35.     public double contempt {  
    36.         get;  
    37.         set;  
    38.     }  
    39.     public double disgust {  
    40.         get;  
    41.         set;  
    42.     }  
    43.     public double fear {  
    44.         get;  
    45.         set;  
    46.     }  
    47.     public double happiness {  
    48.         get;  
    49.         set;  
    50.     }  
    51.     public double neutral {  
    52.         get;  
    53.         set;  
    54.     }  
    55.     public double sadness {  
    56.         get;  
    57.         set;  
    58.     }  
    59.     public double surprise {  
    60.         get;  
    61.         set;  
    62.     }  
    63. }  
    64.   
    65. public class EmotionResponse {  
    66.     public FaceRectangle faceRectangle {  
    67.         get;  
    68.         set;  
    69.     }  
    70.     public Scores scores {  
    71.         get;  
    72.         set;  
    73.     }  
    74.   
    75. }  

Source Code

Output

Azure

X

Build smarter apps with Machine Learning, Bots, Cognitive Services - Start free.

Start Learning Now