Chatbot Image Recognization


In my previous article, I discussed the language understanding tool for the chatbot. We will continue here by adding some services of LUIS to our ChatBot, by using Visual Studio. Also, we will see how to add visual responses using adaptive cards with the chatbot. All the lab tools are remaining the same, we will just move with the coding section using Microsoft Visual Studio as an IDE. Natural interfaces are the same as to understand the voice, text, and touch. In this article, I'll show you a user clicking on an item from an image carousel. Adaptive cards enable a developer to define the user interface element using JSON by the host platform in order to build an intelligent bot.
Before moving towards the visual response, we have to add a service to our bot.

Install LUIS package

Step 1
Click FILE from Visual Studio. 
Step 2
Click Open Project/ Solution.
Step 3
Select the Solution file Downloads\buildanintelligentbot\src\TalkToMyBot.sln wait until it's loaded.
Step 4
Click the Solution and then click Restore NuGet Packages
Check out this image:
Chatbot Image Recognization
Now, we will...

Install the LUIS package from NuGet

Step 1 
Right-click on the ChatBot project and click Manage NuGet Packages.
Chatbot Image Recognization
Step 2
Select the Browse tab and search for Microsoft.Bot.Builder.AI.Luis
Step 3
Click on the NuGet package. Select Version 4.4 and click install. Higher or latest versions might raise errors.
These steps are both shown in this image:
Chatbot Image Recognization

Add the LUIS recognizer to our Bot

LUIS is a cognitive service that is accessible through RESTful endpoints. This Bot Builder SDK has inbuilt middleware that integrates the code processing for this intent, not on natural language.
We have to follow these steps:
Step 1 
Go to Visual Studio, Open the appsettings.json file
Step 2
Update the following configuration values.
Replace <your_luis_app_id> and <your_luis_api_key> with the values that you had stored in Notepad previously from the LUIS Portal.
Step 3
Now open the Startup.cs file.
Step 4
Add the following namespace at the top of this file. 
  1. using Microsoft.Bot.Builder.AI.Luis;
Step 5
Find the comment // and create Luis recognizer around line 67, then replace it with this code snippet:
  1. var luisApplication = new LuisApplication(  
  2.     Configuration["LuisAppId"],  
  3.     Configuration["LuisAPIKey"],  
  4.     "https://" + Configuration["LuisAPIHostName"]);  
  5. services.AddSingleton(new LuisRecognizer(luisApplication));  

Adjust our Bot to process the Results

Do some modifications in the code and get the results from LUIS. 
Step 1
Open EchoBot.cs
Add the following namespace at the top of this file.
  1. using Microsoft.Bot.Builder.AI.Luis;
Step 2
Go to this comment:  // Add Luis Recognizerand replace it with this line:
  1. protected LuisRecognizer _luis;  
Step 3
Constructor header is replaced with:
  1. public EchoBot(EchoBotAccessors accessors, IOptions<MySettings> config, LuisRecognizer luisRecognizer)  
Step 4
Goto the comment line: // Initialize the LUIS recognizer and replace it with this line:
  1. _luis = luisRecognizer;  
Step 5
Check for the comment: // Handle LUIS intents recognition and replace it with this code snippet:
  1. // Perform a call to LUIS to retrieve results for the current activity message.  
  2. var luisResults = await _luis.RecognizeAsync(turnContext, cancellationToken).ConfigureAwait(false);  
  3. var topScoringIntent = luisResults?.GetTopScoringIntent();  
  4. var topIntent = topScoringIntent.Value.score > 0.5 ? topScoringIntent.Value.intent : string.Empty;  
  6. // Your code goes here  
Step 6
Recheck Step 5 again, you will find a line // Your code goes here  and replace it with this code snippet:
  1. switch (topIntent)  
  2. {  
  3.     case "TodaysSpecialty":  
  4.         await turnContext.SendActivityAsync($"For today we have the following options: {string.Join("", BotConstants.Specialties)}");  
  5.         break;  
  6.     default:  
  7.         await turnContext.SendActivityAsync("Sorry, I didn't understand that.");  
  8.         break;  
  9. }  
This switch statement sends the user's message to the right handler based on LUIS intent.

Test LUIS Configuration

Step 1
Run the ChatBot by clicking on the IIS Express button in Visual Studio. 
Step 2
Once the page has loaded in Web Browser, copy the site host from the address bar to Notepad. (Like, http://localhost:XXXX)
Chatbot Image Recognization
Step 3
Come to the Bot Framework Emulator.
Step 4
Click file -> New Bot Configuration
Step 5
Paste the URL and add /api/messages it at the end of the URL. (Like, http://locslhost:XXXX/api/messages)
Chatbot Image Recognization
Step 6 
Click on the Save and Connect button.
Step 7 
Type what is the specialty for today? Then press Enter. 
LUIS is processing the input and the bot can handle it accordingly. 
Chatbot Image Recognization
Step 8
Stop Debugging by clicking the stop button in the Visual Studio toolbar.

Add a Visual Response To Your Bot: Carousel

In Visual Studio open EchoBot.cs 
Go to line 54:
  1. private async Task TodaysSpecialtiesHandlerAsync(ITurnContext context)  
  2. {  
  3.     var actions = new[]  
  4.     {  
  5.         new CardAction(type: ActionTypes.ShowImage, title: "Carbonara", value: "Carbonara", image: $"{BotConstants.Site}/carbonara.jpg"),  
  6.         new CardAction(type: ActionTypes.ShowImage, title: "Pizza", value: "Pizza", image: $"{BotConstants.Site}/pizza.jpg"),  
  7.         new CardAction(type: ActionTypes.ShowImage, title: "Lasagna", value: "Lasagna", image: $"{BotConstants.Site}/lasagna.jpg"),  
  8.     };  
  10.     var cards = actions  
  11.       .Select(x => new HeroCard  
  12.       {  
  13.           Images = new List<CardImage> { new CardImage(x.Image) },  
  14.           Buttons = new List<CardAction> { x },  
  15.       }.ToAttachment())  
  16.       .ToList();  
  17.     var activity = (Activity)MessageFactory.Carousel(cards, "For today we have:");  
  19.     await context.SendActivityAsync(activity);  
  20. }  
Add this method.
Let's modify the OnTurnAsync method, so replace this line: 
  1. await turnContext.SendActivityAsync($"For today we have the following options: {string.Join("", BotConstants.Specialties)}");  
  1. await TodaysSpecialtiesHandlerAsync(turnContext);  


Follow these steps:
Step 1
Run the app, click on IIS Express button(green play icon) in Visual Studio.
Step 2 
Open the Models/BotConstants.cs file.
Step 3
Check Site constant. If there is a different URL, then stop the app, update it with the right port, and run this app again.
Step 4
Open the Bot Framework Emulator and click on Restart Conversation
Step 5
Type this query line: What is today's specialty? Press enter.  
You can see the set of images in a Carousel.
At last, Stop debugging in Visual Studio. 
Chatbot Image Recognization
In this image, there is a kind of online food ordering app, all the recommended options are in front of the viewer and according to the visual recognition, it also looks too good comparatively text reverts from the ChatBot.
Chatbot has many modules such as; foram module: QnA,  Speech Recognization, Voice Recognization, Face Recognization. By using different tools and technologies, we can explore the variety of new techniques to invent something unique and a worthy product for the end-users as well as the experts.   
For any programming file, DM me, please.


In this article, we discussed recognition with a ChatBot using text and visual effects with the help of LUIS and Visual Recognization respectively. There are many different types of modules and usage is going to be included with the chatbot. It all depends on the curiosity and the invention technics of the developer. Kindly check the attachments or links with this article.