In the last article, we learnt about LUIS - Language Understanding Intelligent Service provided by Azure and then learnt to create a conversation app. This was fundamental to create a cognitive service in Azure such that we can obtain a subscription key and endpoint to use in our application.

This article focuses on following up on the app created in Azure to make a full-fledged AI Chatbot. We can learn about all these services provided in Azure for Machine Learning through the article, Azure Cognitive Services. Also read the last article, Luis – Create a conversation app this follows up on.

Computer Vision

Computer Vision is synonymous with its name. This branch of AI aids by supporting computers to analyze data from images, cameras, videos, and other visual content. Computer Vision mainly deals with the processing of visual data such as images or videos. Multitudes of Machine Learning Models can be implemented using various Algorithms to perform different tasks.

Optical Character Recognition

Optical Character Recognition can be understood as the process of transforming handwritten texts, printed or typed text into the form that the machine can encode. This can be performed from images or scanned documents and even superimposed texts on image.

Brand Identification

Brand Identification is simply a process to identify the brand through visual clues such as Logos.

Object Detection

Object Detection is a process of detecting, locating, and identifying objects from a visual image or a video. It is a kind of Image Processing and Computer Vision technology, which are performed mainly with Machine Learning and Deep Learning implementation.

Face Detection

Face detection as the name suggests in a technology using which human faces can be detected in images, videos, and other digital forms. This has huge implications for systems from airports to shopping malls. Its applications and usage are only limited to human imagination.

Face Recognition

Face Recognition is the capability of a system to recognize and distinguish a specific person through matching the digital image or video from the database of faces. From usage for security to unlock phones to identify people in social media platforms like Facebook, face recognition has huge implications.

Image Analysis

Image Analysis goes into deeper computer vision work. It uses multiple functionalities such as object detection, face detection and intent to create tags and provide generation of descriptive subtitles which summarizes the scene in the image.

Computer Vision API

Similar to the LUIS service in Azure, Computer Vision API is also provided by Azure in order to create Computer Vision AI enabled application for users and developers without the need to work themselves on the Machine Learning and Deep Learning prospects. Simply create the Computer Vision service through Cognitive Services following up on the previous articles, Create a Cognitive Service and you’ll be good to go. One can then use the JSON formatted output to use in their applications.

Now, we’ve created our Cognitive Service for Computer Vision. Next, we create our application in Visual Studio in C#. We’ll create a bot that can identify from the text about weather intent plus send details about image user sends by using image recognition. 

using System.Collections.Generic;

using Newtonsoft.Json;

namespace ImageBot.Models
{
    // LUIS classes
    public class GetCityWeather
    {
        public double score { get; set; }
    }

    public class None
    {
        public double score { get; set; }
    }

    public class Intents
    {
        public GetCityWeather GetCityWeather { get; set; }
        public None None { get; set; }
    }

    public class GeographyV2
    {
        public string value { get; set; }
        public string type { get; set; }
        public string text { get; set; }
        public int startIndex { get; set; }
        public int length { get; set; }
        public int modelTypeId { get; set; }
        public string modelType { get; set; }
        public List<string> recognitionSources { get; set; }
    }

    public class Instance
    {
        public List<GeographyV2> geographyV2 { get; set; }
    }

    public class Entities
    {
        public List<GeographyV2> geographyV2 { get; set; }

        [JsonProperty("$instance")]
        public Instance Instance { get; set; }
    }

    public class Prediction
    {
        public string topIntent { get; set; }
        public Intents intents { get; set; }
        public Entities entities { get; set; }
    }

    public class LuisRoot
    {
        public string query { get; set; }
        public Prediction prediction { get; set; }
    }

}

If you want to learn more about creating Whatsapp Bot using Azure AI, watch this video.

Hence, we have created a model LuisModels.cs for all the different types of values we require to fetch from our Azure Endpoint for our LUIS app such as GetCityWeather, Intents, GeographyV2, Entities and Prediction. 

using System.Collections.Generic;

namespace ImageBot.Models
{
    public class WeatherModel
    {
        public Coord coord { get; set; }
        public List<Weather> weather { get; set; }
        public string @base { get; set; }
        public Main main { get; set; }
        public int visibility { get; set; }
        public Wind wind { get; set; }
        public Clouds clouds { get; set; }
        public int dt { get; set; }
        public Sys sys { get; set; }
        public int id { get; set; }
        public string name { get; set; }
        public int cod { get; set; }
    }

    public class Weather
    {
        public int id { get; set; }
        public string main { get; set; }
        public string description { get; set; }
        public string icon { get; set; }
    }

    public class Coord
    {
        public double lon { get; set; }
        public double lat { get; set; }
    }

    public class Main
    {
        public double temp { get; set; }
        public double pressure { get; set; }
        public double humidity { get; set; }
        public double temp_min { get; set; }
        public double temp_max { get; set; }
    }

    public class Wind
    {
        public double speed { get; set; }
    }

    public class Clouds
    {
        public double all { get; set; }
    }

    public class Sys
    {
        public int type { get; set; }
        public int id { get; set; }
        public string country { get; set; }
        public long sunrise { get; set; }
        public long sunset { get; set; }
    }
}

Next, we create a model WeatherModels.cs which can place value for data of the Weather such as temperature, pressure, humidity, cloud, wind, speed, country and more. 

using System;
using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;
using Microsoft.Extensions.Logging;

using Newtonsoft.Json;

using ImageBot.Models;

namespace ImageBot.Services
{
    public class LuisService
    {
        private static string luisURL = "";
        private static readonly HttpClient httpClient = new HttpClient();

        public static async Task<string> GetCity(string text, ILogger log)
        {
            try
            {
                var luisFullURL = $"{luisURL}&query={text}";
                var luisResult = await httpClient.GetStringAsync(luisFullURL);
                log.LogInformation(luisResult);
                var luisModel = JsonConvert.DeserializeObject<LuisRoot>(luisResult);

                if (luisModel.prediction.topIntent == "GetCityWeather")
                {
                    var entity = luisModel.prediction.entities;

                    if (entity != null)
                        if (entity.geographyV2 != null)
                            return entity.geographyV2.FirstOrDefault().value;
                }
                else
                    return "Sorry, I could not understand you!";
            }
            catch (Exception ex)
            {
                log.LogInformation(ex.Message);
            }

            return "Sorry, there was an error!";
        }
    }
}

Furthermore, we now create our service for LUIS as LuisService.cs which can obtain values from our LUIS URL link.

Next, we have our WeatherService.cs which obtains the weatherURL and the apiKey from our LUIS service we created in our last article, Luis – Create a conversation app. 

using System.Linq;
using System.Net.Http;
using System.Threading.Tasks;

using Newtonsoft.Json;

using ImageBot.Models;

namespace ImageBot.Services
{
    public class WeatherService
    {
        private static string weatherURL = $"http://api.openweathermap.org/data/2.5/weather";
        private static string apiKey = "";
        private static readonly HttpClient httpClient = new HttpClient();

        public static async Task<string> GetWeather(string city)
        {
            var weatherFullURL = $"{weatherURL}?appid={apiKey}&q={city}";
            var weatherResult = await httpClient.GetStringAsync(weatherFullURL);
            var weatherModel = JsonConvert.DeserializeObject<WeatherModel>(weatherResult);
            weatherModel.main.temp -= 273.15;

            return $"{weatherModel.weather.First().main} ({weatherModel.main.temp.ToString("N2")} °C)";
        }
    }
}

Next, we now create our Computer Vision Models and Services. 

using System.Text;
using System.Threading.Tasks;
using System.Collections.Generic;

using Microsoft.Azure.CognitiveServices.Vision.ComputerVision;
using Microsoft.Azure.CognitiveServices.Vision.ComputerVision.Models;

namespace ImageBot.Services
{
    public class ComputerVisionService
    {
        private static string subscriptionKey = "";
        private static string endpoint = "";

        public static async Task<string> AnalyzeImage(string url)
        {
            var client = Authenticate(endpoint, subscriptionKey);
            var analysis = await AnalyzeImageUrl(client, url);
            return analysis;
        }

        private static ComputerVisionClient Authenticate(string endpoint, string key)
        {
            return new ComputerVisionClient(
                new ApiKeyServiceClientCredentials(key))
            {
                Endpoint = endpoint
            };
        }

        private static async Task<string> AnalyzeImageUrl(ComputerVisionClient client, string imageUrl)
        {
            var features = new List<VisualFeatureTypes?>()
            {
                VisualFeatureTypes.Categories, VisualFeatureTypes.Description,
                VisualFeatureTypes.Faces, VisualFeatureTypes.ImageType,
                VisualFeatureTypes.Tags, VisualFeatureTypes.Adult,
                VisualFeatureTypes.Color, VisualFeatureTypes.Brands,
                VisualFeatureTypes.Objects
            };

            var results = await client.AnalyzeImageAsync(imageUrl, visualFeatures: features);

            var analysis = new StringBuilder();

            analysis.AppendLine("** Summary **");
            foreach (var caption in results.Description.Captions)
                analysis.AppendLine($"{caption.Text} with confidence {caption.Confidence}");

            analysis.AppendLine("** Categories **");
            foreach (var category in results.Categories)
                analysis.AppendLine($"{category.Name} with confidence {category.Score}");

            analysis.AppendLine("** Tags **");
            foreach (var tag in results.Tags)
                analysis.AppendLine($"{tag.Name} {tag.Confidence}");

            analysis.AppendLine("** Objects **");
            foreach (var obj in results.Objects)
                analysis.AppendLine($"{obj.ObjectProperty} with confidence {obj.Confidence}");

            analysis.AppendLine("** Brands **");
            foreach (var brand in results.Brands)
                analysis.AppendLine($"Logo of {brand.Name} with confidence {brand.Confidence}");

            analysis.AppendLine("** Faces **");
            foreach (var face in results.Faces)
                analysis.AppendLine($"A {face.Gender} of age {face.Age}");

            analysis.AppendLine("** Celebrities **");
            foreach (var category in results.Categories)
                if (category.Detail?.Celebrities != null)
                    foreach (var celeb in category.Detail.Celebrities)
                        analysis.AppendLine($"{celeb.Name} with confidence {celeb.Confidence}");

            analysis.AppendLine("** Landmarks **");
            foreach (var category in results.Categories)
                if (category.Detail?.Landmarks != null)
                    foreach (var landmark in category.Detail.Landmarks)
                        analysis.AppendLine($"{landmark.Name} with confidence {landmark.Confidence}");

            return analysis.ToString();
        }

    }
}

Here, in the ComputerVisionService.cs we connect to our Azure Service for cognitive functionality usage for Computer Vision. We add in the subscriptionKey and endpoint. Next, we fetch all the data for Image Recognition such as Categories, Faces, Tags, Color, Objects with their descriptions, image type, make sure if its adult content or not, and detect brands too. Finally, once we obtain the data, we output it to the user in WhatsApp Bot with summary, the category the image they send falls in, the Face, Objects and Brands detected in the image and in case there are specific celebrities or landmarks.

Next, we access the WhatsApp business API via, https://www.whatsapp.com/business/api and if in case you need a business partner you can get support from Facebook Business Partner for Whatsapp by paying a fee.

WhatsApp Business API

WhatsApp Business API helps businesses boost their communication with their customers across the globe such as by connecting users to businesses in a safe and reliable messaging app WhatsApp. Howsoever, to use the WhatsApp Business API, one needs to have a contract with a global business solution provider that is accessible from the partner directory in Facebook as discussed above.

Conclusion

Thus, in this article, we learnt about Computer Vision and different types of it sub topics such as OCR, Object Detection and more. Then we learnt about the Computer Vision API of Azure which enables developers to use the features of Computer Vision provided by Azure in their application. Next, we dived deep into creating the application in C# using Visual Studio by developing models for Luis and Weather. Finally, we created the ComputerVision service which connects to Azure Service and can fetch data from the Computer Vision API and send the replies to the user through the WhatsApp Bot. We discussed about the WhatsApp Bot and the process to use the WhatsApp Business API.

In the next article, we’ll learn how to use Twilio API for WhatsApp, receive and send data and messages to and from the application and host our application in Azure Functions.