Building Video Indexer Mobile Application Using Cognitive Service

Introduction

Microsoft introduced the public preview of Video Indexer as a part of Cognitive Service. Previously, we used Video API but now it's replaced with Video Indexer. Video Indexer automatically extracts the metadata and builds intelligent innovative AI applications based on Video and Audio.

Cognitive Service

In this article, I will show how to integrate Video Indexer embeddable widgets into the mobile application.

Create Video Indexer Account

For implementing Video Indexer into the application, you must create an account and upload your video using my previous article as a reference .

Video Indexer supports embedding two types of widgets into your Mobile application

  1. Cognitive Insights – it includes all visual insights that were extracted from video indexer.
  2. Player -it enables you to stream the video.

Embedding Video

 I will show the below steps for how to get the embedding source from video indexer portal

Step 1

Login to Video Indexer Portal using Microsoft, google or Azure Active directory account

Step 2

Select your video, whichever one needs to integrate into the application.

Step 3

Click the "embed" button that appears below the video.

Cognitive Service

Step 4

After clicking on “embed” button, select the widget you want to embed with the desired options. (player/insights)

Cognitive Service

Step 5

You can copy the embed codes from embed popups and start implementing them into your application for Public videos. If you want to embed a Private video, you have to pass an access token in the iframe src attribute as a query string .

Create Xamarin / Web Application

Let's start with creating a new Xamarin Forms Project using Visual Studio. Open Run - Type “Devenev.Exe” and enter - New Project (Ctrl+Shift+N) - select Blank Xaml App (Xamarin.Forms) template.

Cognitive Service

It will automatically create multiple projects, like .NET Standard, Android, iOS, and UWP.

Implement Cognitive Insights Widget

Cognitive insights widget contains all the visual insights that were extracted from the video after the indexing process such as people appearances, top keywords, sentiment analysis, transcript, translate, and search.

Let's start implementing it into the Xamarin Application.

You can add webview control inside the content page as per the below xaml design

  1. <?xml version="1.0" encoding="utf-8" ?>  
  2. <ContentPage xmlns="http://xamarin.com/schemas/2014/forms"  
  3.              xmlns:x="http://schemas.microsoft.com/winfx/2009/xaml"  
  4.              xmlns:local="clr-namespace:VideoIndexer"  
  5.              x:Class="VideoIndexer.MainPage">  
  6.     <WebView x:Name="browser"  WidthRequest="1280" HeightRequest="780"></WebView>  
  7.   
  8. </ContentPage>  

You can use your cognitive insights widget code from c# code as a htmlwebviewsource . You can specify the query string for whatever widgets you need to display ( <url>?widgets=people&title=myInsights)

  1. public MainPage()  
  2.         {  
  3.             InitializeComponent();  
  4.            // var browser = new WebView();  
  5.             var htmlSource = new HtmlWebViewSource();  
  6.             htmlSource.Html = @"<html><head><script src='https://breakdown.blob.core.windows.net/public/vb.widgets.mediator.js'></script></head>  
  7. <body><iframe width='580' height='780' src='https://www.videoindexer.ai/embed/insights/9123e16b12' frameborder='0' allowfullscreen></iframe>  
  8. </body></html> ";  
  9.             browser.Source = htmlSource;  
  10.         }  

You can deploy the application, the output looks like below

Cognitive Service

Implement Video Insights Widget

The video player widget is a customized Media Player that provides video streaming, and contains extra features such as playback speed and closed captions. Refer to the below code for video insight wedge implementation

  1.     public MainPage()  
  2.         {  
  3.             InitializeComponent();  
  4.            // var browser = new WebView();  
  5.             var htmlSource = new HtmlWebViewSource();  
  6.             htmlSource.Html = @"<html><head><script src='https://breakdown.blob.core.windows.net/public/vb.widgets.mediator.js'></script></head>  
  7. <body><iframe width='1280' height='720' src='https://www.videoindexer.ai/embed/player/9123e16b12' frameborder='0' allowfullscreen></iframe>  
  8. </body></html> ";  
  9.             browser.Source = htmlSource;  
  10.         }  

Implement Video and Cognitive Insights Widget

Copy the player and cognitive insights embed code and also include javascript communication file for communication with the other widgets

  1.   var htmlSource = new HtmlWebViewSource();  
  2.             htmlSource.Html = @"<html><head><script src='https://breakdown.blob.core.windows.net/public/vb.widgets.mediator.js'></script></head>  
  3. <body><iframe width='1280' height='720' src='https://www.videoindexer.ai/embed/player/9123e16b12' frameborder='0' allowfullscreen></iframe>  
  4. <iframe width = '1280' height = '780' src = 'https://www.videoindexer.ai/embed/insights/9123e16b12/?widgets=people&title=myInsights' frameborder = '0' allowfullscreen ></iframe >   
  5. </body></html> ";  
  6.             browser.Source = htmlSource;  

Summary

In this article, we learned how to sign in to Video Indexer and integrate Video Indexer embeddable widgets into the mobile cross platform application. If you have any questions/feedback/ issues, please write in the comments box.