A Quick Introduction To Computer Vision Using C#

This way, your objects (faces, bodies, materials, textures) etc are recognized in the computer and the computer tells you what it was.


Figure 5 - Objects detected and recognized

As you can see, the computer program is capable of understanding what objects are present in the image and then it recognizes the “known” objects by matching them up against the already saved results in the database.

3. Object tracking

This type of program is based on video analysis, such as fetching the frames from a webcam and then performing image processing techniques on the image itself to learn more about the image and the objects in the image. Not just that, this field of computer vision also stores the states of the objects, such as their location in the frame. Thenthe program tracks the objects in the frames and follows them as they move around in the frames of the video that is being streamed from the webcam. A demonstration can be found in the following representation,


Figure 6 - Human tracking in the frames of a webcam or surveillance camera

As seen in the image above, the program has detected the humans and has been tracking them for awhile. Notice how it has trailed their paths with a line that shows where and how they were moving in the image frames.
This field is (and can be) only applicable in video processing, in image processing (where only one frame is available) this cannot be applied because objects are not going to move here and there.

4. Rest of the stuff

There are many other uses of computer vision, that, if discussed, will cause this post to become very long in length. Keeping the post considerate and short, I am going to talk about a few other uses of computer vision in regular day life.

Suppose you wanted to create a security system which keeps track of the people standing in the street. You can do so by computer vision -- you are allowed you count the number of people standing in the street and once the number rises to a “threshold”, you can trigger an alarm or do what you want to. This is one use.

Otherwise, suppose you were given a few shots of a person from different angles, you can use manipulation of those images to reconstruct a 3D image of that person. 3D image reconstruction is a very open area of research in computer vision and can make good use of some popular research papers in the same field.

I am sure, by now, you must be able to perceive how interesting, amazing, and helpful this field of computer science can be to your projects, to your clients, and to your household applications. I have recently build a server for my home, I am definitely going to apply these to the server and make use of my skills and the power of computer vision to provide features like, services, security and other stuff that I can make with computer vision. In the coming section, I will talk about libraries of computer vision that are actively being used in regular everyday projects by everyone.

Libraries Available

Do you remember, when you were told: Do not reinvent the wheel?

I am sure you do. Likewise, you should never consider rewriting everything from scratch that would consume a lot of time and won’t give any better results at all. There are many libraries written for computer vision and image processing. I would always recommend that you use those libraries. In this section, I will walk you through different libraries and why you should use them.

The Mighty OpenCV

I have tried and used OpenCV frameworks for computer vision programming. Trust me, it is the best library available out there. There are many other libraries and tools available, in which MATLAB is one such tool that allows image processing. But who wants to spend a bunch on a tool that can be used free of cost? I have used MATLAB and I have used OpenCV. OpenCV is my personal recommendation.

A few of the uses and benefits of using OpenCV are,

1. Open source and free — I just couldn’t find any better reason to use OpenCV than this, to be shared as the first benefit.
   
   
   • OpenCV is led and updated by the community. So there is a lot less patent stuff, organization philosophy and “how to use it” stuff going on. You are free to use it anyway, anytime and for any project.
     
     
2. Written natively in C and C++. C++ APIs of OpenCV provide excellent performance and efficiency for real-time processing of images.
   
   
3. Cross-platform support.
   
   
   • Can run on Android.
   • Can be used in iOS applications.
   • Desktop and server applications can surely use the power of OpenCV.
     
     
4. Supports the world’s well known frameworks,
   
   
   • It supports OpenCL. OpenCL is the open computer language, written in C programming language that runs on any machine hardware regardless of their architectures. OpenCV can use OpenCL to enhance the performance and minimize the compile-time modification of code.
     
     
   • Intel IPP is well known for image processing. OpenCV can use Intel IPP if it detects it to be installed on the machine.
     
     
5. Many giants have contributed source code to the repository.
   
   Google, Intel and Microsoft had contributed a good amount of source code to the repository fixing and adding a lot of features on multiple platforms, which makes OpenCV one of the best available libraries for computer vision out there.
   
   Features are “amazingly” outstanding and way too much to handle,
   
   
   • It supports both formats for images; 2D and 3D.
   • Object detection and recognition systems.
   • Gesture recognition.
   • Object tracking.
   • Robotics.

I will be writing a few of the guides for these features soon, but for this one I am just going to give you an overview of the library itself and how you can use this library in your own .NET framework applications.

Other libraries

There are other libraries available too, I don’t want to say they are not available or that they cannot be used. But the fact is, they all have a bit of glitches and as I have said, OpenCV is free from philosophical junk and is community-driven. What is loved in the community, stays in the library, what is not liked, gets removed. Other libraries have a few issues in them,

1. Some are not being monitored and fixed. Some were last reviewed or modified back in 2005.
2. Some have issues in programming. Lack of good programming paradigms for instance.
3. Bad performance as compared to the rest.

So on and so forth.

Using OpenCV in .NET Environment

Since I have said OpenCV is a library written in native C and C++ programming languages, then how can you use it in C#? Luckily, C# supports some of the unsafe programming too, unsafe in the terms of unmanaged programming, where anything can go wrong. There are many wrappers written in C# that allows you to communicate with OpenCV library runtime. These wrappers allow your C# programs to be able to communicate with the APIs of the underlying library that was written with unmanaged code. Unmanaged code can be the code that manipulates the memory itself, such as in the case of C++.

Emgu CV is one such wrapper, I have used it in my own .NET applications and it is very simple to use. Although, just like other wrappers, it does have some issues such as compatibilities if you are a C++ programmer, the design may bug you a bit, because the wrapper was written keeping C# in mind. From this step, I will give you a bit of an idea of using Emgu CV in your own programming in .NET framework.

Adding the library

Emgu CV must be installed before it can be used in your projects. First of all, go and download the packages from the website, one thing to know here is that the versions 2.4 and 3 are different and the version 3 has everything rewritten and rethought from the ground up! So think twice before downloading the packages, however both of the packages come shipped with the sample source code that gets you ready and set up in no time! You can try out those source samples in your own machine and see for yourself how things work. But remember, Emgu CV 3.0 has an enormous amount of performance and bug fixes but currently the most stable release is version 2.4.

The library needs to be available in your executable’s directory, the output folder where your executable resides. You can add the library as an existing item to the directory, or you can use the References → Add Reference → Emgu CV libraries.


Figure 7 - Addition of Emgu CV libraries

These two libraries are required at least to perform the least of the processing in your applications. They are required to be available when your application starts to execute. You can use this library in any of your .NET applications, I started with Console application and raised it to Windows Forms, which leads me to other frameworks to be also supported and so on.

Sample work

Sure, I would love to get you to have a look at what I built recently, sadly I am going to let yousee only the visual of that and not the source code, I am leaving the source code for some time later.


Figure 8 - I am being detected. Application shows other controls and extra information in the window, too

I was able to create this application in much less time using Emgu CV in .NET framework. The frameworkis being used in Windows Forms as well, if you see thoroughly you can see that this application has quite a few features in itself.

1. Face detection.
   
   
2. Face tracking, notice that the grayscale image only holds the face that has been detected. It follows and tracks the face in the frame.
   
   
3. I added a bit of face recognition features to the application too.
   
   
   1. I added the support for training the model.
   2. I added the support for saving the bitmaps of the face patterns associated with the person names. See the textbox.
   3. I added the button to start the detection of the face as a person.
   4. In the last line, notice how the application tells to whom this face belongs. I created a database system to hold the information; that is from where the ID comes from.

I will be sharing this application with you guys soon, at the moment I need to make a few more changes to the application to support even more functionality and to remove any bugs in the algorithms.

Extra features - Image processing tools

That was the sample of what I had created, that was not just what Emgu CV is capable of doing, there are many more features to it than just detecting the faces in the image frames. However, some of these functions are directly related to image processing way more than they are related to computer vision.

1. Edge detection
   
   The following image, is a sample from Emgu CV team and it shows how it can process the image coming from the webcam and detect edges using a canny algorithm, and convert it to grayscale too.
   
   
   Figure 9 - My face being captured and being processed using image processing to get information such as edges, grayscale image etc
   
   
2. Grayscale images
   
   You have already seen what features Emgu CV has to support grayscale image creation and processing.
   
   
3. Histogram equalization
   
   It is a well-known image processing technique used in many ways and for many purposes, one of the purposes that I remember in my “Digital image processing” class was to enhance the contrast of the images in MATLAB using histogram equalization techniques. Emgu CV uses the interfaces of OpenCV to perform the histogram equalization, the technique is pretty simple, you just call the function on the image objects. Such as the following code,
   
   1. Image<Bgr, byte>inputImage = GetImage();
   2. inputImage._EqualizeHist();
   You will learn more on this in the coming posts that I will be writing soon. So, stay tuned.

Performance management

Image processing and computer vision require a lot of computer power to perform the actions that you want to get results from. Even the simplest of the programs require a complex matrix manipulation and would require your computer to perform a huge amount of computation to get the results. So, I am pretty sure you need to understand how you can minimize the overheads.

Parallelism

If you have a computer machine that has multi-cores, dual or quad cores, then surely you can use parallelism to enhance and increase the performance of your application using threadings and send the data to be processed as multiple threads and cores. This can help. But remember, parallelism is totally relative to the algorithm being used. Sometimes, it can help you a lot, sometimes it can just be overkill or sometimes it may be the reason for extra time being taken to process.

To learn more about this, I recommend going through the following links:

1. Parallel computing
2. Parallel Programming in the .NET Framework

I would still say, use with caution.

Code tuning

In my own applications, I consider using this mechanism to enhance the performance of the applications. I fine tune the application’s code to increase the performance. There are many reasons why this should be used, the first one being, your customers may not have the fastest computer machines.

In such cases, the best way of improving the performance and efficiency of the application is to improve the code being used.

Final words

This post was created just to give you an idea of how you can use Emgu CV in your .NET applications to perform computer vision programming in your applications. In coming posts, I will show you how you can perform these actions in your application in the form of source code.

I hope you enjoyed reading this, if I missed something let me know and I will talk about that in the coming posts… See you in the next posts!