Number Plate Recognition with Azure Custom Vision and Azure Computer Vision

Introduction

Number Plate Recognition (NPR) has become increasingly significant in numerous domains, including law enforcement, traffic management, and parking systems. By automating the process of extracting and identifying license plate numbers, NPR systems streamline operations, enhance security, and enable efficient data analysis. In this blog post, we will explore a comprehensive approach to Number Plate Recognition that combines the power of Azure Custom Vision and Azure Computer Vision technologies.

Approach

Our approach to Number Plate Recognition involves two fundamental components: Azure Custom Vision and Azure Computer Vision. Azure Custom Vision, a cloud-based image recognition service, enables us to create a robust and accurate license plate detection model. Conversely, Azure Computer Vision, powered by Azure Cognitive Services, empowers us to extract the plate number from the detected license plate regions.

Training the License Plate Detection Model using Azure Custom Vision

The first step in our approach is to train a license plate detection model using Azure Custom Vision. This process entails collecting a diverse dataset of labeled license plate images, covering various license plate sizes, styles, lighting conditions, and angles. With Azure Custom Vision, we can easily upload the dataset, train a machine learning model, and fine-tune it to accurately detect license plates within images.

From the Azure Portal, search the Cognetive Services multi-service account and create an account.

For this example, the instance name is CarPlateDetection.

Open another tab and go to the Custom Vision Portal. You must sign in with the Microsoft account that used earlier. Click on New Project.

Type the project name, description and select the resource. Click on Object Detection as Project Type and General [A1] as Domain, then Create Project.

Click on Add Images to upload the car images where the plate number is visible. Search at least 20 images for this example.

You will see the images you uploaded.

Now, click on each one to select the region of the plate number you want the Machine Learning model detects. You must type the tag name after you select the region.

After you finished tagging all the images, click on Train.

Choose Quick Training an click on Train.

This can take between 10 and 15 minutes, based on the number of images you tagged.

You will see three metrics:

Precision: It's the percentage of identified classifications that were correct. For example, if the model identified 100 images as dogs and 99 of them were actually dogs, then the precision is 99 percent
Recall: It's the percentage of actual classifications that were correctly identified. For example, if there were actually 100 images of apples, and the model identified 80 as apples, the recall is 80 percent.
Mean average precision (mAP): It's the average value of the average precision (AP). AP is the area under the precision/recall curve (precision plotted against recall for each prediction made).

The Probability Threshold is the desired level of confidence that a prediction needs to have in order to be considered correct. When you interpret prediction calls with a high probability threshold, they tend to return results with high precision at the expense of recall. That is, the detected classifications are correct, but many remain undetected. A low probability threshold does the opposite: most of the actual classifications are detected, but there are more false positives within that set. With this in mind, you should set the probability threshold according to the specific needs of your project. By default, the probability threshold is 50% and can be set between 0% and 100%.

You can test the model by clicking on Quick Test. Upload any car image to test the model.

Once you are comfortable with the testing results, click on Publish and type the model name, then select the prediction resource. Finally, click on Publish.

Click on Prediction URL to see the URL and Prediction Key.

Preprocessing and Localization

After training the license plate detection model, we need to preprocess the input images to enhance their quality and reduce noise. Azure Cognitive Services provide powerful image preprocessing capabilities, such as image resizing, contrast adjustment, and noise reduction, which can be applied to optimize the image for subsequent processing. Once the preprocessing is complete, the license plate detection model is utilized to locate license plate regions within the images.

We'll create a Python Notebook to call the Azure Custom Vision and Azure Computer Vision APIs.

import requests
import cv2
import pandas as pd
import numpy as np
from matplotlib import pyplot as plt
from PIL import Image
from numpy import asarray

Take a look at Custom Vision Prediciton 3.0 for further details.

custom_vision_imgurl = 'https://<resource_name>.cognitiveservices.azure.com/customvision/v3.0/Prediction/<project_id>/detect/iterations/<published_name>/image'
api_key = '<YOUR_PREDICTION_KEY>'

data = open('Cars326.png', 'rb').read()
# decode the image file as a cv2 image, useful for later to display results
img = cv2.imdecode(np.array(bytearray(data), dtype='uint8'), cv2.IMREAD_COLOR)

custom_vision_headers = {
    'Content-Type': 'application/octet-stream', 
    'Prediction-Key': api_key
    }

custom_vision_resp = requests.post(url = custom_vision_imgurl, 
    data = data, 
    headers = custom_vision_headers).json()

# inspect the top result, based on probability 
hit = pd.DataFrame(custom_vision_resp['predictions']).sort_values(by='probability',ascending=False).head(1).to_dict()

{
   "probability":{
      "0":0.99115235
   },
   "tagId":{
      "0":"c5b7e0da-7c90-4e63-bf6f-b299f1c6492a"
   },
   "tagName":{
      "0":"Plate"
   },
   "boundingBox":{
      "0":{
         "left":0.3620827,
         "top":0.6698199,
         "width":0.2792757,
         "height":0.08204675
      }
   }
}

# extract the bounding box for the detected number plate 
boundingbox = list(hit['boundingBox'].values())[0]
l, t, w, h = (boundingbox['left'], 
    boundingbox['top'], 
    boundingbox['width'], 
    boundingbox['height'])

# extract bounding box coordinates and dimensions are scaled using image dimensions 
polylines1 = np.multiply([[l,t],[l+w,t],[l+w,t+h],[l,t+h]], 
    [img.shape[1],img.shape[0]])

# draw polylines based on bounding box results
temp_img = cv2.polylines(img, np.int32([polylines1]), 
    isClosed = True, color = (255, 255, 0), thickness = 2)

# display the original image with the plate region
plt.imshow(cv2.cvtColor(temp_img, cv2.COLOR_BGR2RGB))

# crop the image to the bounding box of the plate region
crop_x = polylines1[:,0].astype('uint16')
crop_y = polylines1[:,1].astype('uint16')

img_crop = img[np.min(crop_y):np.max(crop_y), 
    np.min(crop_x):np.max(crop_x)]

# display the detected plate region
plt.imshow(cv2.cvtColor(img_crop, cv2.COLOR_BGR2RGB))

License Plate Extraction using Azure Computer Vision

Once the license plate regions are identified, we employ Azure Computer Vision techniques to extract the actual plate numbers. This step involves applying image segmentation algorithms to isolate the characters and digits on the license plates. With Azure Computer Vision's OCR capabilities, we can accurately extract alphanumeric information from the segmented regions.

Before calling the Azure Computer Vision API, we must ensure the image we send has at least 50px of height.

img_crop_height = img_crop.shape[0]
if img_crop_height < 50:
      pil_image = Image.fromarray(img_crop)
      img_crop_width = img_crop.shape[1]
      difference = 50 / img_crop_height
      resized_dimensions = (int(img_crop_width * difference), int(img_crop_height * difference))
      pil_image_resized = pil_image.resize(resized_dimensions)
      img_crop_resized = asarray(pil_image_resized)

      plt.imshow(cv2.cvtColor(img_crop_resized, cv2.COLOR_BGR2RGB))

computer_vision_imgurl = 'https://<resource_name>.cognitiveservices.azure.com/computervision/imageanalysis:analyze?api-version=2023-02-01-preview&features=read'

crop_bytes = bytes(cv2.imencode('.png', img_crop_resized)[1])

# make a call to the computer_vision_imgurl
computer_vision_resp = requests.post(
    url=computer_vision_imgurl, 
    data=crop_bytes, 
    headers={
        'Ocp-Apim-Subscription-Key': api_key, 
        'Content-Type': 'application/octet-stream'}).json()

{
   "readResult":{
      "stringIndexType":"TextElements",
      "content":"DL8CX 48.50",
      "pages":[
         {
            "height":50.0,
            "width":224.0,
            "angle":0.5536,
            "pageNumber":1,
            "words":[
               {
                  "content":"DL8CX",
                  "boundingBox":[
                     5.0,
                     12.0,
                     103.0,
                     13.0,
                     102.0,
                     43.0,
                     4.0,
                     42.0
                  ],
                  "confidence":0.677,
                  "span":{
                     "offset":0,
                     "length":5
                  }
               },
               {
                  "content":"48.50",
                  "boundingBox":[
                     127.0,
                     14.0,
                     208.0,
                     14.0,
                     207.0,
                     44.0,
                     127.0,
                     43.0
                  ],
                  "confidence":0.668,
                  "span":{
                     "offset":6,
                     "length":5
                  }
               }
            ],
            "spans":[
               {
                  "offset":0,
                  "length":11
               }
            ],
            "lines":[
               {
                  "content":"DL8CX 48.50",
                  "boundingBox":[
                     4.0,
                     11.0,
                     211.0,
                     13.0,
                     211.0,
                     43.0,
                     4.0,
                     41.0
                  ],
                  "spans":[
                     {
                        "offset":0,
                        "length":11
                     }
                  ]
               }
            ]
         }
      ],
      "styles":[

      ],
      "modelVersion":"2022-04-30"
   },
   "modelVersion":"2023-02-01-preview",
   "metadata":{
      "width":224,
      "height":50
   }
}

In the above response, we can see that the plate number is DL8CX 48.50, which is present in readResult.content.

You can find the source code here.

Post-processing and Analysis

After successfully extracting the license plate numbers, additional post-processing techniques can be applied to refine the results. These may include noise removal, character recognition validation, and formatting adjustments to ensure the extracted plate numbers are accurate and formatted correctly. Once the data is ready, it can be further analyzed, stored, or used for various purposes, such as database integration or real-time monitoring systems.

Conclusion

Number Plate Recognition using Azure Custom Vision and Azure Computer Vision technologies offers a comprehensive solution for accurate license plate detection and extraction. By leveraging the capabilities of Azure's cloud-based services, we can automate the process of recognizing and extracting license plate numbers, opening up a wide range of possibilities for applications in traffic management, parking systems, and law enforcement. In the upcoming sections of this blog series, we will delve deeper into each step of the approach, exploring the underlying techniques and demonstrating their implementation using Azure Custom Vision and Azure Computer Vision.

Stay tuned to unlock the potential of Number Plate Recognition and revolutionize your image processing workflows with Azure's advanced AI capabilities.

Thanks for reading

Thank you very much for reading, I hope you found this article interesting and may be useful in the future. If you have any questions or ideas that you need to discuss, it will be a pleasure to be able to collaborate and exchange knowledge together.