Rotate The Boundingbox According To Angle From Azure ReadResults - OCR API/Read API/Form Recognizer API

Introduction

 
If you have worked with Azure Cognitive Service API's like OCR API, Read API, or Form Recognizer API, you might have come across boundingBox in the readResults of the response. If the input you have given is slightly tilted, the response will also be tilted. The response also contains the angle by which the input page is tilted. To manipulate the results we sometimes need to rotate the boundingBox in the response with the tilted angle to do some operations.
 
Here we are looking at how this rotation can be done. Here is a Receipt from KSEB captured in an angle. We first give it to Azure Read API to get the JSON output.
 
Sample Input is given to Azure Read API
 
 
Sample Output from Azure Read API
  1. {  
  2.    "status""succeeded",  
  3.    "createdDateTime""2020-12-21T15:13:55Z",  
  4.    "lastUpdatedDateTime""2020-12-21T15:13:56Z",  
  5.    "analyzeResult": {  
  6.        "version""3.0.0",  
  7.        "readResults": [  
  8.           {  
  9.                "page": 1,  
  10.                "angle": 7.6307,  
  11.                "width": 445,  
  12.                "height": 1242,  
  13.                "unit""pixel",  
  14.                "lines": [  
  15.                    ....  
  16.                    ....  
  17.                    ....  
  18.                      
  19.               {  
  20.                        "boundingBox": [  
  21.                            185,  
  22.                            91,  
  23.                            366,  
  24.                            113,  
  25.                            364,  
  26.                            131,  
  27.                            183,  
  28.                            109  
  29.                       ],  
  30.                        "text""Demand/Disconnection Notice",  
  31.                        "words": [  
  32.                           {  
  33.                                "boundingBox": [  
  34.                                    186,  
  35.                                    92,  
  36.                                    319,  
  37.                                    109,  
  38.                                    318,  
  39.                                    126,  
  40.                                    184,  
  41.                                    109  
  42.                               ],  
  43.                                "text""Demand/Disconnection",  
  44.                                "confidence": 0.751  
  45.                           },  
  46.                           {  
  47.                                "boundingBox": [  
  48.                                    323,  
  49.                                    109,  
  50.                                    366,  
  51.                                    114,  
  52.                                    365,  
  53.                                    131,  
  54.                                    321,  
  55.                                    126  
  56.                               ],  
  57.                                "text""Notice",  
  58.                                "confidence": 0.907  
  59.                           }  
  60.                       ]  
  61.                   },  
  62.                    ....  
  63.                    ....  
  64.                    ....  
  65.               ]  
  66.           }  
  67.       ]  
  68.   }  
  69. }  
The above JSON response from Azure Read API says the Receipt is tilted by an angle of 7.6307 degrees. In order to do some manipulations based on the coordinates in boundingBox we need to rotate this by the angle 7.6307.
 

Rotate a point p by n degrees with respect to origin o

 
Suppose we need to rotate a point by n degrees with respect to origin o we may call a custom function describes below by calling it:
  1. rotatedPointoint=rotate(p,o,n)   
The custom function rotate is shown below, which also needs numpy to work:
  1. import numpy as np  
  2. def rotate(point, origin, degrees):  
  3.    radians = np.deg2rad(degrees)  
  4.    x, y = point  
  5.    offset_x, offset_y = origin  
  6.    adjusted_x = (x - offset_x)  
  7.    adjusted_y = (y - offset_y)  
  8.    cos_rad = np.cos(radians)  
  9.    sin_rad = np.sin(radians)  
  10.    qx = offset_x + cos_rad * adjusted_x + sin_rad * adjusted_y  
  11.    qy = offset_y + -sin_rad * adjusted_x + cos_rad * adjusted_y  
  12.    return qx, qy  

Building a function for correcting the angle of coordinates boundingBox in response

 
The boundingBox contains 8 points as a list in the order:
  1. ["left-top-x","left-top-y","right-top-x",right-top-y","right-bottom-x","right-bottom-y","left-bottom-x","left-bottom-y"]   
So to rotate a boundingBox we may loop through the list by incrementing by 2 on each cycle as below where:
  1. angle=analysis["analyzeResult"]["readResults"][index]["angle"]  
Here angle=7.6307 and I try to rotate the picture by origin so that is equal to (0,0)
  1. for ind in range(0, 7, 2):  
  2. bBox[ind], bBox[ind + 1] = rotate((bBox[ind], bBox[ind + 1]), (0, 0),angle )   
Now we know how to rotate a bounding box. Let's move to how we can rotate all boudingBox's corrponding to lines in readResult as a whole.Nesting the above code snippet inside a loop of lines which is inside a loop of pages wiil make you achive the same. Below is the code for the same:
  1. def correctAngle(analysis):  
  2.    for page in analysis["analyzeResult"]["readResults"]:  
  3.   for line in page['lines']:  
  4.         bBox = line['boundingBox']  
  5.            for ind in range(0, 7, 2):  
  6.           bBox[ind],bBox[ind+1]=rotate((bBox[ind], bBox[ind + 1]),(0, 0),angle)  
  7.                line['boundingBox'] = bBox  
  8.    return analysis  
Now we have rotated the boudingBox's corresponding to lines but we know there are boundingBox corresponding to words too so in order to achieve rotation on those bounding boxes you may try the same logic in the loop. We know we need to rotate only pages which have an angle so to optimize the code we check if the angle is a non zero value before entering the rotation function. Below is the code for the same.
  1. def correctAngle(analysis):  
  2.    for page in analysis["analyzeResult"]["readResults"]:  
  3.        if page["angle"] != 0:  
  4.            for line in page['lines']:  
  5.                bBox = line['boundingBox']  
  6.                for ind in range(0, 7, 2):  
  7.                    bBox[ind], bBox[ind + 1] = rotate((bBox[ind], bBox[ind + 1]), (0, 0), page["angle"])  
  8.                line['boundingBox'] = bBox  
  9.                for word in line['words']:  
  10.                    wbBox = word['boundingBox']  
  11.                    for ind in range(0, 7, 2):  
  12.                        wbBox[ind], wbBox[ind + 1] = rotate((wbBox[ind], wbBox[ind + 1]), (0, 0), page["angle"])  
  13.                    word['boundingBox'] = wbBox  
  14.             page["angle"]=0  
  15.    return analysis  
Sample Output
 
Output when process the above JSON response with the above function is:
  1. {  
  2.    "status""succeeded",  
  3.    "createdDateTime""2020-12-21T15:13:55Z",  
  4.    "lastUpdatedDateTime""2020-12-21T15:13:56Z",  
  5.    "analyzeResult": {  
  6.        "version""3.0.0",  
  7.        "readResults": [  
  8.           {  
  9.                "page": 1,  
  10.                "angle": 0,  
  11.                "width": 445,  
  12.                "height": 1242,  
  13.                "unit""pixel",  
  14.                "lines": [  
  15.                    ....  
  16.                    ....  
  17.                    ....  
  18.                      
  19.                   {  
  20.                        "boundingBox": [  
  21.                            202.42927798617825,  
  22.                            39.094595713404814,  
  23.                            382.83721732032967,  
  24.                            12.675371175104381,  
  25.                            385.64576445111106,  
  26.                            30.567046977392415,  
  27.                            205.23782511695967,  
  28.                            56.98627151569286  
  29.                       ],  
  30.                        "text""Demand/Disconnection Notice",  
  31.                        "words": [  
  32.                           {  
  33.                                "boundingBox": [  
  34.                                    203.65723612684363,  
  35.                                    39.796107512859024,  
  36.                                    336.4417810450956,  
  37.                                    21.187920313329045,  
  38.                                    339.95183997533115,  
  39.                                    37.85163797495165,  
  40.                                    206.2025600870195,  
  41.                                    56.72304834508725  
  42.                               ],  
  43.                                "text""Demand/Disconnection",  
  44.                                "confidence": 0.751  
  45.                           },  
  46.                           {  
  47.                                "boundingBox": [  
  48.                                    340.3007209253349,  
  49.                                    20.135027630906585,  
  50.                                    383.1004404909352,  
  51.                                    13.640106145164204,  
  52.                                    386.6104994211709,  
  53.                                    30.30382380678678,  
  54.                                    342.84604488551065,  
  55.                                    37.061968463134804  
  56.                               ],  
  57.                                "text""Notice",  
  58.                                "confidence": 0.907  
  59.                           }  
  60.                       ]  
  61.                   },  
  62.                  
  63.                    ....  
  64.                    ....  
  65.                    ....  
  66.               ]  
  67.           }  
  68.       ]  
  69.   }  
  70. }  
Final code
  1. import numpy as np  
  2. def rotate(point, origin, degrees):  
  3.    radians = np.deg2rad(degrees)  
  4.    x, y = point  
  5.    offset_x, offset_y = origin  
  6.    adjusted_x = (x - offset_x)  
  7.    adjusted_y = (y - offset_y)  
  8.    cos_rad = np.cos(radians)  
  9.    sin_rad = np.sin(radians)  
  10.    qx = offset_x + cos_rad * adjusted_x + sin_rad * adjusted_y  
  11.    qy = offset_y + -sin_rad * adjusted_x + cos_rad * adjusted_y  
  12.    return qx, qy  
  13. ​  
  14. def correctAngle(analysis):  
  15.    for page in analysis["analyzeResult"]["readResults"]:  
  16.        if page["angle"] != 0:  
  17.            for line in page['lines']:  
  18.                bBox = line['boundingBox']  
  19.                for ind in range(0, 7, 2):  
  20.                    bBox[ind], bBox[ind + 1] = rotate((bBox[ind], bBox[ind + 1]), (0, 0), page["angle"])  
  21.                line['boundingBox'] = bBox  
  22.                for word in line['words']:  
  23.                    wbBox = word['boundingBox']  
  24.                    for ind in range(0, 7, 2):  
  25.                        wbBox[ind], wbBox[ind + 1] = rotate((wbBox[ind], wbBox[ind + 1]), (0, 0), page["angle"])  
  26.                    word['boundingBox'] = wbBox  
  27.    return analysis  

Conclusion

 
This blog was about how to rotate the boundingBox in resulted response with the tilted angle. I hope you were able to achieve the same.
 
To know how to Build a Flask application and getting stared with Azure Cognitive Services go to the article series starting at "Python Flask App And Azure Cognitive Services Read API - Render HTML Page And File Transfer Between Client And Server"