Detecting Objects With Precision: Polygonal Bounding Box-Based Object Detection

Object detection is a crucial task in computer vision that involves identifying and localizing objects of interest in an image or video. It is used in a wide range of applications, including surveillance, autonomous driving, robotics, and medical imaging. The goal of object detection is to not only detect the presence of objects but also accurately localize them in the image and label them with the correct category. In recent years, significant progress has been made in object detection algorithms, largely driven by advances in deep learning. Today, various techniques and models are available for object detection, including bounding box-based methods and polygonal object detection. In this blog post, we will focus on polygonal object detection and explore its advantages over traditional bounding box-based methods. In the later part of the blog, we will also train a deep-learning model to do the same.

Advantages of Polygonal Bounding Box-Based Object Detection

While traditional rectangular-based object detection algorithms have been successful in identifying objects in images, they have some limitations. One of the primary challenges with using rectangular bounding boxes is that they do not accurately capture the shapes of objects with irregular geometries, such as buildings, roads, and other man-made structures. This can lead to inaccuracies in object localization and misclassification. On the other hand, polygonal bounding boxes provide a more accurate representation of the shape of an object, which can lead to better localization and classification performance. In addition, polygonal object detection can be particularly effective in detecting objects with complex boundaries, such as natural landscapes or geological features. By using a polygonal representation, object detection algorithms can more precisely delineate the contours of these objects and reduce false positive detections.

Annotating Objects for Polygonal Object Detection

For performing polygonal or rotated object detection the annotation needs to be created in the format where the boxes are rotated hence there is a need for an additional angle parameter that can be used to rotate the bounding boxes. The latest version of the labelimg tool provides such an option where we can draw rotated bounding boxes. These boxes are drawn in a similar fashion as the rectangular boxes but can be rotated to fit the object. Depending on the rotations made the angle is stored in the annotation file. The below figure demonstrates such annotation with the labelimg version 2.

It can be noted in the above figure that the number plate of the vehicle is slanted and hence it is annotated with the rotated box.

Modifications Required in Object Detectors for Polynomial Object Detection

Having understood how data is created for rotated object detection let's understand what changes need to be made in regular object detectors like YOLO V5 for doing rotated bounding box detection.

Model Architecture:

By default, the YOLO V5 predicts the xmin,ymin,width,height,score and cls. xmin and ymin represent the top left coordinate of the bounding box. All of these are normalized in the range 0 to 1. With the top left corner coordinates and width and height information, the bottom right coordinate is estimated. However for drawing rotated boxes either we have to predict the coordinates of all four corners or we can predict the angle and use it to rotate the rectangular bounding box. The latter is much easier and most of the annotation tools support the annotation in the latter format hence the head of the YOLO V5 needs to be modified to incorporate the angle prediction. The rest of the network can be the same.

Anchor boxes:

Default anchor boxes of YOLO V5 are in rectangular format and hence they can’t fit well to the object by default and thus be replaced with polygonal equivalents. In the official implementation of YOLO V5, the anchors are generated by the KMeans algorithm. Hence that portion needs to be modified to generate the new polygonal anchors.

IOU and Loss Function

The model is optimized to better localize the objects hence at every iteration during the training process IOU loss is calculated between the predicted boxes and the ground truth boxes. During inference the IOU is used to remove the extra predicted boxes that have a certain overlap with others as they would be redundant hence the IOU is one of the most important parts that needs to be modified.

Fortunately, all of these have been done by XinzeLee. He has also written custom CUDA kernels for speeding up the IOU calculation on GPUs which we would use in this blog for training and doing the inference. He had also listed the files he has modified from the original YOLO v5 code base so anyone who is interested in diving deep into the modifications can take a look from the repo’s readme.

Training a Polygonal YOLO V5 Model

In this section, we would step-by-step train a model on the UCAS50 dataset which has an annotation in rotated boxes format. This dataset contains two objects, airplanes and cars and are aerial images. So let’s dive in.

Setting up the Environment

This repo requires some specific versions of Python and system libraries for running the code since there are custom CUDA kernels written for speeding up polygonal IOU calculation on GPU. In my experiments, I found CUDA version 11.1 and Cudnn version 8.0.4 to be working well. I was using the Google Colab environment to run the code and the default version of CUDA was 12.0 while the torch was 2.0 hence both need to be removed and CUDA version 11.1 and torch 1.10 for CUDA 11.1 needs to be installed, so let's go ahead and install them.

          !sudo apt-get purge *nvidia*
!sudo apt remove --autoremove nvidia-*
!sudo rm /etc/apt/sources.list.d/cuda*
!sudo apt remove --autoremove nvidia-cuda-toolkit
!sudo apt-get autoremove && sudo apt-get autoclean
!sudo rm -rf /usr/local/cuda*

!sudo apt-get update
!sudo apt-get upgrade

!sudo apt-get install g++ freeglut3-dev build-essential libx11-dev libxmu-dev libxi-dev libglu1-mesa libglu1-mesa-dev

!sudo add-apt-repository ppa:graphics-drivers/ppa
!sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
!echo "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" | sudo tee /etc/apt/sources.list.d/cuda.list
!sudo apt-get update

!sudo apt-get -o Dpkg::Options::="--force-overwrite" install cuda-11-1 cuda-drivers

!echo 'export PATH=/usr/local/cuda-11.1/bin:$PATH' >> ~/.bashrc
!echo 'export LD_LIBRARY_PATH=/usr/local/cuda-11.1/lib64:$LD_LIBRARY_PATH' >> ~/.bashrc
!source ~/.bashrc
!sudo ldconfig
      

By running the above commands we can install CUDA version 11.1 in Ubuntu 18.04.

Now let’s install Cudnn.

          !wget -O cudnn-11.1-linux-x64-v8.0.4.30.tgz https://developer.download.nvidia.com/compute/machine-learning/cudnn/secure/8.0.4/11.1_20200923/cudnn-11.1-linux-x64-v8.0.4.30.tgz?lafc32BPTuk0EeFTK8C2dAEcGd8PFshfAUEHz89fdiHORbCAN-gLHPmAzwlSy8m6u8IQg2Fo-HcycvpMSjUeL0gWI88jT6OM3x4K6shsiwpt1Pa02NrMGT22Wmqxz_xjhYQdmncGKc_J_bRHrkxNFtRa0giIpsJFundT226ZX6vglBZhfO4_CJJmWjKwmhQEh6C3slnME5NmC9Q=&t=eyJscyI6ImdzZW8iLCJsc2QiOiJodHRwczovL3d3dy5nb29nbGUuY29tLyJ9
!tar -xzvf /content/cudnn-11.1-linux-x64-v8.0.4.30.tgz

!sudo cp -P cuda/include/cudnn*.h /usr/local/cuda-11.1/include
!sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda-11.1/lib64/
!sudo chmod a+r /usr/local/cuda-11.1/lib64/libcudnn*

!sudo cp -P cuda/include/cudnn*.h /usr/local/cuda-11.1/include
!sudo cp -P cuda/lib64/libcudnn* /usr/local/cuda-11.1/lib64/
!sudo chmod a+r /usr/local/cuda-11.1/lib64/libcudnn*
        

In the above shell, we need to pay attention to the download link for Cudnn. It might not work for you if you directly run the above command because that link is temporarily generated after you head to Nvidia's official download page and sign in with your id and download Cudnn. For running the above command you might need to login to the download page and copy the link once the download starts in browsers like Chrome or Firefox. If you face issues please let me know in the comment section. Alternatively, you can also follow any other tutorial to install CUDA 11.1 and compatible Cudnn.

Python 3.8 Installation

Python 3.10 was the default version in Colab at the time I ran the code and the wheel file for most of the libraries of the older version is not available for Python 3.10 hence we need to roll back to Python 3.8 or we could build the wheel file for all these libraries. Doing the former will be simpler and quicker as building libraries will require a lot of time. Hence let's install Python 3.8 as well as its developer tools.

          !sudo apt-get install python3.7
!sudo apt-get update -y
!sudo update-alternatives --install /usr/bin/python3 python3 /usr/bin/python3.7 1
!sudo update-alternatives --config python3 #Here itmight asks for version choice in colab to select.


!sudo apt install python3-pip
!sudo apt install python3.8-distutils
!sudo apt install python3.8-dev
       

Here we installed development tools for Python as well which would be essential for installing libraries that require building.

Installing required Python libraries

For installing the required we will use the requirement file of the repo. So let's clone the repo and use the file needed to install the dependencies. Here I will clone from my own copy of the original repo as in the original repo there are some bugs associated which I have fixed and uploaded to my repo. Besides bug fixing I haven’t made changes to the code hence Mr.XinzeLee owns the full credit to the code.

          !git clone https://github.com/VikasOjha666/PolygonYOLOv5
%cd /content/PolygonObjectDetection/polygon-yolov5
# install python package requirements
!pip install -r requirements.txt

!pip install torch==1.10.1+cu111 torchvision==0.11.2+cu111 torchaudio==0.10.1 -f https://download.pytorch.org/whl/cu111/torch_stable.html
!pip install setuptools==59.5.0
!pip install numpy==1.20.3

Installing IOU CUDA library

          %cd utils/iou_cuda
!python setup.py install
# cd back to polygon-yolov5 folder
%cd .. 
%cd ..
        

Here we are building and installing the CUDA-based polygon IOU calculation library.

Configurations to Train the Model

We need to make certain configuration changes to the already available configuration in the repository for training on our own dataset. Taking the example of the default configuration only I am trying to explain the required changes below. For more questions related to training, you can comment in the comment section.

Generating anchors

For polygonal bounding boxes, the anchors also need to be polygonal. The same can be achieved by modifying the KMeans-based auto anchors in the original YOLO v5 repo which has been already done by XinzLee so let's use that and generate anchors.

          from utils.autoanchor import polygon_kmean_anchors

nl = 3 # number of anchor layers
na = 3 # number of anchors
img_size = 1024 # image size for training and testing

datacfg = "data/polygon_ucas.yaml"
anchors = polygon_kmean_anchors(datacfg, n=nl*na, gen=3000, img_size=img_size)
print(anchors.reshape(nl, na*2).astype(int))
       

This would print anchors in the form of numpy arrays.

[[ 23 44 37 37 52 38] [ 45 50 55 54 66 66] [ 88 88 111 111 160 155]]

These values need to be updated in polygon_yolov5s_ucas.yaml.

%%writefile /content/PolygonYOLOv5/polygon-yolov5/models/polygon_yolov5s_ucas.yaml
# parameters
nc: 2  # number of classes
depth_multiple: 0.33  # model depth multiple
width_multiple: 0.50  # layer channel multiple

# anchors
# Generated through utils.polygon_kmean_anchors
# [10,13, 16,30, 33,23] represents 3 predefined anchors of x1, y1, x2, y2, x3, y3
# there are three anchor layers: layer 1 [31, ..., 31]; layer 2 [46, ..., 74]; layer 3 [94, ..., 151]
anchors:
# For UCAS dataset with imgsz 1024
  - [31, 30, 28, 49, 50, 31]  # P3/8
  - [46, 45, 58, 58, 74, 74]  # P4/16
  - [94, 94, 115, 115, 151, 151]  # P5/32
# For Coco dataset with imgsz 640
#   - [ 29,  34,  60,  60, 127,  69,  80, 127, 139, 127]  # P3/8
#   - [134, 207, 237, 118, 214, 196, 317, 198, 193, 337]  # P4/16
#   - [499, 162, 345, 298, 548, 288, 417, 453, 618, 443]  # P5/32

# YOLOv5 backbone
backbone:
  # [from, number, module, args]
  [[-1, 1, Focus, [64, 3]],  # 0-P1/2
   [-1, 1, Conv, [128, 3, 2]],  # 1-P2/4
   [-1, 3, C3, [128]],
   [-1, 1, Conv, [256, 3, 2]],  # 3-P3/8
   [-1, 9, C3, [256]],
   [-1, 1, Conv, [512, 3, 2]],  # 5-P4/16
   [-1, 9, C3, [512]],
   [-1, 1, Conv, [1024, 3, 2]],  # 7-P5/32
   [-1, 1, SPP, [1024, [5, 9, 13]]],
   [-1, 3, C3, [1024, False]],  # 9
  ]

# YOLOv5 head
head:
  [[-1, 1, Conv, [512, 1, 1]],
   [-1, 1, nn.Upsample, [None, 2, 'nearest']],
   [[-1, 6], 1, Concat, [1]],  # cat backbone P4
   [-1, 3, C3, [512, False]],  # 13

   [-1, 1, Conv, [256, 1, 1]],
   [-1, 1, nn.Upsample, [None, 2, 'nearest']],
   [[-1, 4], 1, Concat, [1]],  # cat backbone P3
   [-1, 3, C3, [256, False]],  # 17 (P3/8-small)

   [-1, 1, Conv, [256, 3, 2]],
   [[-1, 14], 1, Concat, [1]],  # cat head P4
   [-1, 3, C3, [512, False]],  # 20 (P4/16-medium)

   [-1, 1, Conv, [512, 3, 2]],
   [[-1, 10], 1, Concat, [1]],  # cat head P5
   [-1, 3, C3, [1024, False]],  # 23 (P5/32-large)

   [[17, 20, 23], 1, Polygon_Detect, [nc, anchors]],  # POLYGON: Polygon_Detect(P3, P4, P5) [17, 20, 23] represents the layer index
  ]

    

Next, we set the hyperparameters.

          %%writefile /content/PolygonYOLOv5/polygon-yolov5/data/hyp.ucas.yaml
# Hyperparameters for polygon training from scratch
# python polygon_train.py --batch 40 --cfg polygon_yolov5s.yaml --weights '' --data polygon_coco.yaml --img 640 --epochs 100
# See tutorials for hyperparameter evolution https://github.com/ultralytics/yolov5#tutorials


lr0: 0.01  # initial learning rate (SGD=1E-2, Adam=1E-3)
lrf: 0.2  # final OneCycleLR learning rate (lr0 * lrf)
momentum: 0.937  # SGD momentum/Adam beta1
weight_decay: 0.0005  # optimizer weight decay 5e-4
warmup_epochs: 3.0  # warmup epochs (fractions ok)
warmup_momentum: 0.8  # warmup initial momentum
warmup_bias_lr: 0.1  # warmup initial bias lr
box: 0.05  # box loss gain
cls: 0.5  # cls loss gain
cls_pw: 1.0  # cls BCELoss positive_weight
obj: 1.0  # obj loss gain (scale with pixels)
obj_pw: 1.0  # obj BCELoss positive_weight
iou_t: 0.10  # IoU training threshold
anchor_t: 4.0  # anchor-multiple threshold
# anchors: 3  # anchors per output layer (0 to ignore)
fl_gamma: 0.0  # focal loss gamma (efficientDet default gamma=1.5)

# The following to test Polygon_LoadImagesAndLabels
hsv_h: 0.015  # image HSV-Hue augmentation (fraction)
hsv_s: 0.7  # image HSV-Saturation augmentation (fraction)
hsv_v: 0.4  # image HSV-Value augmentation (fraction)
degrees: 30.0  # image rotation (+/- deg)
translate: 0.1  # image translation (+/- fraction)
scale: 0.0  # image scale (+/- gain)
shear: 5.0  # image shear (+/- deg)
perspective: 0.0005  # image perspective (+/- fraction), range 0-0.001
flipud: 0.5  # image flip up-down (probability)
fliplr: 0.5  # image flip left-right (probability)
mosaic: 0.0  # image mosaic (probability) # This may be too strong property for polygon! Causing many noisy labels.
mixup: 0.0  # image mixup (probability) # This may be too strong property for polygon!
        

Next, we update the path to the dataset, classes, etc.

          %%writefile /content/PolygonYOLOv5/polygon-yolov5/data/polygon_ucas.yaml
# UCAS-AOD dataset http://cocodataset.org
# Default dataset location is next to Polygon-YOLOv5:
#   /parent_folder
#     /UCAS50
#     /polygon-yolov5



# train and val data as 1) directory: path/images/, 2) file: path/images.txt, or 3) list: [path1/images/, path2/images/]
train: ../UCAS50/train.txt  # 40 images
val: ../UCAS50/val.txt  # 9 images

# number of classes
nc: 2

# class names
names: ['plane', 'car']
        

Training the model

          !python polygon_train.py --weights polygon-yolov5s-ucas.pt --cfg polygon_yolov5s_ucas.yaml \
     --data polygon_ucas.yaml --hyp hyp.ucas.yaml --img-size 1024 \
     --epochs 3 --batch-size 12 --noautoanchor --polygon --cache

        

Finally, we train the model by running a polygonal_train.py file.

Running detection

For running detection, one can use polygon_detect.py or can modify the detect function in it for a particular use case. In this tutorial, we just run the file.

Above image was the prediction after training the model for 3 epochs. It can be observed that the cars being detected in red polygonal boxes.

References and Further Readings

Link to the notebook:

Congrats you have just learned an amazing concept in object detection that can solve a lot of problems. Thanks for reading for this blog. If you have question please feel free to add in the comments sections.