Optimize Object Detection Model for High FPS On CPU

According to research, 90% of the models built by data scientists never make out to production. The reason for this is that the production environment has several constraints. It could be inference time or it could be hardware in some cases. Hence to make the model ready, we need to first think about the model itself which we are going to use for production. For deciding the model different experiments are carried out and then a trade-off between the hardware and accuracy is compared. The accuracy term used here doesn’t just only means accuracy but it includes any appropriate metric depending on the use case we are working on.

In this article, we will work on optimizing an object detection model and will increase its FPS on the CPU. For the purpose of the demo, we will take a pretrained SSD model that has been trained on the COCO dataset. The COCO dataset is a general dataset that has data for 80 classes.

Downloading the Pretrained Model

The model which we will optimize will be the SSD mobilenet v1 model as this model is lightweight and hence good for inferencing over the CPU. To download the model we need to visit tensorflow 1.x model-zoo and click on the linked text ssd_mobilenet_v1_coco or right-click on it and select copy link address and use wget to download the model. The tar file obtained has checkpoint files as well as the frozen model. In case there are just checkpoints, you will need to freeze the model. If you face difficulties in freezing the model you can comment in the comment section and I will love to help. It should be noted that tensorflow 2 doesn’t support graphs and hence for freezing tensorflow object detection 2.x API model, we will be required to follow a different approach. You can follow this notebook to freeze the TF1 API model if the frozen model with extension pb doesn’t exist in the tar file or for your own custom-trained models.

Benchmarking the Unoptimized Model

Before doing any improvement to a model it is necessary to benchmark it before and after doing improvement so that we can get the picture of the difference being made due to improvement. For doing the benchmark we will be using Google Colab.

First, let's install the Tensorflow object detection API

          #Cloning the Tensorflow API models repo.
!git clone https://github.com/tensorflow/models.git

#Navigating to models directory.
%cd models/research
# Compile protos.
!protoc object_detection/protos/*.proto --python_out=.
# Install TensorFlow Object Detection API.
!cp object_detection/packages/tf1/setup.py .
!python -m pip install --use-feature=2020-resolver .

#Performing the testing to check whether we have sucessfully installed the TF object detection API.
!python object_detection/builders/model_builder_tf1_test.py
 

As we can run shell commands in Google Colab, we run the above commands to install Tensorflow object detection API. Before running this you need to run the %tensorflow_version 1.x command in Colab as we are using the TF 1.15 version of object detection API.

Download and extract the SSD model and navigate to the models/research/object_detection directory

          !wget -O /content/ssd_mobilenet_v1_coco_2018_01_28.tar.gz http://download.tensorflow.org/models/object_detection/ssd_mobilenet_v1_coco_2018_01_28.tar.gz
!tar -xvf /content/ssd_mobilenet_v1_coco_2018_01_28.tar.gz
%cd /content/models/research/object_detection       

The extracted model exist at /content/ location of Colab.

Import necessary libraries

          import numpy as np
import os
import six.moves.urllib as urllib
import sys
import tarfile
import tensorflow as tf
import zipfile
import cv2
from collections import defaultdict
from io import StringIO
from matplotlib import pyplot as plt
from PIL import Image
import time
# This is needed since the notebook is stored in the object_detection folder.
sys.path.append("..")
from object_detection.utils import ops as utils_ops

# This is needed to display the images.
%matplotlib inline

from object_detection.utils import label_map_util
from object_detection.utils import visualization_utils as vis_util

Define some important parameters and helper functions for loading the model and making predictions

          # Defining the import param

num_classes = 80 # Change this value to the number of classes of the model
IMAGE_SIZE = (12, 8) # Output display size as you want


# Set paths to the trained model
PATH_TO_LABELS = '/content/models/research/object_detection/data/mscoco_label_map.pbtxt'
PATH_TO_CKPT = '/content/models/research/ssd_mobilenet_v1_coco_2018_01_28/frozen_inference_graph.pb'

# Set tensorflow graph
detection_graph = tf.Graph()
with detection_graph.as_default():
    od_graph_def = tf.GraphDef()
    with tf.gfile.GFile(PATH_TO_CKPT, 'rb') as fid:
        serialized_graph = fid.read()
        od_graph_def.ParseFromString(serialized_graph)
        tf.import_graph_def(od_graph_def, name='')

# Set categories
label_map = label_map_util.load_labelmap(PATH_TO_LABELS)
categories = label_map_util.convert_label_map_to_categories(
    label_map, max_num_classes=num_classes, use_display_name=True)
category_index = label_map_util.create_category_index(categories)

# Convert input image to a numpy array
def load_image_to_numpy(image):
    (im_width, im_height) = image.size
    return np.array(image.getdata()).reshape(
        (im_height, im_width, 3)).astype(np.uint8)

# Inference pipeline
def run_inference(image, graph):
    with graph.as_default():
        with tf.Session() as sess:
            # Get handles to input and output tensors
            ops = tf.get_default_graph().get_operations()
            all_tensor_names = {
                output.name for op in ops for output in op.outputs}
            tensor_dict = {}
            for key in [
                'num_detections', 'detection_boxes', 'detection_scores',
                'detection_classes', 'detection_masks'
            ]:
                tensor_name = key + ':0'
                if tensor_name in all_tensor_names:
                    tensor_dict[key] = tf.get_default_graph().get_tensor_by_name(
                        tensor_name)
            if 'detection_masks' in tensor_dict:
                # The following processing is only for single image
                detection_boxes = tf.squeeze(
                    tensor_dict['detection_boxes'], [0])
                detection_masks = tf.squeeze(
                    tensor_dict['detection_masks'], [0])
                # Reframe is required to translate mask from box coordinates to image coordinates and fit the image size.
                real_num_detection = tf.cast(
                    tensor_dict['num_detections'][0], tf.int32)
                detection_boxes = tf.slice(detection_boxes, [0, 0], [
                                           real_num_detection, -1])
                detection_masks = tf.slice(detection_masks, [0, 0, 0], [
                                           real_num_detection, -1, -1])
                detection_masks_reframed = utils_ops.reframe_box_masks_to_image_masks(
                    detection_masks, detection_boxes, image.shape[0], image.shape[1])
                detection_masks_reframed = tf.cast(
                    tf.greater(detection_masks_reframed, 0.5), tf.uint8)
                # Follow the convention by adding back the batch dimension
                tensor_dict['detection_masks'] = tf.expand_dims(
                    detection_masks_reframed, 0)
            image_tensor = tf.get_default_graph().get_tensor_by_name('image_tensor:0')

            # Run inference
            output_dict = sess.run(tensor_dict,
                                   feed_dict={image_tensor: np.expand_dims(image, 0)})

            # all outputs are float32 numpy arrays, so convert types as appropriate
            output_dict['num_detections'] = int(
                output_dict['num_detections'][0])
            output_dict['detection_classes'] = output_dict[
                'detection_classes'][0].astype(np.uint8)
            output_dict['detection_boxes'] = output_dict['detection_boxes'][0]
            output_dict['detection_scores'] = output_dict['detection_scores'][0]
            if 'detection_masks' in output_dict:
                output_dict['detection_masks'] = output_dict['detection_masks'][0]
    return output_dict
   

Load the image and preprocess it for benchmarking

          image=cv2.imread('/content/car1.jpg')[:,:,::-1]
image_copy=image.copy()
image_np=cv2.resize(image,(320,320),interpolation=cv2.INTER_CUBIC)
# Expand dimensions since the model expects images to have shape: [1, None, None, 3]
image_np_expanded = np.expand_dims(image_np, axis=0)
# Perform the interence
output_dict = run_inference(image_np, detection_graph)

Benchmark the model

          #Calculate the Throughput.
start_time=time.time()
for _ in range(100):
  output_dict = run_inference(image_np, detection_graph)
end_time=time.time()

total_time=end_time-start_time
print("FPS=",100/total_time)   

Here first we get the time at the start and then run the inference inside the for loop for around 100 times and then again get the time at the end of the loop then we calculate the total time. Finally, we calculate the throughput by dividing 100/total time.

After running this code I got the FPS=0.31. This value will be different if you run the above code on another hardware.

Disappointing huh?. Well, that's where OpenVino comes into the picture which optimizes the models for Intel CPUs.

How Does Openvino Optimize the Model

Before jumping into the theory let’s understand a bit about how the openvino optimizes the model so that we can get the picture of what the openvino is doing under the hood. Although this theory is not the prerequisite to do the model optimization. Below operations are performed by openvino at the neural network level.

Batch Normalization and Scale shift decomposition

In this stage, the batch normalization layer is decomposed to Mul → Add → Mul → Addsequence and scale shift layer is decomposed to Mul → Add layers sequence.

Linear operations merge

In this stage, the Mul and Add operations are merged into a single Mul → Add instance. For example, if there is a BatchNormalization → ScaleShift the sequence in the topology, it is replaced with Mul → Add in the first stage. In the next stage, the latter is replaced with a ScaleShift layer if there is no available Convolution or FullyConnected layer to fuse into next.

Linear operations fusion

In this stage, the tool fuses Mul and Add operations to Convolution or FullyConnected layers. Notice that it searches for Convolution and FullyConnected layers both backward and forward in the graph (except for Add the operation that cannot be fused to Convolution layer in forward direction).

Optimizing Our SSD Model Using Openvino

Installing the openvino on our system

Below commands will take care of the installations on the ubuntu 18.04 system. For other systems please refer to the openvino installation documentation mentioned in the references.

          !sudo apt-get install pciutils
!sudo apt-get update -y && sudo apt-get install -y cpio
!cat /etc/os-release
!wget https://registrationcenter-download.intel.com/akdlm/irc_nas/17662/l_openvino_toolkit_p_2021.3.394.tgz
!tar -xvzf l_openvino_toolkit_p_2021.3.394.tgz
%cd /content/l_openvino_toolkit_p_2021.3.394
!bash install.sh
%cd /opt/intel/openvino_2021/install_dependencies
!sudo -E ./install_openvino_dependencies.sh
!source /opt/intel/openvino_2021/bin/setupvars.sh
%cd /opt/intel/openvino_2021/deployment_tools/model_optimizer/install_prerequisites
!sudo ./install_prerequisites.sh
!ls  /opt/intel/openvino_2021/deployment_tools/open_model_zoo/tools/downloader
        

! sign is used to run Linux commands in google colab notebook so we need to remove it when we want to install it on our local computer. But in my case, since I am doing it in colab I don’t have to remove it.

Download the frozen mobilenet model and extract it

          !gdown --id 12yUA2oe0ES8yE0_5UWmHKb6s1Lp-ARGg
!unzip /content/frozen_graph_mobilenet_model.zip
        

Optimize the model

The openvino optimizes the model and generates an intermediate representation from it which can be parsed by the OpenVino’s inference model and prediction could be made. The IR format consists of two files. One XML file contains the architectural information of the model while the other BIN file contains the weight of the model.

          !python /opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/mo_tf.py \
--input_shape=[1,300,300,3] \
--input_model=/content/content/inference_graph/frozen_inference_graph.pb \
--transformations_config /opt/intel/openvino_2021.3.394/deployment_tools/model_optimizer/extensions/front/tf/ssd_support_api_v1.15.json \
--tensorflow_object_detection_api_pipeline_config /content/content/inference_graph/pipeline.config \
--reverse_input_channels --output_dir /content/
        

The model optimizer script mo_tf.py is used for the optimization of tensorflow models. This scripts resides in INSTALLATION_DIRECTORY/openvino_2021.3.394/deployment_tools/model_optimizer/. Where INSTALLATION_DIRECTORY is the directory where the openvino has been installed. In my case, the installation directory was /opt/intel/ which is most common in Linux-based systems.

Now let's understand the arguments

input_shape: This specifies the input dimension of the image our model will accept after the optimization. Some models are trained with fixed input shapes and hence manually specifying the input shape might result in model conversion errors. Some models such as Yolo support multiple sizes and hence multiple image sizes can be specified. In our case the mobile model accepts 300x300 size hence we specify it. When this argument is not specified the openvino takes the default input shape of the model.

input_model: This specifies the path to the frozen pb file. In my case it was /content/content/inference_graph/frozen_inference_graph.pb

transformations_config: This is one of the most important parameters in this command. This argument specifies the path to the transformation file. This transformation file contains the transformations which should be applied to the model to optimize it. Ex- For a certain model A, the layers 5 and 6 could be merged together to boost the speed hence that operation might be declared in these configurations by the Intel team. Since I am using the mobilenet model trained using tensorflow object detection API hence the correct transformation file must be specified for it.

Please find the right name of the configuration file from this documentation link. In my case, the file is ssd_support_api_v1.15.json as I am using Tensorflow version 1.15. All these configuration files for different versions of Tensorflow object detection API can be found at INSTALLATION_DIRECTORY/openvino_2021.3.394/deployment_tools/model_optimizer/extensions/front/tf/

tensorflow_object_detection_api_pipeline_config: This specifies the path to the configuration file used to train the model. In my case it was /content/content/inference_graph/pipeline.config

reverse_input_channels: Whether to reverse the input channel format i.e. channel last format to channel first format.

output_dir: The path where to generate the optimized model's files. In our case, it is at /content/.

After running this command there will be two files frozen_inference_graph.xml and frozen_inference_graph.bin in the /content/ directory. Now, these files can be loaded by the OpenVino inference engine and can be used to make predictions.

Benchmarking the Optimized Model

First, we need to install the openvino inference engine which can be simply installed using the pip install openvino command.

Importing necessary libraries

          from openvino.inference_engine import IENetwork
from openvino.inference_engine import IECore
import cv2
import numpy as np
import matplotlib.pyplot as plt
import tensorflow as tf
import time
%matplotlib inline      

Create the inference engine object

ie = IECore()
# Read IR
net = IENetwork(model='/content/frozen_inference_graph.xml', weights='/content/frozen_inference_graph.bin')
# check net.inputs.keys(), net.outputs
input_blob = next(iter(net.inputs))
out_blob = next(iter(net.outputs))
# exec_net = plugin.load(network=net)
exec_net = ie.load_network(network=net, device_name="CPU")
del net

Here we have loaded the model in IR format and created the inference engine object. The inputs attribute of the IENetwork class gives the input name and outputs give the output names of the model. The exec_net is the object which contains the instance of the inference engine.

Let’s open the image, preprocess it, and convert it into channel first format

          #Load the image and resize to 300x300 and expand dims.
img=cv2.imread('/content/car1.jpg')[:,:,::-1]
img=cv2.resize(img,(300,300))
img=np.expand_dims(img,axis=0)

#Let's flip over the channel.
img=np.rollaxis(img, 3, 1)

As the model as by default flipped the colored channels. i.e. the size changed from (1,300,300,3) to (1,3,300,300) i.e 3 which represents the number of channels is reversed.

Finally benchmark the model

          start_time=time.time()

for _ in range(100):
  res = exec_net.infer(inputs={'image_tensor': img})
end_time=time.time()
tt=end_time-start_time

print('FPS=',100/tt)

The FPS of this model turned to be around 28.98 isn’t that's great. It's real-time speed.

Further Work

The model can be further optimized for speed using quantization which converts the neural network weights to some low-end data formats like float16 or int8. But since these operations are not supported on my CPU I didn’t go for it. Optimization could be done in a similar way for Intel GPUs or VPUs.

References and Further Readings

Archive — OpenVINO™ documentation

docs.openvino.ai

Congrats you have just learned one of the most important skills in the life of a Deep Learning/Machine Learning Engineer. If you have any questions feel free to drop it in comments section. Thanks for taking time to read the blog.