Binary Neural Networks: A Game Changer in Machine Learning

Whenever we think of neural networks, GPUs automatically pop into our minds because they are very compute-heavy and often require powerful GPUs to train them. Most of the state-of-the-art models require GPUs even for fast inference which makes them not so cost-friendly for the end users. Often while deploying neural networks quantization is used widely to speed up the inference.

In quantization, the precision of weights and activations are reduced from float32 to float16 in the case of GPUs and int8 in the case of CPUs this reduction in precision not only saves the memory but also speeds up the network as reducing precision also reduces the memory access time. But still, the weights have int8 values.

In this blog, we would explore a new type of neural network called Binary Neural Networks which stores weights in binary values ie. 1 and -1 which is also termed 1-bit quantization. Due to 1-bit quantization, these networks are extremely efficient to train and inference and are suitable to be deployed on embedded devices and microcontrollers.

Some Important Theoretical Concepts

Any process that happens in a computer can be classified into either computation or memory access operation. Performing computation is costly but performing memory access is even more energy-demanding. That is the reason why performing calculations at high precision is often slower in comparison to doing the same with low precision hence quantization in neural networks helps in improving their inference speed as well the size. In binary neural networks, the weights and activations are converted into binary values i.e -1 and 1. Let's understand how it is done and several other improvements in detail.

Binarizing Weights and Activations

The weights of a neural network in general are initialized with very small values as we know. In the case of binary neural networks, these are binarized with the help of a suitable binarizing function. Below are the two functions used to binarize the weights and activation.

The sign function is given by:

And stochastic binarization is given by:

The sign function is a deterministic function while the stochastic binarization is non-deterministic. The sign function might appear to be more appealing but it is hard to implement in practice due to the fact that it requires random bit generation during quantization hence sign function is preferred mostly with the exception of the case of certain activation in some experiments.

Problems Associated With Binarization

While converting the weights and activation to binary values certainly has a lot of benefits but there is a huge problem that occurs if we think of keeping the whole training and inference process in binary. The problem is that the standard stochastic gradient descent algorithm doesn’t work with binary values as updating steps in gradient descent happen in very small values and real numbers are required for such small values. Further, adding noises to the weights and activations helps the neural networks to generalize better as have explained in certain research. So the gradient calculation needs to be modified in order to solve this issue.

Additionally using the sign function to binarize the weights and activation can have another issue i.e. zero grad values as the derivate of the sign function at all times is 0.

To solve the problem associated with gradient descent and sign function. The researchers have used a technique called straight-through estimator which was introduced in Hinton’s 2012 lectures that take into account the saturation effect introduced by the sign function and does use deterministic rather than a stochastic sampling of the bit. Consider the sign function quantization

and assume that an estimator gq of the gradient ∂C/∂q has been obtained (with the straight-through estimator when needed). Then, our straight-through estimator of ∂C /∂r is simply

Note that this preserves the gradient’s information and cancels the gradient when r is too large. Not canceling the gradient when r is too large significantly worsens the performance. The derivative of the lr≤=1 can also be thought of as hard tanh which is the following piece-wise linear activation function:

Well, all of these seem a bit complicated, right? But at this point, we need to understand that it uses conditions to prevent the grad from reaching 0 or a very high value which can affect the training process.

Replacing Multiplication Operations With Binary Shift Operations

Because the weights are in binary values many multiplication operations can be replaced with binary operations to achieve the same result which helps to cut the computational demand remarkably.

Batch Normalization helps the neural networks to converge faster and helps them to generalize better and hence is used widely to train the networks. But batch norm contains a lot of multiplication operations which can significantly increase the computation demand of the neural networks hence the authors have modified the batch norm to achieve the same results using bitwise operations instead of matrix multiplication. After conducting certain experiments they found that there is no loss of accuracy in shift-based batch normalization as compared to vanilla batch normalization.

The authors have also replaced the multiplication-based Adam optimizer with a shift-based AdaMax optimizer. Doing that too didn’t cause any loss of accuracy.

Benchmark And Results

The authors used three commonly used datasets for their experimentations MNIST,cifar-10, and SVHN dataset. Their BNN implementation was 7x faster on the GPU without any loss of accuracy. For information, one can refer to the original paper in the references and link section at the end of the blog.

Implementation of BNNs in Code

Having understood a good amount of theory let's now implement a BNN for performing classification on our favourite dog vs cat dataset :).

We would use a python library called larq to implement the Binary neural network and larq compute engine for running the inference. Larq implements neural network layers that run the computation in binary weights. Larq is built on top of tensorflow/Keras so its usage is the same as Keras. Just in place of Keras layers, we need to use larq layers. For layers that are not implemented in Larq we can use Keras layers. The larq compute engine is built on top of tensorflow lite with kernels optimized for binary inference and using it we can deploy binary networks on edge devices like a mobile phone or a raspberry pi.

So let's begin by installing both modules.

          !pip install larq
!pip install larq-compute-engine        

Cloning the Dog vs Cat Dataset

          !git clone https://github.com/laxmimerit/dog-cat-full-dataset.git        

Here we clone the dog vs cat dataset from the laxmimerit’s repo.

Importing Necessary Libraries

          import numpy as np
import cv2
import matplotlib.pyplot as plt
from tensorflow.keras.preprocessing.image import ImageDataGenerator
import larq_compute_engine as lce
import tensorflow as tf
import larq as lq
from PIL import Image
import os

%matplotlib inline   

Here we are just importing all the libraries required for building the neural network.

Defining the Preprocessing Function

          def preprocess_input(img):
  img=cv2.resize(img,(256,256))
  img=img/127.5-1
  return img

Here we have defined a preprocessing function that performs resizing and normalization. Since binary neural networks perform calculations in binary values i.e. -1 and 1 we normalize the input to be in this range.

Defining Data Generators

          #Defining data generators.
train_datagen = ImageDataGenerator(preprocessing_function=preprocess_input,
                    rotation_range=20,
                    width_shift_range=0.2,
                    height_shift_range=0.2,
                    zoom_range=0.2)
val_datagen = ImageDataGenerator(preprocessing_function=preprocess_input)

train_generator=train_datagen.flow_from_directory('/content/dog-cat-full-dataset/data/train')
validation_generator=val_datagen.flow_from_directory('/content/dog-cat-full-dataset/data/test')  

Here we define the train data generator and validation data generator. We use the same preprocessing function in both data generators. The train data generator also contains some random augmentations like rotation, width shift, height shift, and zoom.

Defining the Classification Model

          #Define the BNN classification model.


# All quantized layers except the first will use the same options
kwargs = dict(input_quantizer="ste_sign",
              kernel_quantizer="ste_sign",
              kernel_constraint="weight_clip",
              use_bias=False)


model = tf.keras.models.Sequential([
    # In the first layer we only quantize the weights and not the input
    lq.layers.QuantConv2D(64, 3,
                          kernel_quantizer="ste_sign",
                          kernel_constraint="weight_clip",
                          use_bias=False,
                          input_shape=(256, 256, 3)),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),
    tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),

    lq.layers.QuantConv2D(64, 3, padding="same", **kwargs),
    tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),

    lq.layers.QuantConv2D(128, 3, padding="same", **kwargs),
    tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),

    lq.layers.QuantConv2D(128, 3, padding="same", **kwargs),
    tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),

    lq.layers.QuantConv2D(128, 3, padding="same", **kwargs),
    tf.keras.layers.MaxPool2D(pool_size=(2, 2), strides=(2, 2)),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),

    tf.keras.layers.Flatten(),

    lq.layers.QuantDense(64, **kwargs),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),
    lq.layers.QuantDense(2, **kwargs),
    tf.keras.layers.BatchNormalization(momentum=0.999, scale=False),
    tf.keras.layers.Activation("softmax")])      

As mentioned above the neural network in larq can be defined in the same way as we define any Keras model. Notice that some of the layers are being used from the Keras layers themselves as the same are not defined in larq. Since larq is built on top of tensorflow itself the same layers work without creating any issues. It can also be noticed that we have defined a kwargs dictionary having arguments defining the binarization function. The first layer i.e the input layer binarizes only the input and not the weights hence we don’t pass the kwargs dictionary to it since it contains the input_quantizer argument as well.

Compile and Train The Model

          #Compile the model.
model.compile(
    tf.keras.optimizers.Adam(learning_rate=0.01),
    loss="categorical_crossentropy",
    metrics=["accuracy"],
)

#Train the model.
history=model.fit_generator(train_generator,epochs=10,validation_data=validation_generator)

#Save the model.
model.save("bnn_dogcat_clf.h5")

        

Here we compile the model with Adam optimizer having a learning rate of 0.01 and categorical_crossentropy as the loss function. Finally, we train the model and save it as an h5 file.

Below were the training output of the model.

We can see that we obtained a training accuracy of 78.72 on training data while it's 66.93 on validation data. This can be improved by performing better hyperparameter tuning on the model however since the goal of this blog is to introduce the binary neural networks and how they can be implemented and not create a perfect classification model I leave this to the readers to do the same due to my tight schedule at the moment.

Exporting the Larq Model for Inference

During the training time, Larq still uses the float values as we can see in the previous section we are using BatchNormalization from Keras, and instead of AdaMax optimizer which is bit shift based we are using the Adam from Keras. Hence the weights have to be converted to binary values for getting the performance and storage gain hence we use the larq compute engine’s convert_keras_model function for the same.

          # Convert our Keras model to a TFLite flatbuffer file
with open("bnn_dogcat_clf.tflite", "wb") as flatbuffer_file:
    flatbuffer_bytes = lce.convert_keras_model(model)
    flatbuffer_file.write(flatbuffer_bytes)    

Here we take the Keras model that we defined above and convert it into the flat buffer format in which the tflite stores the model. The same file can be loaded in larq compute engine to run inference.

Since the Keras model stored the weights in float32 format it consumed 9.39 MB of memory while the exported model consumed 1.6 MB as its weights are in binary.

However, the speedup gained from binarization can only be noticed on the platform supported by Larq compute engine.

Deployment on Android

The larq compute engine library is also available on the android platform. The prebuilt libraries for arm64-v8a, x86_64, and x86 Android ABIs. However handcrafted speedup operations are only available for arm64-v8a. Fortunately, my smartphone has arm64-v8a architecture hence I converted the quicknet model from the larq zoo and modified the standard image classification example from TensorFlow tutorials to run inference using Larq compute library. With 1 thread of my CPU quantized mobilenet model was taking 20-25ms while quicknet was taking 48–50ms. With 4 threads mobilenet was taking 18–24 ms while quicknet was taking 20–25 ms. The quicknet model has 3.1x more parameters than the mobilenet model.

If I get good support on this blog then in the future will come up with a blog having step by step tutorials for android deployment.

References and Further Readings

Binarized Neural Networks: Training Deep Neural Networks with Weights and Activations Constrained to +1 or -1

We introduce a method to train Binarized Neural Networks (BNNs) - neural networks with binary weights and activations at run-time. At training-time the binary weights and activations are used for computing the parameters gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs we conduct two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.

arxiv.org

Binarized Neural Network (BNN) and Its Implementation in Machine Learning

Explore Binarized Neural Networks, understand shift-based normalization and optimization, and explore implementation.

neptune.ai

Getting Started -

Larq is an open-source deep learning library based on TensorFlow and Keras for training neural networks with extremely low-precision weights and activations, such as Binarized Neural Networks.

docs.larq.dev

Congrats you have just learned about the binary neural network that has the potential to revolutionize the field of machine learning if enough efforts are put into it. If you liked this blog please consider sharing with your network as doing that would motivate me more to invest my time in writing blogs like this.