Using Python Code in Android Studio With Chaquopy

Whenever we think of android app development android studio is the first tool that pops up in our minds. It is an environment that includes various tools for developing android applications. Java or Kotlin is the default language that android studio supports for development. However, most of the machine learning/deep learning projects are developed in Python. Usually, most of the code has to be rewritten in Java or Kotlin for deployment as a mobile app. Hence most of us who have worked on developing machine learning-based mobile apps might have wondered at some point if there was a way to directly use the python code and create an android app.

There are frameworks like Kivy and Beeware which explicitly use python for android app development but these frameworks are recent and not as maturated as an android studio in terms of functionality. Hence in this blog, we would learn about a framework called Chaquopy that allows us to use python code in android studio.

.

Prerequisites

In order to understand this blog properly the reader should have knowledge of below domains :

* Android Studio

* Basic Java

* Object detection with python

Getting Started

We need to start by installing the Android Studio, Java Development Kit 8.0+. Also since we are using python in the code, it needs to be installed on the system. Assuming the user has familiarity with android and python I am skipping the installation part.

Setting up Chaquopy in the Android Studio

Currently, chaquopy is available as a grade plugin in android studio and hence can be set up by specifying certain configurations in the gradle files.

Create a new project

First, we start off by creating a new project in android studio. While creating the project select Java as the development language. Doing that would create certain gradle files like build.gradle, setting.gradle, etc.

Add Chaquopy as Plugin in Gradle

To add chaquopy we need to add the below line in the top level or project level build.gradle file.

          plugins {
    id 'com.chaquo.python' version '14.0.2' apply false
}

In the module-level build.gradle file, add the below lines.

          plugins {
    id 'com.android.application'
    id 'com.chaquo.python'
}

ABI Selection

The python interpreter which is used to run the python code is a native component that is specific to the ABIs or architectures for which we want the app to work hence we must specify the architecture. We do that by adding the below lines in defaultConfig of module-level build.gradle.

defaultConfig {
    ndk {
       abiFilters "armeabi-v7a", "arm64-v8a", "x86", "x86_64"
    }
}

Python Version Selection

The minimum version of python required by the 14.0.2 version of chaquopy is 3.8 and also it has good library support as well. I tried with the python 3.9 version as well but unfortunately, the chaquopy version of tensorflow was not available for it. So let's set the version of python in our project to 3.8. Also with it, we need to provide the path to the python installation in the system.

          python {
         version "3.8"
         buildPython "C:\\Users\\ADMIN\\AppData\\Local\\Programs\\Python\\Python38\\python.exe"
        }

Including the Python Libraries Installation Command

We also need to include the python library installation command in the gradle. We do this by the below lines as these are the libraries needed to run our python code:

python{
            pip{
                install "opencv-contrib-python-headless"
                install "Keras==2.3.1"
                install "tensorflow"
                install "numpy"
                install "pillow"
            }
        }

Specifying the Path to Look for Python Source Code

Finally, we need to define the local path to which the chaquopy should look for the python source files. We do it by including the below line in the gradle.

sourceSets {
            main {
                python.srcDir "src/main/python"
            }
        }

Here all our python source codes are stored inside src/main/python folder.

To have a full understanding of where to include these lines in the gradle file. Please consider looking at the git project repo link attached at the end of the blog.

Designing a Basic UI

In this blog, we are designing a simple app that captures images from the camera, performs object detection on it, and displays the image. So we will need only a button to start the camera and capture and an ImageView to display the image. Assuming the reader's familiarity with the android I am not going too deep into it.

          <?xml version="1.0" encoding="utf-8"?>
<androidx.constraintlayout.widget.ConstraintLayout xmlns:android="http://schemas.android.com/apk/res/android"
    xmlns:app="http://schemas.android.com/apk/res-auto"
    xmlns:tools="http://schemas.android.com/tools"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    tools:context=".MainActivity">

    <Button
        android:id="@+id/capture_btn"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:layout_gravity="bottom"
        android:background="@color/cardview_light_background"
        android:orientation="vertical"
        android:text="capture"
        android:textColor="#D81B60"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintTop_toBottomOf="@+id/result_image"
        app:layout_constraintVertical_bias="0.5"
        tools:ignore="MissingConstraints"
        tools:layout_editor_absoluteX="0dp" />

    <ImageView
        android:id="@+id/result_image"
        android:layout_width="match_parent"
        android:layout_height="match_parent"
        android:layout_below="@id/capture_btn"
        android:layout_marginBottom="64dp"
        app:layout_constraintBottom_toBottomOf="parent"
        app:layout_constraintTop_toTopOf="parent"
        app:layout_constraintVertical_bias="0.0"
        tools:layout_editor_absoluteX="0dp" />

</androidx.constraintlayout.widget.ConstraintLayout>

We defined the above components in the activity_main.xml file.

Python Code for Object Detection

In this section we will be looking at the python code we used for performing the object detection. However, we will be looking at the code from a high-level perspective as this blog assumes that the user is already familiar with object detection code. Deeper explanations will be done for those code parts which are specific to this blog.

Adapting from experiencor’s repo I am using tiny YOLO v2 for performing the object detection. The reason for using YOLO v2 is the fact that this project have been designed around 3 years back and since at that time I had no familiarity with PyTorch and this was the Keras implementation which I found easily. Also, yolov3 was the latest. This code can be modified to use any object detection framework.

          from keras.models import Sequential, Model
from keras.layers import Reshape, Activation, Conv2D, Input, MaxPooling2D, BatchNormalization, Flatten, Dense
from keras.layers.advanced_activations import LeakyReLU
import numpy as np
import scipy.io
import random
import os
import xml.etree.ElementTree as ET
import tensorflow as tf
import copy
import cv2
import sys
import time
import base64
import io
import os
from PIL import Image
from os.path import dirname,join


GRID_H, GRID_W = 13 , 13
BOX = 5
ORIG_CLASS = 20

WT_PATH = 'tiny-yolo-voc.weights'

THRESHOLD = 0.2
ANCHORS = '1.08,1.19,  3.42,4.41,  6.63,11.38,  9.42,5.11,  16.62,10.52'
ANCHORS = [float(ANCHORS.strip()) for ANCHORS in ANCHORS.split(',')]



# UTILS FUNCTIONS

class BoundBox:
    def __init__(self, class_num):
        self.x, self.y, self.w, self.h, self.c = 0., 0., 0., 0., 0.
        self.probs = np.zeros((class_num,))

    def iou(self, box):
        intersection = self.intersect(box)
        union = self.w*self.h + box.w*box.h - intersection
        return intersection/union

    def intersect(self, box):
        width  = self.__overlap([self.x-self.w/2, self.x+self.w/2], [box.x-box.w/2, box.x+box.w/2])
        height = self.__overlap([self.y-self.h/2, self.y+self.h/2], [box.y-box.h/2, box.y+box.h/2])
        return width * height

    def __overlap(self, interval_a, interval_b):
        x1, x2 = interval_a
        x3, x4 = interval_b
        if x3 < x1:
            if x4 < x1:
                return 0
            else:
                return min(x2,x4) - x1
        else:
            if x2 < x3:
                return 0
            else:
                return min(x2,x4) - x3


def interpret_netout(image, netout):
    boxes = []

    # interpret the output by the network
    for row in range(GRID_H):
        for col in range(GRID_W):
            for b in range(BOX):
                box = BoundBox(CLASS)

                # first 5 weights for x, y, w, h and confidence
                box.x, box.y, box.w, box.h, box.c = netout[row,col,b,:5]

                box.x = (col + sigmoid(box.x)) / GRID_W
                box.y = (row + sigmoid(box.y)) / GRID_H
                box.w = ANCHORS[2 * b + 0] * np.exp(box.w) / GRID_W
                box.h = ANCHORS[2 * b + 1] * np.exp(box.h) / GRID_H
                box.c = sigmoid(box.c)

                # rest of weights for class likelihoods
                classes = netout[row,col,b,5:]
                box.probs = softmax(classes) * box.c
                box.probs *= box.probs > THRESHOLD

                boxes.append(box)

    # suppress non-maximal boxes
    for c in range(CLASS):
        sorted_indices = list(reversed(np.argsort([box.probs[c] for box in boxes])))

        for i in range(len(sorted_indices)):
            index_i = sorted_indices[i]

            if boxes[index_i].probs[c] == 0:
                continue
            else:
                for j in range(i+1, len(sorted_indices)):
                    index_j = sorted_indices[j]

                    if boxes[index_i].iou(boxes[index_j]) >= 0.4:
                        boxes[index_j].probs[c] = 0

    

    # draw the boxes using a threshold
    for box in boxes:
        max_indx = np.argmax(box.probs)
        max_prob = box.probs[max_indx]
        
        # if(max_prob>0.01):
        # 	print("Detected box with probability : {}".format(max_prob))


        if max_prob > THRESHOLD:
            xmin  = int((box.x - box.w/2) * image.shape[1])
            xmax  = int((box.x + box.w/2) * image.shape[1])
            ymin  = int((box.y - box.h/2) * image.shape[0])
            ymax  = int((box.y + box.h/2) * image.shape[0])

            cv2.rectangle(image, (xmin,ymin), (xmax,ymax), (200,0,0), 2)
            cv2.putText(image, labels[max_indx], (xmin, ymin - 12), 0, 1e-3 * image.shape[0], (0,255,0), 1)

    return image



def sigmoid(x):
    return 1. / (1.  + np.exp(-x))

def softmax(x):
    return np.exp(x) / np.sum(np.exp(x), axis=0)




class WeightReader:
    def __init__(self, weight_file):
        self.offset = 4
        self.path_w=join(dirname(__file__),'tiny-yolo-voc.weights')
        self.all_weights = np.fromfile(self.path_w, dtype='float32')
        
    def read_bytes(self, size):
        self.offset = self.offset + size
        return self.all_weights[self.offset-size:self.offset]
    
    def reset(self):
        self.offset = 4

weight_reader = WeightReader(WT_PATH)


# Load network

model = Sequential()

# Layer 1
model.add(Conv2D(16, (3,3), strides=(1,1), padding='same', use_bias=False, input_shape=(416,416,3)))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling2D(pool_size=(2, 2)))

# Layer 2 - 5
for i in range(0,4):
    model.add(Conv2D(32*(2**i), (3,3), strides=(1,1), padding='same', use_bias=False))
    model.add(BatchNormalization())
    model.add(LeakyReLU(alpha=0.1))
    model.add(MaxPooling2D(pool_size=(2, 2)))

# Layer 6
model.add(Conv2D(512, (3,3), strides=(1,1), padding='same', use_bias=False))
model.add(BatchNormalization())
model.add(LeakyReLU(alpha=0.1))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(1,1), padding='same'))

# Layer 7 - 8
for _ in range(0,2):
    model.add(Conv2D(1024, (3,3), strides=(1,1), padding='same', use_bias=False))
    model.add(BatchNormalization())
    model.add(LeakyReLU(alpha=0.1))

# Layer 9
model.add(Conv2D(BOX * (4 + 1 + ORIG_CLASS), (1, 1), strides=(1, 1), kernel_initializer='he_normal'))
model.add(Activation('linear'))
model.add(Reshape((GRID_H, GRID_W, BOX, 4 + 1 + ORIG_CLASS)))



# Load pre-trained weights

# model.load_weights(orig_weight_path)


weight_reader.reset()
nb_conv = 9

for i in range(1, nb_conv+1):
    conv_layer = model.get_layer('conv2d_' + str(i))
    
    if i < nb_conv:
        norm_layer = model.get_layer('batch_normalization_' + str(i))
        
        size = np.prod(norm_layer.get_weights()[0].shape)

        beta  = weight_reader.read_bytes(size)
        gamma = weight_reader.read_bytes(size)
        mean  = weight_reader.read_bytes(size)
        var   = weight_reader.read_bytes(size)

        weights = norm_layer.set_weights([gamma, beta, mean, var])       
        
    if len(conv_layer.get_weights()) > 1:
        bias   = weight_reader.read_bytes(np.prod(conv_layer.get_weights()[1].shape))
        kernel = weight_reader.read_bytes(np.prod(conv_layer.get_weights()[0].shape))
        kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))
        kernel = kernel.transpose([2,3,1,0])
        conv_layer.set_weights([kernel, bias])
    else:
        kernel = weight_reader.read_bytes(np.prod(conv_layer.get_weights()[0].shape))
        kernel = kernel.reshape(list(reversed(conv_layer.get_weights()[0].shape)))
        kernel = kernel.transpose([2,3,1,0])
        conv_layer.set_weights([kernel])



# get correct labels
labels = ['aeroplane','bicycle','bird','boat','bottle','bus','car','cat','chair','cow','diningtable','dog','horse','motorbike','person','pottedplant','sheep','sofa','train','tvmonitor']

print (labels);

# CLASS = 184
CLASS = 20 # 23




def detectObjects(image):
    input_image = cv2.resize(image, (416, 416))
    input_image = input_image / 255.
    input_image = input_image[:,:,::-1]
    input_image = np.expand_dims(input_image, 0)

    # netout = new_model.predict(input_image)
    netout = model.predict(input_image)
    image = interpret_netout(image, netout[0])

    return image

def detect_and_draw(strn):
    decoded_data=base64.b64decode(strn)
    np_data=np.fromstring(decoded_data,np.uint8)
    img=cv2.imdecode(np_data,cv2.IMREAD_UNCHANGED)
   
    img=detectObjects(img)
    img=cv2.cvtColor(img,cv2.COLOR_BGR2RGB)

    pil_img=Image.fromarray(img)
    buff=io.BytesIO()
    pil_img.save(buff,format="PNG")
    img_str=base64.b64encode(buff.getvalue())
    return ""+str(img_str,'utf-8')    

In the above section, I have modified the experiencor’s code for just running the prediction. Since originally the YOLO was developed with a darknet framework, the official weights up to YOLO v4 are available in darknet. But since we want our inference in Keras we are here creating an equivalent model in Keras and copying the weights from the darknet model’s layer to the Keras model’s layer.

The function which is worth paying attention to is detect_and_draw. This function takes a base64 string and converts it into a numpy array and then performs object detection on it. After getting an image on which bounding boxes have been drawn, it again converts it back to the base64 string and returns it. The reason for doing this would be clear in the next section when we will be calling this function from the java code event handler.

Defining the Main Activity Java Code

Here we define the MainActivity class which has the Java event handler code for the UI we designed. Here we would be defining the camera object and event handler which does all the operations like capturing images to show on the ImageView after processing.

          package com.example.tiny_yolo;

//library imports.
import androidx.appcompat.app.AppCompatActivity;

import android.os.Bundle;

import androidx.annotation.Nullable;

import android.content.Intent;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.os.Bundle;
import android.provider.MediaStore;
import android.util.Base64;
import android.view.View;
import android.widget.Button;
import android.widget.ImageView;
import android.widget.Toast;

import com.chaquo.python.PyObject;
import com.chaquo.python.Python;
import com.chaquo.python.android.AndroidPlatform;

import java.io.ByteArrayOutputStream;

// Main class definition.
public class MainActivity extends AppCompatActivity {

    //Defining Image view and button objects.
    ImageView resultImage;
    Button captureBtn;
    public String imgString;

    private static final int RC_PIC_CODE=101;


    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_main);
        
        //Variable to reference the Image view and capture button in the UI.
        resultImage=(ImageView)findViewById(R.id.result_image);
        captureBtn=(Button)findViewById(R.id.capture_btn);


        //Seting event listener on the capture button.
        captureBtn.setOnClickListener(new View.OnClickListener() {
            @Override
            public void onClick(View view){
                capture();
            }
        });

    }

    //Function to capture images using the camera.
    private void capture(){
        Intent takePictureIntent=new Intent(MediaStore.ACTION_IMAGE_CAPTURE);
        startActivityForResult(takePictureIntent,RC_PIC_CODE);

    }
    
    //Function to call the python function and run detection.

    @Override
    protected void onActivityResult(int requestCode, int resultCode, @Nullable Intent data) {
        super.onActivityResult(requestCode, resultCode, data);
        if (requestCode==RC_PIC_CODE){
            if (resultCode==RESULT_OK){
                //Getting bitmap data.
                Bitmap bp=(Bitmap) data.getExtras().get("data");
                resultImage.setScaleType(ImageView.ScaleType.FIT_CENTER);
                //Code for conversion to grayscale.
                
                //Starting python interpreter.
                if (!Python.isStarted()) {
                    Python.start(new AndroidPlatform(this));
                }
                
                //Getting python interpreter instance.
                final Python py=Python.getInstance();
                //Converting bitmap data to base 64 string.
                
                imgString=getStringImage(bp);
                //Creating the object of python source file yolo_module.
                PyObject pyo=py.getModule("yolo_module");
                //Calling function detect_and_draw from module.
                PyObject obj=pyo.callAttr("detect_and_draw",imgString);
                //Converting to string.
                String str=obj.toString();
                //Decoding back the base64 string to byte data.
                byte data2[]= Base64.decode(str,Base64.DEFAULT);
                //Creating bitmap image from bitmap data.
                Bitmap bmp= BitmapFactory.decodeByteArray(data2,0,data2.length);
                //Setting the bitmap image in the image view.
                resultImage.setImageBitmap(bmp);
            }
        }
        else if (resultCode==RESULT_CANCELED){
            Toast.makeText(this,"cancelled",Toast.LENGTH_SHORT).show();
        }
    }
    
    
    //Funcion to convert the bitmap to base64 string.
    private String getStringImage(@org.jetbrains.annotations.NotNull Bitmap bitmap){
        ByteArrayOutputStream baos=new ByteArrayOutputStream();
        bitmap.compress(Bitmap.CompressFormat.PNG,100,baos);
        byte [] imageBytes=baos.toByteArray();
        String encodedImage=android.util.Base64.encodeToString(imageBytes, Base64.DEFAULT);
        return encodedImage;

    }
}

Here we first import all the submodules required from the packages android,androidx, and chaquopy then we inherit the AppCompatAcitivity class. We then declare various methods in the class. The onCreate method creates an ImageView and result button. Inside the onCreate method, we set the event listener to the capture button which on clicking calls the capture method which opens the camera for capture. The capture method uses the android inbuilt Intent class for using the camera.

Then inside the onActivityResult method, we first get the image data returned from the camera and convert it to bitmap data and then to the grayscale format. After that, we start an instance of the python interpreter. In this python interpreter instance, we load our yolo_module python file. Before starting the python instance we convert the image into base64 string as we cannot directly pass the bitmap data to the python instance since the way in which python represents the image data is different from the way in which Java does. We pass this base64 string to the detect_and_draw function in the yolo_module which detects objects and draws bounding boxes on the image and returns the image converted back as base64 string. We decode this encoded base64 string to byte data and convert this byte data to bitmap image. We finally set this image in the Imageview.

Below is the demo of the app on my phone:

We can see that it had successfully detected the bottle on the app.

Further Improvements

1.  Instead of tiny YOLO v2, current state of art models like tiny YOLO v7 or v8 could be used. At the time I developed this project YOLO v3 was the latest, and keras implementation of tiny YOLO v2 was readily available so I choose that for developing this app.

2. Instead of using Keras for running prediction, TensorFlow lite can be used as its both light and optimized for phone kernels.

3. The model which I am using here is not optimized at all. Hence optimization like pruning and quantization can be applied before deployment.

I have used the open-source version of Chaquopy for developing this app as this app is just for educational purposes and not for any commercial use. If you want to develop an app for commercial purposes you might need to buy the licensed version of Chaquopy.

References and Further Readings

Congrats you have just learned how you can use python code in android studio with the help of Chaquopy with which you can bring all the leverages of python to your android app. If you have any questions regarding the blog. Please feel free to leave in the comments section.