Ideal PodiumBringing Python to Browser for Doing Image Processing
Vikas Kumar Ojha
MachineLearning@Avalara Ex-Samsung
Today apps or web apps are more popular than desktop apps. You can do almost everything in your browser and don’t need to install anything. Any web app written in Javascript doesn’t require any installation of additional software all thanks to the powerful V8 engine of apps like chrome. But until a few months back only JavaScript was that language that can be used to write web apps. In case of use cases like machine learning or image processing separate server is used for doing the backend processing. In certain cases, developers do use frameworks like TensorFlow JS to develop such apps but still, the TensorFlow python version is more mature and popular in the community because of the popularity of the python in the field. In most cases, the API is usually deployed in the cloud which costs more and is less secure than a completely local server. Hence it would be good if we would be having a way by which we could run python in the browser.
This article explains how the above problem can be solved and we can run python in the browser with help of Pyscript. At the end of this article, you would learn how you can do image processing in the browser using pyscript.
PyScript
PySript is a technology developed by the anaconda team which aims at running python in the web browser. It utilizes web assembly for running python. The code we write in python is compiled into web assembly and is executed by our browser. The PyScript is based on pyiodide which provides a web assembly port of Python. Pyscript comes as a single JS file that can be added to a webpage which inturns load the whole python runtime. Since pyscript is based on pyiodide most of the libraries available in pyiodide can be used. Right now pyscript is still in the development stage and hence some functionality might not work in future releases or might work in a different way.
Basics of PyScript
Let's learn some basics of PyScript before developing the actual application. The PyScript can be thought of as an HTML page with python in it which is executed by the browser runtime environment. It is implemented with the help of HTML tags.
The <py-script> tag
The pyscript tag just like any HTML tag starts with <py-script> and ends with </py-script>. In between this, we write all the python code that we want to execute while the page opens.
The <py-env> tag
The pyscript tag just like any HTML tag starts with <py-env> and ends with </py-env>. This tag is used for importing external libraries which are used in our source code. It tells the pyiodide environment to load the listed libraries from the source. Below is an example where we want to import external libraries matplotlib,numpy,imageio, and scipy.
<py-env>
- matplotlib
- imageio
- numpy
- scipy
</py-env>
DOM Manipulation in PyScript
PyScript can manipulate different objects in the HTML document/webpage. The python interpreter can access all the objects on the page like the inputs button, text areas, etc. The object/elements in the page can be accessed with the help of the Element and document class from the js module of PyScript. Below is an example where we are using the DOM manipulation feature of PyScript.
<py-script>
from js import document
def say_hello(*args,**kwargs):
name_txt=Element('Name').value
message_field = document.getElementById("message_txt")
message_field.value="Hello "+name_txt
</py-script>
In the above function, we are getting the value contained in the text field named ‘Name’ and then appending hello with it. Finally, we are accessing the text field with id message_txt with help of getElementById and setting its value as the new text.
Image Processing in PyScript
Let's first see the code and then understand it in bits.
<html>
<head>
<title>
Image Processing with PyScript
</title>
<link rel="stylesheet" href="https://pyscript.net/alpha/pyscript.css" />
<script defer src="https://pyscript.net/alpha/pyscript.js"></script>
<py-env>
- matplotlib
- imageio
- numpy
- scipy
</py-env>
</head>
<style>
input[type=text], select {
width: 100%;
padding: 12px 20px;
margin: 8px 0;
display: inline-block;
border: 1px solid #ccc;
border-radius: 4px;
box-sizing: border-box;
}
input[type=submit] {
width: 100%;
background-color: #4c4faf;
color: white;
padding: 14px 20px;
margin: 8px 0;
border: none;
border-radius: 4px;
cursor: pointer;
}
input[type=submit]:hover {
background-color: #45a049;
}
p {
border-radius: 5px;
background-color: #f2f2f2;
padding: 20px;
}
</style>
<body>
<form onsubmit="return false">
<label for="name">Image URL:</label><br>
<input type="text" id="imageurl" name="imageurl" value="https://cataas.com/cat"><br>
<label for="label action option">Select a image processing option:</label><br>
<select name="img_proc_op" id="img_proc_op">
<option value="grayscale">grayscale</option>
<option value="edge_detection">edge_detection</option>
<option value="image_binarization">image_binarization</option>
</select>
<input pys-onClick="sub" type="submit" id="btn-form" value="submit">
</form>
<div id="textfield">Processed Image</div>
<div id="pyplotdiv"><img id="pyplotfigure"/></div>
<py-script>
from pyodide.http import pyfetch
from js import document
import asyncio
from io import BytesIO
import matplotlib.pyplot as plt
import imageio.v3 as iio
import io, base64
import numpy as np
import scipy.signal as sg
def rgb2gray(rgb):
return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140])
def convert_binary(image_matrix, thresh_val):
#Source https://medium.com/analytics-vidhya/binarization-of-image-using-numpy-65df2b82e189
white = 255
black = 0
initial_conv = np.where((image_matrix <= thresh_val), image_matrix, white)
final_conv = np.where((initial_conv > thresh_val), initial_conv, black)
return final_conv
def edge_detection(img):
img = img[:, :, 0]
img = img.astype(np.int16)
edge = np.array([[-1, -1, -1], [-1, 8, -1], [-1, -1, -1]])
results = sg.convolve(img, edge, mode='same')
results[results > 255] = 255
results[results < 0] = 0
results = results.astype(np.uint8)
return results
async def sub(*args,**kwargs):
url_val=Element('imageurl').value
image_process_action=Element('img_proc_op').value
response = await pyfetch(url=url_val, method="GET")
img = iio.imread(BytesIO(await response.bytes()), index=None)
img=np.array(img)
if image_process_action=='grayscale':
img=rgb2gray(img)
plt.axis('off')
plt.imshow(img,cmap='gray')
elif image_process_action=='edge_detection':
img=edge_detection(img)
plt.axis('off')
plt.imshow(img,cmap='gray')
elif image_process_action=='image_binarization':
img=rgb2gray(img)
img=convert_binary(img,127)
plt.axis('off')
plt.imshow(img,cmap='gray')
else:
img=img
plt.axis('off')
plt.imshow(img)
buf = io.BytesIO()
plt.savefig(buf, format='png')
buf.seek(0)
img_str = 'data:image/png;base64,' + base64.b64encode(buf.read()).decode('UTF-8')
pfig = document.getElementById("pyplotfigure")
pfig.setAttribute("src",img_str)
</py-script>
</body>
</html>
Credits: The HTML-CSS part used in this code has been inspired by the 1littlecoder pyscript tutorial on youtube.
Code Explanation
* In lines 11–12 we are importing pyscript CSS and pyscript runtime environment using link and script tags respectively.
* In lines 13–18 we are importing external libraries in the PyScript environment with the help of the <py-env> tag.
* In lines 22–53, we have written CSS code for styling our input text field and drop list object in the HTML part.
* In lines 57–74 we are using basic HTML to create an HTML tag. Note that in the form tag we have set onsubmit attribute to return false meaning we don’t want to switch pages with the click of submit button. In the HTML form, there are three objects one of which is a text field that is used to input the URL of the image to be processed another object is a drop list which is used to select the type of image processing operation we want to perform. The last object is a submit button which calls our python function which does all the processing. The pys-onClick attribute which is a pyscript-specific attribute is used to call our python function sub-defined inside pyscript. Outside of the form tag, we can notice two more div tags one is just for label and another div element contains an img tag which we will eventually manipulate for displaying our image.
* From lines 76–84 we are importing necessary libraries for performing image processing. It can be observed that we are also importing the async module and pyiodide module but that's not mentioned in the <py-env> tag. The reason is that these libraries are internal python/pyscript libraries.
* In lines 86–110 we have written some image processing functions using numpy and scipy as opencv was not available at the time in pyiodide.
* In line 114 we are defining an async function called sub.
* In lines 115 and 116 we are accessing the values of the image URL text field and image processing selection drop list.
* In lines 117–119 we are obtaining the image from the URL using pyiodide’s pyfetch function then we are converting the bytes obtained in the response as a buffer with help of BytesIO and reading it as an image with help of imread function. Finally, we are converting images into numpy format.
* From lines 120–137 we are calling the function according to our selection in the droplist.
* In line 141 we are initializing a BytesIO buffer. In line 142 we are saving our plot to this buffer. In line 143 we are pointing at the start of the buffer.
* In lines 144–146 we are encoding the image as base64 string. We then access the img tag and set its src attribute with this string. Setting this places the image data in the img tag which is rendered by the browser just like any img tag would be.
The above code can be saved as HTML document and opened in chrome.
Output of the app
Above is the working of the pyscript app we built. The URL in the imageurl text field is a URL that randomly returns the image of a cat. One more thing to notice here is that the URL in the browser is pointing directly to the HTML file in my disk as I have saved this file as html doc and opened it in chrome. In some tutorials they create a frontend sever first and run the app. In that case, we might be able to access the filesystem of the system directly. But creating a server would involve some installation which would hurt the portability of the PyScript hence I decided to do it in a different way by reading images from the URL.
References and Further Readings
If you reached here then congrats you have just learned an amazing concept. Thanks for spending your time to read this blog. If you have question please feel free to add in the comments sections.
Comments
Stripathi1
Nice tutorial