Giving Language Models the Ability to See Without Retraining or Fine-Tuning

Advancements in artificial intelligence have led to the creation of large language models that can generate human-like text. However, these models lack the ability to understand images or visual information. Imagine teaching these language models to ‘see’ and make sense of the visual world, without explicitly training them on images. In this blog, we explore a new approach to achieving this. Using a combination of techniques that allow the models to learn implicitly and understand text and images, we aim to bridge the gap between language and vision. This could open up exciting possibilities, such as having these models describe images accurately, answer questions about pictures, and much more.

Computer Vision Through the LENS of Natural Language

Commonly if we want to give the language models the ability to see we need to train them on multi-modal data i.e. image and text pairs but often finding such paired data is hard and large language models are simply too expensive to be retrained by a small organization or individual. Recently in June 2023, ContextualAI and Stanford University recently proposed a paper titled ‘Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language’’ that solves the above problem. In the paper instead of training the model end to end on image and text pair the authors followed a modularized multi-stage approach to process the input which we will discuss in detail in the next section.

High-level Methodology

                                        Source: LENS paper.

The proposed method consists of four components 3 vision modules(Tag module, Attribute Module, Intensive Captioning Module) and 1 reasoning Module(Frozen LLM). Let's understand that in detail below.

The image is passed individually to each of these vision modules. The tag module extracts tags for the image as shown above in the figure. There is a picture of a dog where he is surfing. So the tag module predicts tags for the image like surfing, surfer, pug, water dog, pug, and dog. The attribute module predicts attributes of the objects in the image like pug dog has double coated fur. The captioning module predicts the captions for the image like a small dog riding a surfboard in the water. These predicted texts by vision modules are passed along with the question to the LLM which acts as a reasoning module and provides answers according to the question asked.

Understanding the Vision and Reasoning Modules

Well, it does make sense if we combine the visual information with reasoning for doing visual and language tasks because as a human if we want to recognize something we not only depend on what we see but also we use our logic to recognize and differentiate. But what are these vision models and how exactly they are predicting this information? Let’s answer this question in detail below.

Tag Module

The tag module is a pretrained CLIP model. The OpenAI CLIP model instead of directly classifying a given image into fixed sets of classes, tries to associate the image with a set of tags that makes it generalized to a large set of objects. The CLIP model has been trained on pairs of images and their tags. For example, for an image, the tag could be a red t-shirt. For predicting this tag the model needs to learn different attributes in the image like red and t-shirt making it learn multiple features in the image which makes the model generalizable to a wide variety of objects. To understand it better, it is highly recommended to read the CLIP paper.

In the LENS paper, the authors collected various tags from a large number of image datasets and created a tag vocabulary. The CLIP model is then employed to associate the tags which can be related to the input image. Instead of predicting one tag per image as we do in greedy decoding or beam search, in LENS, we use top N sampling to generate some N tags.

Attribute Model

For attribute prediction, the authors employed the GPT-3 to create attribute vocabulary. They used GPT-3 to generate visual descriptions of each class in a large number of datasets. For attribute prediction, the CLIP model is asked to associate the image with the attributes in the attribute vocabulary. Similar to the tag module N attributes are predicted.

Captioning Module

For generating captions BLIP model is implemented. Similar to the tag and attribute module top N captions are predicted.

Reasoning Module

It is a pretrained language model. The best part of the LENS is that any language model can be used for the reasoning module.

                                          Source: LENS paper.

Above is the comparison of zero-shot object recognition performance between the LENS pipeline and the clip model. It can be clearly inferred from the above table that the LENS pipeline is superior or equal to CLIP on most datasets which confirms the fact that combining reason with visual information helps in improving visual tasks also.

Some Code Hands-On Visual Question Answering

Installing the required dependencies

!pip install llm-lens
!pip install opencv-python
!pip install matplotlib
!pip install torch
!pip install pillow
!pip install numpy
!pip install transformers

Here we are installing the llm-lens package developed by the paper's authors which contains all the preprocessing functions, model structure, and metadata to run the LENS model. We also install other libraries like opencv-python, torch, pillow, transformers,matplotlib, and numpy.

          import cv2
import requests
import torch
import numpy as np
from PIL import Image
import matplotlib.pyplot as plt
from lens import Lens, LensProcessor
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM   

Here we are loading all the necessary libraries for loading the models and running the prediction.

Loading the models and preprocessors

tokenizer = AutoTokenizer.from_pretrained("google/flan-t5-base",truncation_side = 'left',padding = True)
LLM_model = AutoModelForSeq2SeqLM.from_pretrained("google/flan-t5-base")

lens = Lens()
processor = LensProcessor()  

Here we are loading LENS model as well as the T5 transformer as our language model. We are also loading their tokenizer and preprocessor.

Question Answering on a image

          
img_url = 'https://images.unsplash.com/photo-1465056836041-7f43ac27dcb5?w=720'
raw_image = Image.open(requests.get(img_url, stream=True).raw).convert('RGB')
question = "What is the image about?"


#Generating the prompts.
with torch.no_grad():
    samples = processor([raw_image],[question])
    output = lens(samples)
    prompts = output["prompts"]
    
    
#Predicting the answers with languge model.  
input_ids = tokenizer(prompts, return_tensors="pt").input_ids
outputs = LLM_model.generate(input_ids)
print(tokenizer.decode(outputs[0],skip_special_tokens=True))    

Here we are loading the image and passing it to the preprocessor module of the LENS pipeline and running the LENS model to predict the prompts. Once prompts are generated they are passed to the language model to predict the answer.

Input Image

Question: What is this image about?

Generated Prompt:

‘Tags:\n-Glacial lake\n-Mountain range\n-mountains\n-Lake district\n-Steppe\nAttributes:\n-destination which has Natural features such as mountains, rivers, or lakes\n-lake which has large body of water surrounded by land\n-peace which is a peaceful landscape or scenery\n-place which has natural features like mountains, rivers, and lakes\n-mountains which has snow-capped peaks\nCaptions:\n-a herd of sheep graze near a mountain lake\n-a large snow covered mountain with a lake\n-a boat on an island next to a body of water\n-a sheep looking at a lake near very tall mountains\n-a small, distant area filled with snow capped mountains\n-a lake and mountains under a cloudy sky\n-a view of mountains near a lake in a field\n-a small animal in shallow water near the mountains\n-a landscape view of lots of mountains and a lake with a few boats in it\n-a very scenic view of a very very big lake\nQuestion:What is the image about?\nShort Answer:’

Language Model Output: a mountain lake with a lake

Small tip:

Doing the experimentation I realized if you have several questions regarding the same image you don’t need to run the LENS model every time. You might just have to modify the Question portion in the prompt and pass the prompt to the language model you would get the same result saving computation. Now let’s see more examples.

Question: Is there a lake in the image?

Language Model Output: yes

Question: Are mountains covered in snow?

Language Model Output: yes

Question: Is there a panda in the image?

Language Model Output: no

References and Further Readings

Towards Language Models That Can See: Computer Vision Through the LENS of Natural Language

We propose LENS, a modular approach for tackling computer vision problems by leveraging the power of large language models (LLMs). Our system uses a language model to reason over outputs from a set of independent and highly descriptive vision modules that provide exhaustive information about an image. We evaluate the approach on pure computer vision settings such as zero- and few-shot object recognition, as well as on vision and language problems. LENS can be applied to any off-the-shelf LLM and we find that the LLMs with LENS perform highly competitively with much bigger and much more sophisticated systems, without any multimodal training whatsoever. We open-source our code at https://github.com/ContextualAI/lens and provide an interactive demo.

arxiv.org

GitHub - ContextualAI/lens: This is the official repository for the LENS (Large Language Models Enhanced to See) system.

This is the official repository for the LENS (Large Language Models Enhanced to See) system. - ContextualAI/lens

github.com

Link to the notebook:

GitHub - VikasOjha666/LENS_paper_demo

Contribute to VikasOjha666/LENS_paper_demo development by creating an account on GitHub.

github.com

Congrats you have just learned an amazing concept in deep learning. If you have any questions please feel free to drop in the comment section. Thanks for spending your time to read this blog.