Deep Intuition Behind the GANs: From Maths to Code

GANs or Generative Adversarial Networks are regarded as one of the most interesting ideas in Deep Learning. This network was created in 2014 by sir Ian Goodfellow and his colleagues as described in Goodfellow et al.

At that time it drew a lot of attention from the Deep Learning folks and today it is one of the hot areas of research in Deep Learning. Sir Yann LeCun termed the GANs as one of the most interesting ideas in 10 years.

There were already some generative algorithms in Deep Learning like Autoencoders, GMMs, etc. But none of them were as good as GANs. Today GANs are used in a wide range of applications like generating colored images from sketches, product designing, image colorization, image super-resolution, and so many other cool stuff.

Hence I thought of writing this article to give full intuition behind the GANs and to connect the dots from maths to code. In this article, we will build a simple GAN which generates MNIST digits.

Overview

GANs or the Generative Adversarial Networks are generative models that are a type of unsupervised algorithms that learn the latent representation of the data on which they are trained as a result they can generate real-looking data on which they are trained. A GAN is composed of two networks competing against each other to fool each other. The first network is called discriminator and its job is to differentiate fake samples from original. The other network is called a generator and its job is to generate the fake sample. It is similar to the real world condition where criminals always try to create fake currency and the bank staffs identify whether the currency is original or fake. Similarly, the generator can be thought of as a criminal generating fake sample while the discriminator can be thought of as a cop or bank staff trying to identify fake samples.

In this article, we will first understand the math behind the GAN and then we will understand how it is actually implemented in code.

Intuition behind GAN

Understanding the loss function

As described in the overview section GAN is composed of two networks i.e. discriminator and generator where both are trying their best to optimize themselves and fool the other. Now why this works?Why at the end the GAN is able to generate real-looking samples?

To give an answer to the above question I will like you to put a little effort into your brain and try to recall what's the real objective of a classifier? The answer to this question is that whenever a classifier is trained it tries to learn the probability distribution of the data on which it is trained. So if the probability distribution of two data samples is similar the classification result might be the same for each of them. Hence to fool the discriminator the probability distribution of the samples generated from the generator should be similar to the probability distribution of the original data and for that to happen the images should be similar.

Hence the discriminator learns the probability distribution of data to classify samples properly while the generator is trying to learn the latent representation of the data in order to generate real-looking samples.

The optimization problem in GAN can be thought of as a minimax problem and the below equation represents it:-

Now lets understand this equation.

Here D(x) represents the probability of that the sample came from real data and D(G(x)) is the probability that the sample came from the generated data.

The equation consists of two parts the D(x) one represents the predictions on real data and D(y) term representing predictions on fake data.

The first part is the expectation of the log of the discriminator output i.e if we sample some points from the original data then whats the average predictions on them. Similarly the second part is for fake data.

The discriminator wants to maximize this equation since it wants to correctly classify the samples. One thing to notice here is that,in the second part we are trying to maximize (1-D(G(z))) because if D(G(z)) is maximum that means the discriminator is classifying the sample as fake without any difficulty but we want to fool the discriminator and hence we want to minimize it but we know that in a equation at a time we could either do maximization or minimization and hence we change D(G(x)) to (1-D(G(x))) so that we can maximize it.

The generator wants to minimize this cost function as it wants the discriminator to be uncertain whether the sample came from real data or from the generated data.

Now how the GAN is trained?

Above image contains the algorithm for training the GAN. It has been taken from GAN paper by Ian Goodfellow and his colleagues.

The GAN is trained by doing alternative minimization or adversarial training.Which has been demonstrated in the above equation.

According to the above algorithm we train the discriminator for k times while freezing the generator model. In original GAN paper they set the k=1 that is they train the discriminator for once. For training the discriminator, we sample m noise samples from a normal distribution and we pass it through generator to generate some fake images. We also sample m images from original dataset and then we train our discriminator on that.For updating the weights we find the derivative of the cost function defined above and we update by ascending its stochastic gradient. As demonstrated by first equation in the image.

Now we train the generator. For doing this we sample m noise vectors from normal distribution and find the gradient of cost function and update the generator parameters descending its stochastic gradient. Since the cost function is now differentiated w.r.t the parameters of generator so we are left with equation independent of D(x) term as shown in second equation of above image

Now the question arises here is when does the network converge and how do we know which checkpoint is best?

The Answer to the above question is that there is no straight forward convergence. When we observe that the discriminator is being unable to differentiate between original and fake sample given that the image generated is of decent quality that is the time we can say that the GAN has reached its optimal point as the training in GANs is not about minimizing or maximizing a cost function but it’s more like finding a saddle point between the two networks. The best checkpoint will be that which generates good images .To find that we will plot a few images generated per epoch and from there we will determine the best checkpoint.

Now as we have understood the theory about the GANs, lets dive into the code and implement one for generating MNIST digits.

Implementing GAN in code

Lets visualize the GAN we want to build.

The input to our GAN is a 100 dim noise vector sampled from a standard distribution. This noise is passed to our generator which is composed of a dense layer of units=128*7*7 for 7 x 7 image which is followed by a Conv2D Transpose layer of filter=128, kernel (4x4), and stride=2 which upsamples it to 14 x 14 x 128. It is again followed by a second Conv2D Transpose layer of filter=128, kernel (4x4), and stride=2 which upsamples it to 28 x 28x128 size but at that point, it has size 28x28x128 so a Conv2D layer of filter=1 and kernel (7,7) is applied which turns the image to 28x28x1.

Our discriminator consists of two convolutional layers of filters=64, kernel (3,3), and stride=2 in a sequence which are later followed by a dense layer of 1 unit to classify whether the image is real or fake.

All the layers have Leaky ReLU as activation both in discriminator and generator expect the sigmoid at the end of both discriminator and generator.

Now lets dive deep into the code.

Lets import all the required libraries.

          #Importing required library.
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import keras
from keras.layers import Dense, Dropout, Input,Conv2D,Conv2DTranspose,Flatten,Reshape
from keras.models import Model,Sequential
from keras.datasets import mnist
from keras.layers.advanced_activations import LeakyReLU
from keras.optimizers import Adam
from keras import layers
from tqdm import tqdm
%matplotlib inline  

As we can see we have also imported mnist dataset from the keras along with the other packages.So lets load the data and normalize it.

          #Loading and normalizing the data.
(X_train,y_train),(X_test,y_test)=mnist.load_data() 
X_train=X_train.astype('float32')
X_train=np.expand_dims(X_train,-1)
X_train=X_train/255 

Now let's define the generator as described above.

          def define_generator(latent_dim):

	model = Sequential()
	# foundation for 7x7 image
	n_nodes = 128 * 7 * 7
	model.add(Dense(n_nodes, input_dim=latent_dim))
	model.add(LeakyReLU(alpha=0.2))
	model.add(Reshape((7, 7, 128)))
	# upsample to 14x14
	model.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))
	model.add(LeakyReLU(alpha=0.2))
	# upsample to 28x28
	model.add(Conv2DTranspose(128, (4,4), strides=(2,2), padding='same'))
	model.add(LeakyReLU(alpha=0.2))
	model.add(Conv2D(1, (7,7), activation='sigmoid', padding='same'))
	return model   

Let’s define discriminator.

          def define_discriminator(in_shape=(28,28,1)):
	model = Sequential()
	model.add(Conv2D(64, (3,3), strides=(2, 2), padding='same', input_shape=in_shape))
	model.add(LeakyReLU(alpha=0.2))
	model.add(Dropout(0.4))
	model.add(Conv2D(64, (3,3), strides=(2, 2), padding='same'))
	model.add(LeakyReLU(alpha=0.2))
	model.add(Dropout(0.4))
	model.add(Flatten())
	model.add(Dense(1, activation='sigmoid'))
	# compile model
	opt = Adam(lr=0.0002, beta_1=0.5)
	model.compile(loss='binary_crossentropy', optimizer=opt, metrics=['accuracy'])
	return model 

Now let’s finally connect this two models to create the GAN. 

Here we set the trainable attribute to the discriminator to false in order to prevent it from training while we train our generator with help of its errors.

          
def define_gan(g_model, d_model):
    d_model.trainable=False
    #Input tensor taking 100 dim noise vec.
    inp=Input(shape=(100,))
    #Passing it through generator.
    x=g_model(inp)
    #Feeding the generator output to discriminator.
    x=d_model(x)
    model=Model(inputs=inp,outputs=x)
  
    # compile model
    opt = Adam(lr=0.0002, beta_1=0.5)
    model.compile(loss='binary_crossentropy', optimizer=opt)
    return model

The above is a function which returns the GAN model when we pass the discriminator and generator model to it. 

So now let’s create the object of discriminator model and generator model and pass it to this function.

          # size of the latent space
latent_dim = 100
# create the discriminator
d = define_discriminator()
# create the generator
g= define_generator(latent_dim)
# create the gan
gan = define_gan(g, d) 

Now we are all set to train our model.

We will train our model for 100 epochs with a batch size of 256 on our X_train array containing 60000 MNIST digit images.

          #Train the model.

#Basic parameters.
batch_size=256
half_batch=128
epochs=100
latent_dim=100
batch_per_epoch=X_train.shape[0]//batch_size




for epoch in range(epochs):
  d1_loss=[]
  d2_loss=[]
  g_loss=[]
  for _ in tqdm(range(batch_per_epoch)):
    X_real=X_train[np.random.randint(0,X_train.shape[0],half_batch)] #Selecting n=batch_size random images from training data.
    y_real=np.ones((half_batch,1)) #Setting label to 1 as they are real samples 

    X_fake=np.random.randn(latent_dim*half_batch) #Creating batch_size no of latent vector of dim 100.
    X_fake=X_fake.reshape(half_batch,latent_dim)
    #Generating fake image on which we want to train generator.
    x_fake_img=g.predict(X_fake)
    y_fake=np.zeros((half_batch,1)) #Setting the label to 0 as they are fake samples.

    #Train the generator with both real and fake samples seperately.
    dLReal,_=d.train_on_batch(X_real,y_real)
    d1_loss.append(dLReal)
    
    dLFake,_=d.train_on_batch(x_fake_img,y_fake)
    d2_loss.append(dLFake)

    #Now lets train generator.For training generator we have to train it through discriminator.
    #Sample batch_size no of latent vector.
    X_fake_gan=np.random.randn(latent_dim*batch_size) #Creating batch_size no of latent vector of dim 100.
    X_fake_gan=X_fake_gan.reshape(batch_size,latent_dim)
    """ Here we set label=1. Its more like telling the generator to update itself in 
        such a way that discriminator classifies the sample as 1 that is as real 
        sample.
    """
    y_gan=np.ones((batch_size,1))
    g_l=gan.train_on_batch(X_fake_gan,y_gan)
  
  #Plot 10 generated images per 5 epochs.
  if epoch%5==0:
    l_vec=np.random.randn(latent_dim*10) 
    l_vec=l_vec.reshape(10,latent_dim)
    gen_images=g.predict(l_vec)
    
    count=0
    fig,ax=plt.subplots(2,5)
    for i in range(2):
      for j in range(5):
        ax[i][j].axis('off')
        ax[i][j].imshow(np.squeeze(gen_images[count],axis=-1),cmap='gray')
        count+=1
    fig.savefig(f'gen_images{epoch}.jpg')
    g_loss.append(g_l)
  print(f'\nEpoch={epoch} DLReal={sum(d1_loss)/(batch_per_epoch)} DLFake={sum(d2_loss)/(batch_per_epoch)} GAN_loss={sum(g_loss)/batch_per_epoch}' )
        

The training goes like this. We first train the discriminator on real and fake examples separately. For training the discriminator on real images we randomly sample batch_size//2 no of images from our training data and we create an equivalent number of labels which are 1. For training the discriminator on fake data we first create batch_size//2 100 dimensional noise vector and we pass them through the generator to generate that no of fake images. For these images, the label is set to 0 as they are fake samples.

Now we train the generator.

This is one of the most important things to understand. How the generator is trained to fool the discriminator? This question is answered here. To train the generator we create batch_size no of noise vectors and we create the same number of labels set to 1. Now why this is set to 1 is crucial to understand. When we set the label to 1 it means we want to update the weights of the generator in such a way that the discriminator classifies them as 1 and hence as a result of this generator is optimized to generate better as a result of training.

Now since we have trained our GAN now let’s generate some example.

          def generate_and_visualize():
    l_vec=np.random.randn(latent_dim*10) 
    l_vec=l_vec.reshape(10,latent_dim)
    gen_images=g.predict(l_vec)
    
    count=0
    fig,ax=plt.subplots(2,5)
    for i in range(2):
      for j in range(5):
        ax[i][j].axis('off')
        ax[i][j].imshow(np.squeeze(gen_images[count],axis=-1),cmap='gray')
        count+=1
    plt.show()

generate_and_visualize()
        

Following was the output after the training.

Congrats you just learned how to create a GAN and generate MNIST digit. This is the first article on the GAN series and there will be more covering advanced and cool applications of GAN. So if you like this please share the blog across your network. And if you have some doubts regarding the above concepts please comment I will be happy to help.

References and further reading

Generative Adversarial Networks

We propose a new framework for estimating generative models via an adversarial process, in which we simultaneously train two models: a generative model G that captures the data distribution, and a discriminative model D that estimates the probability that a sample came from the training data rather than G. The training procedure for G is to maximize the probability of D making a mistake. This framework corresponds to a minimax two-player game. In the space of arbitrary functions G and D, a unique solution exists, with G recovering the training data distribution and D equal to 1/2 everywhere. In the case where G and D are defined by multilayer perceptrons, the entire system can be trained with backpropagation. There is no need for any Markov chains or unrolled approximate inference networks during either training or generation of samples. Experiments demonstrate the potential of the framework through qualitative and quantitative evaluation of the generated samples.

arxiv.org

How to Develop a GAN for Generating MNIST Handwritten Digits - MachineLearningMastery.com

Generative Adversarial Networks, or GANs, are an architecture for training generative models, such as deep convolutional neural networks for generating images. Developing a GAN for generating images requires both a discriminator convolutional neural network model for classifying whether a given image is real or generated and a generator model that uses inverse convolutional layers to […]

machinelearningmastery.com