Classifying my Own Images

When I was an undergraduate, I kept a diary of my daily life and I’ve always wanted to digitise it, to keep it safe. I kept it over the course of five years, and I have scanned in a limited number of pages (so far). As mentioned, as a long-term goal, I’d quite like to digitise the written words into plain text. But for now, the scanned images provide a useful resource with which to practise my machine learning skills. You can see an example scan above.

I wrote a quick Python program to split these images into left and right pages (with the split point selected manually), to give 700 x 1000 pixel images.

My aim was to build a Keras model using ConvNets which would be able to determine whether a given page is a left or right page: a simple binary classification problem.

Reading the Images into Python

The first step in this process was to read the images from files on my computer into tensors, suitable for processing by the Keras model. I also needed to create a training set, a validation set and a test set with labels (left or right), to be used for training and then assessing the model.

I have recently read about Python Generators in the excellent “Deep Learning with Python” book (François Chollet, published by Manning), in particular the ImageDataGenerator provided as part of Keras. This provides a means of reading the images into input tensors with many different options, I used the mechanism that also infers the labels from the directory structure that you provide.

I created the following directory structure for use by the generator and placed the images in the correct locations. I split my 320 images into three sets (allocated randomly): training (240), validation (40) and test (40). Each of these was then divided into left and right directories.

A generator is then created as in the following code block, it will be used later to train the model using the fit_generator (instead of “fit”) function of the model. In a similar manner the validation and test generators were also created.

batch_size = 10
train_datagen = ImageDataGenerator(rescale=1./255)
train_generator = train_datagen.flow_from_directory(
                                                directory="images\\" + image_path + "\\train",
                                                class_mode="binary",
                                                target_size=(500, 350),
                                                classes=["left","right"],
                                                batch_size=batch_size,
                                                shuffle=True)

This code first converts the [0 to 255] RGB values of the images to [0.0 to 1.0] and also reduces their size to 350 x 500 in order to reduce the model size. Other transformations can also be applied by ImageDataGenerator, allowing it to be used for data augmentation (that is, creating extra training data by transforming the same images in some way and adding them to the data).

Remember, image dimensions are usually represented as width x height (e.g. 350 x 500) whereas Python NumPy arrays are described as (rows, columns) (e.g. target_size=(500, 350)). A bit confusing, but I think we can cope!

The Model

As these are images, I decided to use ConvNets to process them (although this is such a simple problem that straight Dense layers may have sufficed). For a first attempt, I added two ConvNet layers, with Max Pooling in-between, followed by a Dense layer before the final single-node classifier. Here’s the code.

model = models.Sequential()
model.add(layers.Conv2D(32, (3,3), activation="relu", input_shape=(500, 350, 3)))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Conv2D(64, (3,3), activation="relu"))
model.add(layers.MaxPooling2D((2,2)))
model.add(layers.Flatten())
model.add(layers.Dense(256, activation="relu"))
model.add(layers.Dense(1, activation="sigmoid"))

Compiling and Training the Model

As mentioned earlier, I used the fit_generator function to train the model. My first attempts at running this code resulted in an accuracy of 0.5, which meant that the model was no better than chance. After checking my code, I started tweaking the learning rate and eventually found the best value for the validation set.

num_train_images = 240
num_val_images = 40
steps_per_epoch = num_train_images / batch_size
val_steps_per_epoch = num_val_images / batch_size
learning_rate = 0.00001                                     # tune this!

model.summary()

model.compile(loss="binary_crossentropy", optimizer=optimizers.rmsprop(lr=learning_rate), metrics=["acc"])

history = model.fit_generator(train_generator, steps_per_epoch=steps_per_epoch, epochs=30, validation_data=validation_generator, validation_steps=val_steps_per_epoch, shuffle=True)

# Save the trained model for later use
model.save("left_right_page_v2.h5")

Here’s the output.

Using TensorFlow backend.
Found 40 images belonging to 2 classes.
Found 240 images belonging to 2 classes.
_________________________________________________________________
Layer (type)                 Output Shape              Param #
=================================================================
conv2d_1 (Conv2D)            (None, 498, 348, 32)      896
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 249, 174, 32)      0
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 247, 172, 64)      18496
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 123, 86, 64)       0
_________________________________________________________________
flatten_1 (Flatten)          (None, 676992)            0
_________________________________________________________________
dense_1 (Dense)              (None, 256)               173310208
_________________________________________________________________
dense_2 (Dense)              (None, 1)                 257
=================================================================
Total params: 173,329,857
Trainable params: 173,329,857
Non-trainable params: 0
_________________________________________________________________
Epoch 1/30
2019-02-17 13:46:29.456091: I tensorflow/core/platform/cpu_feature_guard.cc:141] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2
24/24 [==============================] - 72s 3s/step - loss: 0.8597 - acc: 0.5375 - val_loss: 0.6948 - val_acc: 0.5000
Epoch 2/30
24/24 [==============================] - 66s 3s/step - loss: 0.6978 - acc: 0.5417 - val_loss: 0.7016 - val_acc: 0.5000
Epoch 3/30
24/24 [==============================] - 66s 3s/step - loss: 0.6820 - acc: 0.5792 - val_loss: 0.6656 - val_acc: 0.7750
Epoch 4/30
24/24 [==============================] - 66s 3s/step - loss: 0.6754 - acc: 0.5625 - val_loss: 0.6561 - val_acc: 0.5000
Epoch 5/30
24/24 [==============================] - 66s 3s/step - loss: 0.6678 - acc: 0.5583 - val_loss: 0.6603 - val_acc: 0.5000
Epoch 6/30
24/24 [==============================] - 66s 3s/step - loss: 0.6502 - acc: 0.6167 - val_loss: 0.6397 - val_acc: 0.5000
Epoch 7/30
24/24 [==============================] - 65s 3s/step - loss: 0.6351 - acc: 0.6875 - val_loss: 0.6171 - val_acc: 0.8250
Epoch 8/30
24/24 [==============================] - 64s 3s/step - loss: 0.6136 - acc: 0.7792 - val_loss: 0.6395 - val_acc: 0.5000
Epoch 9/30
24/24 [==============================] - 65s 3s/step - loss: 0.6128 - acc: 0.7458 - val_loss: 0.6155 - val_acc: 0.5000
Epoch 10/30
24/24 [==============================] - 64s 3s/step - loss: 0.6023 - acc: 0.7250 - val_loss: 0.5707 - val_acc: 1.0000
Epoch 11/30
24/24 [==============================] - 65s 3s/step - loss: 0.5654 - acc: 0.8083 - val_loss: 0.5663 - val_acc: 0.6750
Epoch 12/30
24/24 [==============================] - 65s 3s/step - loss: 0.5489 - acc: 0.7708 - val_loss: 0.5789 - val_acc: 0.5000
Epoch 13/30
24/24 [==============================] - 65s 3s/step - loss: 0.5454 - acc: 0.8750 - val_loss: 0.5184 - val_acc: 1.0000
Epoch 14/30
24/24 [==============================] - 65s 3s/step - loss: 0.5056 - acc: 0.9083 - val_loss: 0.5146 - val_acc: 0.7750
Epoch 15/30
24/24 [==============================] - 65s 3s/step - loss: 0.4935 - acc: 0.8667 - val_loss: 0.4846 - val_acc: 0.9750
Epoch 16/30
24/24 [==============================] - 65s 3s/step - loss: 0.4691 - acc: 0.9292 - val_loss: 0.4579 - val_acc: 1.0000
Epoch 17/30
24/24 [==============================] - 65s 3s/step - loss: 0.4509 - acc: 0.9167 - val_loss: 0.4403 - val_acc: 1.0000
Epoch 18/30
24/24 [==============================] - 65s 3s/step - loss: 0.4289 - acc: 0.9583 - val_loss: 0.4327 - val_acc: 0.9250
Epoch 19/30
24/24 [==============================] - 65s 3s/step - loss: 0.4057 - acc: 0.9500 - val_loss: 0.4009 - val_acc: 1.0000
Epoch 20/30
24/24 [==============================] - 65s 3s/step - loss: 0.3794 - acc: 0.9792 - val_loss: 0.3798 - val_acc: 1.0000
Epoch 21/30
24/24 [==============================] - 65s 3s/step - loss: 0.3768 - acc: 0.9750 - val_loss: 0.3576 - val_acc: 1.0000
Epoch 22/30
24/24 [==============================] - 65s 3s/step - loss: 0.3380 - acc: 0.9833 - val_loss: 0.3392 - val_acc: 1.0000
Epoch 23/30
24/24 [==============================] - 65s 3s/step - loss: 0.3211 - acc: 0.9833 - val_loss: 0.3168 - val_acc: 1.0000
Epoch 24/30
24/24 [==============================] - 64s 3s/step - loss: 0.3009 - acc: 0.9833 - val_loss: 0.3129 - val_acc: 1.0000
Epoch 25/30
24/24 [==============================] - 64s 3s/step - loss: 0.2826 - acc: 0.9958 - val_loss: 0.2970 - val_acc: 1.0000
Epoch 26/30
24/24 [==============================] - 64s 3s/step - loss: 0.2631 - acc: 1.0000 - val_loss: 0.2744 - val_acc: 1.0000
Epoch 27/30
24/24 [==============================] - 65s 3s/step - loss: 0.2441 - acc: 0.9958 - val_loss: 0.2424 - val_acc: 1.0000
Epoch 28/30
24/24 [==============================] - 65s 3s/step - loss: 0.2342 - acc: 0.9875 - val_loss: 0.2348 - val_acc: 1.0000
Epoch 29/30
24/24 [==============================] - 64s 3s/step - loss: 0.2140 - acc: 1.0000 - val_loss: 0.2136 - val_acc: 1.0000
Epoch 30/30
24/24 [==============================] - 64s 3s/step - loss: 0.1897 - acc: 1.0000 - val_loss: 0.1956 - val_acc: 1.0000

Pretty good results on the validation set, I’m sure you’ll agree!

Plotting the Accuracy and Loss

Finally, I used the following code to plot the results.

# Show plots of training loss and accuracy
loss = history.history["loss"]
val_loss = history.history["val_loss"]

acc = history.history["acc"]
val_acc = history.history["val_acc"]

epochs = range(1, len(acc) + 1)

plt.plot(epochs, loss, 'bo', label="Training loss")
plt.plot(epochs, val_loss, 'b', label="Validation loss")
plt.title("Training and validation loss")
plt.legend()

plt.figure()

plt.plot(epochs, acc, 'bo', label="Training acc")
plt.plot(epochs, val_acc, 'b', label="Validation acc")
plt.title("Training and validation accuracy")
plt.legend()

plt.show()

And here are the results.