Image recognition

Someone at work told me he had a lot of things to do, he had to look the bill consecutive number and write it down in an Excel sheet. When the company sell something they emit a bill and that bill is send personally to the buyer. Then the buyer put a stamp on it and the date they received the bill. One of the jobs of this person is to assure that all emitted bill are received with the buyer stamp, because that is a legal implication that the client owns some money. He has to put the bill consecutive number and the date in the excel sheet.

We received probably 90 bill per day and on Monday we received ~300, so this guy has a lot of work to do. I offer him a hand in exchange of some work I hate to do (inventory). I tell him it was possible to make a program that create an excel file with the bill consecutive number by using the scan image. With this procedure he know takes by far 10 minutes to scan 300 bills and now he is happier at job (but I'm not because he didn't do inventory for me).

When I offer the hand, I have no clue how to do that I just now it was possible and probably wouldn't be that hard since python has libraries for almost everything. I did some research in the web and found the right information to do the job. I'm going to explain here the approach I went trough. I will be glad if by exchange people reading this post and find it useful correct my spelling mistakes because English is not my first language. This could be a nice way to contribute.

Ok so in summary the program do three things 1) Found an image inside an image 2) find the coordinates of each of the consecutive bill number and 3) run a classification algorithm.

Lets take a look to some random bill I download from google:

We are interesting in the number surrounding by the red box. Since all bills are the same, this number is going to be some pixels under "Account Number" this is perfect because we can fin this especific space in the image using scikit-image librarie. We have to save another image (the template we are going to find) which is a cropped image of the original bill:

The first thing the program will do is to find the template in the bill and extract the square that contain the bill number. We need to have installed scikit-image in python, the anaconda distribution already comes with this library installed so if you download Anaconda python you are good to go.

We load the Image module from PIL library and load the images. Then we convert it to a gray scale and use the function "match_template" from the skimage library.

from PIL import Image
from skimage.feature import match_template
import numpy as np

bill = Image.open('example_bill.jpg').convert('L')
template = Image.open('template.jpg').convert('L')

billA = np.array(bill)
tempA = np.array(template)

## http://scikit-image.org/docs/dev/auto_examples/features_detection/plot_template.html
result = match_template(billA, tempA)
ij = np.unravel_index(np.argmax(result), result.shape)
x, y = ij

After you load the image you have to convert it to an array to pass them to the function match_template. This function computes the correlation between the template and each of the pixel. With the instruction np.unravel_index(np.argmax(result), result.shape) we are getting the position of the pixel with the highest correlation with our template image. What we need now is to isolate the consecutive bill number from the rest of the bill, for this we extract a rectangle arround the pixel with highest correlation:

xx = x + tempA.shape[0] + 20
y -= 10
yy = y + tempA.shape[1] + 60

billNumber = billA[x:xx, y:yy]

This code result in the following image:

Notice that x represent the vertical position of the image and y the horizontal position. Other thing no notice is that ee don't really need the ACCOUNT NUMBER writting in top of the bill number, we can remove it setting x and xx like this:

x += 10
xx = x + tempA.shape[0] + 10
y -= 10
yy = y + tempA.shape[1] + 60

billNumber = billA[x:xx, y:yy]

This will give a cleaner view of the part we are really interest:

We have finish the firt job finding our template image and isolating the bill number. In the following post I will do the second part that is isolating each of the consecutive bill number into a square of a fix dimension.