Image Search using Image Similarity Measures

Question

I am working on a project which is image searching engine. The logic behind is that there will be some images stored in a database and the user will input a new image and it will be matched with the ones stored in the database. The result will be a list of closest match of the query image with stored images in the database.

The images are of stamps. Now the problem is that there are New and Used stamps. New is just a stamp image and Used is with a some part of it obscured by a black cancellation mark so it cannot be a perfect match.

Here are few sample of both (New and Used):

I have used various measures, such as compare_mse, compare_ssim and compare_nrmse. But they all tilt towards dissimilarity. I have also used the https://github.com/EdjoLabs/image-match algorithm but it also is same giving low similarity score.

Do you guys think I neeed to use some preprocessing or something? I have also removed black borders from the image but the result is somewhat better not satisfactory though. I have converted them into gray-scale and matched, still no satisfactory results. Any recommendations and suggestion on how to get high similarity scores would be greatly appreciated! Here is my code:

img1 = cv2.imread('C:\\Users\\Quin\\Desktop\\1frclean2.jpg', cv2.IMREAD_GRAYSCALE)
img2 = cv2.imread('C:\\Users\\Quin\\Desktop\\1fr.jpg', cv2.IMREAD_GRAYSCALE)
compare_mse(cv2.resize(img1, (355, 500)), cv2.resize(img2, (355, 500)))
compare_ssim(cv2.resize(img1, (355, 500)), cv2.resize(img2, (355, 500)))

MSE returned 4797.232123943662 and SSIM returned 0.2321816144043102.

Answer 1

MSE and SSIM are not good for the problem, as they aim at pixel by pixel comparison. Here is an article by NVIDIA showing, for example, how SSIM dramatically fails for even simple cases.

Here is a collection of research papers on image similarity to understand what is possible.

Two ideas that could be relevant:

1. Reverse search with embeddings.

2. Since stamps usually contain text, you might use some vision cloud service to recognize/OCR it, and then compare the text strings for similarity.

Answer 2

The classical approach:

• normalize the images in position, size and angle, as much as possible; for this, you will need a reliable way to find the outer edges and/or the corners; as the aspect ratios can differ, you can normalize the height and adjust the width accordingly;

• to recognize a given sample, normalize it and compare to the reference images with a width that is compatible; you can use the normalized grayscale correlation, which is not sensitive to changes in illumination; you make it sensitive to colors by handling the three color planes independently;

• you can work at full scale or reduced scale (for speed); or at reduced scale, keep the best candidates and discriminate at a higher scale;

• it can also be useful to try a few neighboring positions and keep the best, to deal with imperfect alignment.

To set an acceptance threshold on the matching score, you can test with a few sample, look at matching scores for the same and for different stamps. Hopefully, the worst score for matching stamps (even with a cancellation mark) will be better than the best score for mismatching stamps.

---

The "modern approach": deep learning.