CV - Extract differences between two images

Question

I am currently working on an intrusion system based on video surveillance. In order to complete this task, I take a snapshot of the background of my scene (assume it's totally clean, no people or moving objects). Then, I compare the frame I get from the (static) video camera and look for the differences. I have to be able to check any differences, not only human shape or whatever, so I cannot specific feature extraction.

Typically, I have:

I am using OpenCV, so to compare I basically do:

cv::Mat bg_frame;
cv::Mat cam_frame;
cv::Mat motion;

cv::absdiff(bg_frame, cam_frame, motion);
cv::threshold(motion, motion, 80, 255, cv::THRESH_BINARY);
cv::erode(motion, motion, cv::getStructuringElement(cv::MORPH_RECT, cv::Size(3,3)));

Here is the result:

As you can see, the arm is stripped (due to color differential conflict I guess) and this is sadly not what I want.

I thought about add the use of cv::Canny() in order to detect the edges and fill the missing part of the arm, but sadly (once again), it only solves the problem in few situation not most of them.

Is there any algorithm or technique I could use to obtain an accurate difference report?

PS: Sorry for the images. Due to my newly subscription, I do not have enough reputation.

EDIT I use grayscale image in here, but I am open to any solution.

Answer 1

I use Python, this is my result:

The code:

# 2017.12.22 15:48:03 CST
# 2017.12.22 16:00:14 CST
import cv2
import numpy as np

img1 = cv2.imread("img1.png")
img2 = cv2.imread("img2.png")
diff = cv2.absdiff(img1, img2)
mask = cv2.cvtColor(diff, cv2.COLOR_BGR2GRAY)

th = 1
imask =  mask>th

canvas = np.zeros_like(img2, np.uint8)
canvas[imask] = img2[imask]

cv2.imwrite("result.png", canvas)

---

Update, here is C++ code:

//! 2017.12.22 17:05:18 CST
//! 2017.12.22 17:22:32 CST

#include <opencv2/opencv.hpp>
#include <iostream>
using namespace std;
using namespace cv;
int main() {

    Mat img1 = imread("img3_1.png");
    Mat img2 = imread("img3_2.png");

    // calc the difference
    Mat diff;
    absdiff(img1, img2, diff);

    // Get the mask if difference greater than th
    int th = 10;  // 0
    Mat mask(img1.size(), CV_8UC1);
    for(int j=0; j<diff.rows; ++j) {
        for(int i=0; i<diff.cols; ++i){
            cv::Vec3b pix = diff.at<cv::Vec3b>(j,i);
            int val = (pix[0] + pix[1] + pix[2]);
            if(val>th){
                mask.at<unsigned char>(j,i) = 255;
            }
        }
    }

    // get the foreground
    Mat res;
    bitwise_and(img2, img2, res, mask);

    // display
    imshow("res", res);
    waitKey();
    return 0;
}

---

Similar answers:

1. CV - Extract differences between two images

2. While finding a difference between 2 pictures OpenCV difference is bigger than it is supposed to be

Answer 2

Another technique to obtain the exact pixel differences between two images is to use the Structural Similarity Index (SSIM) first introduced in the paper Image Quality Assessment: From Error Visibility to Structural Similarity. This method can be used to determine if two images are identical and/or showcase differences due to tiny image discrepancies. SSIM is already implemented in the scikit-image library for image processing as skimage.metrics.structural_similarity()

The structural_similarity() function returns a score and a difference image, diff. The score represents the mean structural similarity index between the two input images and can fall between the range [-1,1] with values closer to one representing higher similarity. But since you're only interested in where the two images differ, the diff image is what we'll focus on. Specifically, the diff image contains the actual image differences with darker regions having more disparity. Larger areas of disparity are highlighted in black while smaller differences are in gray.

---

Using these two input images

We get this result

The SSIM score after comparing the two images show that they are very similar

Image Similarity: 95.8648%

from skimage.metrics import structural_similarity
import cv2

# Load images as grayscale
image1 = cv2.imread('1.png', 0)
image2 = cv2.imread('2.png', 0)

# Compute SSIM between the two images
(score, diff) = structural_similarity(image1, image2, full=True)

# The diff image contains the actual image differences between the two images
# and is represented as a floating point data type in the range [0,1] 
# so we must convert the array to 8-bit unsigned integers in the range
# [0,255] image1 we can use it with OpenCV
diff = (diff * 255).astype("uint8")
print("Image Similarity: {:.4f}%".format(score * 100))

cv2.imshow('diff', diff)
cv2.waitKey()

Answer 3

This is well-known classic computer vision problem called background subtraction. There are many approaches which can be used to solve this problem, most of them are already implemented, so I think you should first take a look at multiple existing algorithms, here is opensource implementation of most of them: https://github.com/andrewssobral/bgslibrary (I personally found SUBSENSE giving best results, but its deadly slow)