How to remove the background from an image

Question

I want to remove the background, and draw the outline of the box shown in the image(there are multiple such images with a similar background) . I tried multiple methods in OpenCV, however I am unable to determine the combination of features which can help remove background for this image. Some of the approaches tried out were:

• Edge Detection - Since the background itself has edges of its own, using edge detection on its own (such as Canny and Sobel) didn't seem to give good results.
• Channel Filtering / Thresholding - Both the background and foreground have a similar white color, so I was unable to find a correct threshold to filter the foreground.
• Contour Detection - Since the background itself has a lot of contours, just using the largest contour area, as is often used for background removal, also didn't work.

I would be open to tools in Computer Vision or of Deep Learning (in Python) to solve this particular problem.

Answer 1

The Concept

This is one of the cases where it is really useful to fine-tune the kernels of which you are using to dilate and erode the canny edges detected from the images. Here is an example, where the dilation kernel is np.ones((4, 2)) and the erosion kernel is np.ones((13, 7)):

The Code

import cv2
import numpy as np

def process(img):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    img_blur = cv2.GaussianBlur(img_gray, (3, 3), 2)
    img_canny = cv2.Canny(img_blur, 50, 9)
    img_dilate = cv2.dilate(img_canny, np.ones((4, 2)), iterations=11)
    img_erode = cv2.erode(img_dilate, np.ones((13, 7)), iterations=4)
    return cv2.bitwise_not(img_erode)

def get_contours(img):
    contours, _ = cv2.findContours(process(img), cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    cnt = max(contours, key=cv2.contourArea)
    cv2.drawContours(img, [cv2.convexHull(cnt)], -1, (0, 0, 255), 2)

img = cv2.imread("image2.png")
get_contours(img)
cv2.imshow("result", img)

cv2.waitKey(0)
cv2.destroyAllWindows()

The Output

Output for each of the two images provided:

Image 1:

Image 2:

Notes

Note that the processed image (which is binary) is inverted at cv2.bitwise_not(img_erode). Observe the processed version of both images (returned by the process() function defined above), with the inversion:

Processed Image 1:

Processed Image 2:

Tools

Finally, if you happen to have other images where the above program doesn't work properly on, you can use OpenCV Trackbars to adjust the values passed into the methods with the program below:

import cv2
import numpy as np

def process(img, b_k, b_s, c_t1, c_t2, k1, k2, k3, k4, iter1, iter2):
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    b_k = b_k // 2 * 2 + 1
    img_blur = cv2.GaussianBlur(img_gray, (b_k, b_k), b_s)
    img_canny = cv2.Canny(img_blur, c_t1, c_t2)
    img_dilate = cv2.dilate(img_canny, np.ones((k1, k2)), iterations=iter1)
    img_erode = cv2.erode(img_dilate, np.ones((k3, k4)), iterations=iter2)
    return cv2.bitwise_not(img_erode)

d = {"Blur Kernel": (3, 50),
     "Blur Sigma": (2, 30),
     "Canny Threshold 1": (50, 500),
     "Canny Threshold 2": (9, 500),
     "Dilate Kernel1": (4, 50),
     "Dilate Kernel2": (2, 50),
     "Erode Kernel1": (13, 50),
     "Erode Kernel2": (7, 50),
     "Dilate Iterations": (11, 40),
     "Erode Iterations": (4, 40)}

cv2.namedWindow("Track Bars")
for i in d:
    cv2.createTrackbar(i, "Track Bars", *d[i], id)

img = cv2.imread("image1.png")

while True:
    img_copy = img.copy()
    processed = process(img, *(cv2.getTrackbarPos(i, "Track Bars") for i in d))
    contours, _ = cv2.findContours(processed, cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    if contours:
        cnt = max(contours, key=cv2.contourArea)
        cv2.drawContours(img_copy, [cv2.convexHull(cnt)], -1, (0, 0, 255), 2)
    cv2.imshow("result", img_copy)
    
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cv2.waitKey(0)
cv2.destroyAllWindows()

Answer 2

You Can use Rembg (tool to remove images background). This will work well even with pre-trained weights. I tried for the test image, here is my results using Rembg

You can Simply Download Rembg using pip

pip install rembg

Remove the background from a single file

rembg i path/to/input.png path/to/output.png

Remove the background from all images in a folder

rembg p path/to/input path/to/output

Answer 3

You can try to use SIFT or SURF algorithms if you are given a perfect sample of what you are looking for. I took one from the 'intact' dataset as an example. The SIFT algorithm will try to match features in the sample with the actual image. From there, you can clean the matches and find an homography (RANSAC works best in this case) to find the contours.

Test image: Reference image (cut from a test one): Result (you can skip grayscale conversion):

The code I show here refers to execution in Colab with custom parameters you can tune some more.

import cv2 
import numpy as np
from google.colab.patches import cv2_imshow

img = cv2.imread("reference_1.png", cv2.IMREAD_GRAYSCALE) 
frame = cv2.imread("top.png", cv2.IMREAD_GRAYSCALE)

# if SIFT_create() gives problems, try downgrading opencv with
# pip uninstall opencv-python
# pip install opencv-contrib-python==3.4.2.17
sift = cv2.xfeatures2d.SIFT_create() 
kp_image, desc_image = sift.detectAndCompute(img, None) 
kp_frame, desc_frame = sift.detectAndCompute(frame, None) 

index_params = dict(algorithm=0, trees=5) 
search_params = dict() 
flann = cv2.FlannBasedMatcher(index_params, search_params)
matches = flann.knnMatch(desc_image, desc_frame, k=2)

# clean the matches
good_points=[] 
for m, n in matches: 
    if(m.distance < 0.6 * n.distance): 
        good_points.append(m)

query_pts = np.float32([kp_image[m.queryIdx].pt for m in good_points]).reshape(-1, 1, 2) 
train_pts = np.float32([kp_frame[m.trainIdx].pt for m in good_points]).reshape(-1, 1, 2)

# find homography to find mask
matrix, mask = cv2.findHomography(query_pts, train_pts, cv2.RANSAC, 5.0) 
matches_mask = mask.ravel().tolist()
h,w = img.shape
pts = np.float32([ [0,0],[0,h-1],[w-1,h-1],[w-1,0] ]).reshape(-1,1,2)
dst = cv2.perspectiveTransform(pts, matrix)
homography = cv2.polylines(frame, [np.int32(dst)], True, (255, 0, 0), 3) 

cv2_imshow(homography)