How to perform image segmentation of apples using Python OpenCV?

Question

I have pictures of apple slices that have been soaked in an iodine solution. The goal is to segment the apples into individual regions of interest and evaluate the starch level of each one. This is for a school project so my goal is to test different methods of segmentation and objectively find the best solution whether it be a single technique or a combination of multiple techniques.

The problem is that so far I have only come close on one method. That method is using HoughCircles. I had originally planned to use the Watershed method, Morphological operations, or simple thresholding. This plan failed when I couldn't modify any of them to work.

The original images look similar to this, with varying shades of darkness of the apple

I've tried removing the background tray using cv2.inRange with HSV values, but it doesn't work well with darker apples.

This is what the HoughCircles produced on the original image with a grayscale and median blur applied, also with an attempted mask of the tray.

Any advice or direction on where to look next would be greatly appreciated. I can supply the code I'm using if that will help.

Thank you!

EDIT 1 : Adding some code and clarifying the question

Thank you for the responses. My real question is are there any other methods of segmentation that this scenario lends itself well to? I would like to try a couple different methods and compare results on a large set of photos. My next in line to try is using k-means segmentation. Also I'll add some code below to show what I've tried so far.

HSV COLOR FILTERING

import cv2
import numpy as np

# Load image
image = cv2.imread('ApplePic.jpg')

# Set minimum and max HSV values to display
lower = np.array([0, 0, 0])
upper = np.array([105, 200, 255])

# Create HSV Image and threshold into a range.
hsv = cv2.cvtColor(image, cv2.COLOR_BGR2HSV)
mask = cv2.inRange(hsv, lower, upper)
maskedImage = cv2.bitwise_and(image, image, mask=mask)

# Show Image
cv2.imshow('HSV Mask', image)
cv2.waitKey(0)

HoughCircles

# import the necessary packages
import numpy as np
import argparse
import cv2
import os

directory = os.fsencode('Photos\\Sample N 100')

for file in os.listdir(directory):

    filename = os.fsdecode(file)

    if filename.endswith('.jpg'):
        # Load the image
        image = cv2.imread('Photos\\Sample N 100\\' + filename)

        # Calculate scale
        scale_factor = 800 / image.shape[0]
        width = int(image.shape[1] * scale_factor)
        height = 800
        dimension = (width, height)
        min_radius = int((width / 10) * .8)
        max_radius = int((width / 10) * 1.2)

        # Resize image
        image = cv2.resize(image, dimension, interpolation=cv2.INTER_AREA)

        # Copy Image 
        output = image.copy()

        # Grayscale Image
        gray = cv2.medianBlur(cv2.cvtColor(image, cv2.COLOR_BGR2GRAY), 5)

        # Detect circles in image
        circles = cv2.HoughCircles(gray, cv2.HOUGH_GRADIENT, 1, min_radius * 2, 4, 60, 20,  min_radius, max_radius)

        # ensure at least some circles were found
        if circles is not None:
            # convert the (x, y) coordinates and radius of the circles to integers
            circles = np.round(circles[0, :]).astype("int")
            # loop over the (x, y) coordinates and radius of the circles
            for (x, y, r) in circles:
                # draw the circle in the output image, then draw a rectangle
                # corresponding to the center of the circle
                cv2.circle(output, (x, y), r, (0, 255, 0), 4)
                cv2.rectangle(output, (x - 5, y - 5), (x + 5, y + 5), (0, 128, 255), -1)
                cv2.putText(output, '(' + str(x) + ',' + str(y) + ',' + str(r) + ')', (x, y),
                        cv2.FONT_HERSHEY_COMPLEX_SMALL, 1, 255)


            # show the output image
            cv2.imshow("output", np.hstack([image, output, maskedImage]))
            cv2.waitKey(0)
        continue
    else:
        continue

Answer 1

An alternative approach to segmenting the apples is to perform Kmeans color segmentation before thresholding then using contour filtering to isolate the apple objects:

1. Apply Kmeans color segmentation. We load the image, resize smaller using imutils.resize then apply Kmeans color segmentation. Depending on the number of clusters, we can segment the image into the desired number of colors.

2. Obtain binary image. Next we convert to grayscale, Gaussian blur and Otsu's threshold.

3. Filter using contour approximation. We filter out non-circle contours and small noise.

4. Morphological operations. We perform a morph close to fill adjacent contours

5. Draw minimum enclosing circles using contour area as filter. We find contours and draw the approximated circles. For this we use two sections, one where there was a good threshold and another where we approximate the radius.

---

Kmeans color quantization with clusters=3 and binary image

Morph close and result

The "good" contours that had the radius automatically calculated using cv2.minEnclosingCircle is highlighted in green while the approximated contours are highlighted in teal. These approximated contours were not segmented well from the thresholding process so we average the "good" contours radius and use that to draw the circle.

Code

import cv2
import numpy as np
import imutils

# Kmeans color segmentation
def kmeans_color_quantization(image, clusters=8, rounds=1):
    h, w = image.shape[:2]
    samples = np.zeros([h*w,3], dtype=np.float32)
    count = 0

    for x in range(h):
        for y in range(w):
            samples[count] = image[x][y]
            count += 1

    compactness, labels, centers = cv2.kmeans(samples,
            clusters, 
            None,
            (cv2.TERM_CRITERIA_EPS + cv2.TERM_CRITERIA_MAX_ITER, 10000, 0.0001), 
            rounds, 
            cv2.KMEANS_RANDOM_CENTERS)

    centers = np.uint8(centers)
    res = centers[labels.flatten()]
    return res.reshape((image.shape))

# Load image, resize smaller, perform kmeans, grayscale
# Apply Gaussian blur, Otsu's threshold
image = cv2.imread('1.jpg')
image = imutils.resize(image, width=600)
kmeans = kmeans_color_quantization(image, clusters=3)
gray = cv2.cvtColor(kmeans, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (9,9), 0)
thresh = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)[1]

# Filter out contours not circle
cnts = cv2.findContours(thresh, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
for c in cnts:
    peri = cv2.arcLength(c, True)
    approx = cv2.approxPolyDP(c, 0.04 * peri, True)
    if len(approx) < 4:
        cv2.drawContours(thresh, [c], -1, 0, -1)

# Morph close
kernel = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (5,5))
close = cv2.morphologyEx(thresh, cv2.MORPH_CLOSE, kernel, iterations=3)

# Find contours and draw minimum enclosing circles 
# using contour area as filter
approximated_radius = 63
cnts = cv2.findContours(close, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
cnts = cnts[0] if len(cnts) == 2 else cnts[1]
for c in cnts:
    area = cv2.contourArea(c)
    x,y,w,h = cv2.boundingRect(c)
    # Large circles
    if area > 6000 and area < 15000:
        ((x, y), r) = cv2.minEnclosingCircle(c)
        cv2.circle(image, (int(x), int(y)), int(r), (36, 255, 12), 2)
    # Small circles
    elif area > 1000 and area < 6000:
        ((x, y), r) = cv2.minEnclosingCircle(c)
        cv2.circle(image, (int(x), int(y)), approximated_radius, (200, 255, 12), 2)

cv2.imshow('kmeans', kmeans)
cv2.imshow('thresh', thresh)
cv2.imshow('close', close)
cv2.imshow('image', image)
cv2.waitKey()