How to detect this kind of artifacts/noises in my image?

Question

I've been processing some image frames in videos and I discovered that sometimes one or two frames of the video will have artifacts or noise like the images below:

The artifacts look like abrasions of paint with noisy colors that covers only a small region (less than 100x100 in a 1000x2000 frame) of the image. I wonder if there are ways to detect the noisy frames? I've tried to use the difference of frames with SSIM, NMSE or PSNR but found limited effectiveness. Saliency map (left) or sobel/scharr filtering (right) providing more obvious view but regular borders are also included and I'm not sure how to form a classifier.

Scharr saliency map:

Since they are only a few frames in videos it's not quite necessary to denoising and I can just remove the frames one detected. The main problem here is that it's difficult to distinguish those frames in playing videos.

Can anybody offer some help here?

Answer 1

Detailing the comment as an answer with a few more details:

The Scharr and saliency map looks good.

Thresholding will result in a binary image which can be cleaned up with morphological filters (erode to enhance artefacts, dilate to 'erase' gradient contours).

Finding contours will result in lists of points which can be further processed/filtered using contour features.

If the gradients are always bigger than the artefacts, contour features, such as the bounding box dimensions and aspect ratio should help segment artefact contours from gradient contours (if any: hopefully dilation would've cleaned up the thresholded/binary image).

Another idea could be looking into oriented gradients:

• either computer the oriented gradients (see visualisations): with the right cell size you might strike a balance where the artefacts have a high magnitude while gradient edges don't
• you could try a full histogram of oriented gradients (HoG) classifier setup (using an SVM trained on histograms (as features)) The above options do rely on hand crafted features/making assumptions about the size of artefacts.

ML could be an interesting route too, hopefully it can generalise well enough. Depending how many example images you have available, you could test a basic prototype using Teachable Machine (which behind the scenes would apply KNN to a transfer learning layer on top of MobileNet (or similar net)) fairly fast.

(Note: I've posted OpenCV Python links, but there are libraries that can help (e.g. scikit-image, scikit-learn, kornia, etc. in Python, cvv in c++, BoofCV in java, etc. (and there might be toolboxes for Matlab/Octave with similar features))