Interactive labeling of images in jupyter notebook

Question

I have a list of pictures:

pictures = {im1,im2,im3,im4,im5,im6}

Where

im1:

im2:

im3:

im4:

im5:

im6:

I want to assign the pictures to labels (1,2,3,4 etc.)

For instance, here pictures 1 to 3 belong to label 1, picture 4 belongs to label 2, picture 5 to label 3, and picture 6 to label 4.

-> label = {1,1,1,2,3,4}

Since I need to see the images when I label them, I need a method to do that while labeling them. I was thinking of creating an array of images:

And then I define the ranges by clicking on the first and last picture belonging to the same labels, so for example:

What do you think ? Is this somehow possible ?

I would like to assign different labels to different ranges of pictures.

For instance: When one has finished selecting the first label one could indicate it by a Double-click and then do the selection of the second label range, then Double-click, then do the selection of the third label range, then Double-click, then do the selection of the fourth label range, etc.

It does not have to be Double-clicking to change the selection of the labels, it could also just be a buttom or any other idea that you might have.

In the end one should have the list of labels.

Answer 1

Essentially, most of the interaction you are looking for boils down to being able to display images, and detect clicks on them in real time. As that is the case, you can use the jupyter widgets (aka ipywidgets) module to achieve most (if not all) of what you are looking for.

Take a look at the button widget which is described here with explanation on how to register to its click event. The problem - we can't display an image on a button, and I didn't find any way to do this within the ipywidgets documentation. There is an image widget, but it does not provide an on_click event. So construct a custom layout, with a button underneath each image:

COLS = 4
ROWS = 2
IMAGES = ...
IMG_WIDTH = 200
IMG_HEIGHT = 200

def on_click(index):
    print('Image %d clicked' % index)

import ipywidgets as widgets
import functools

rows = []

for row in range(ROWS):
    cols = []
    for col in range(COLS):
        index = row * COLS + col
        image = widgets.Image(
            value=IMAGES[index], width=IMG_WIDTH, height=IMG_HEIGHT
        )
        button = widgets.Button(description='Image %d' % index)
        # Bind the click event to the on_click function, with our index as argument
        button.on_click(functools.partial(on_click, index))

        # Create a vertical layout box, image above the button
        box = widgets.VBox([image, button])
        cols.append(box)

    # Create a horizontal layout box, grouping all the columns together
    rows.append(widgets.HBox(cols))

# Create a vertical layout box, grouping all the rows together
result = widgets.VBox(rows)

You can technically also write a custom widget to display an image and listen for a click, but I simply don't believe it's worth your time and effort.

Good luck!

Answer 2

The qsl package provides widgets that do this. For your case, the following code would allow you to label images in batches. Full disclosure, qsl is a project I started because I, like you, wanted to label images from inside Jupyter notebooks.

import qsl
from IPython.display import display

labeler = qsl.MediaLabeler(
    items=[
      {"target": "https://i.stack.imgur.com/cML6z.jpg"},
      {"target": "https://i.stack.imgur.com/6EVAP.jpg"},
      {"target": "https://i.stack.imgur.com/CAxUw.jpg"},
      {"target": "https://i.stack.imgur.com/8fhan.jpg"},
      {"target": "https://i.stack.imgur.com/eMXn5.jpg"},
      {"target": "https://i.stack.imgur.com/YFBfM.jpg"}
    ],
    # Optional, you can also configure the labeler from
    # the UI.
    config={
      "image": [
        {
          "name": "Type",
          "options": [
            {"name": "Foo"},
            {"name": "Bar"}
          ]
        }
      ]
    },
    # Optional, set to 1 if you want to label
    # one image at a time.
    batch_size=4,
    # Optionally, save labels to JSON. You
    # can also get the labels using `labeler.items`.
    jsonpath="labels.json"
)
display(labeler)

This generates a UI that looks like this.

Here is a Google Colab notebook that shows how to do this in Google Colab.