Unet two images as a input and two images as a output in single pass?

Question

I am building a UNET model . I got stucked in one thing that if we have to pass two images as a input to UNET , we can complete it by concat them both just like below code

inputs = []
for _ in range(num_inputs):
    inputs.append(Input((self.input_height, self.input_width, self.input_features)))
x = concatenate(inputs)
x = Conv2D(32, (3, 3), activation='relu', padding='same')(x)

But how can we get two images as output in single pass ?

Answer 1

An approach would be set your output layer as a 2-Channel layer (one channel for every image). One aspect to have in consideration for this approach, is the label formatting. Your label has to have 2 channels as well, so the Unet can compare the output-channels with the labels (the 2 images in this case) and back propagate the error correctly. Then you can uncouple the 2 channels and work with the images separately.

Extra: If your images have multiple channels (e.g. RGB) then you can still concatenate it and generate a backpropagation-error if you format the label correctly as well. In this case: concatenation(1 image of 3 channels, 1 image 3 channels) = 6 channels.

Answer 2

I have the same problem as yours and the efficient way that actually works very well for me is to make both input images into one tensor of shape (6, HEIGHT, WEIGHT) if both images as RGB, if they are not you will have the shape (2, HEIGHT, WEIGHT). And do the same for the output images. Finally, you can apply some simple PP to make extract the images from the output tensor.

Answer 3

U-Net is used for image segmentation. Why would you pass two images as input? Do the segmentation of these two images depend on each other? If not I recommend to simply create two independent U-Net models.

In any case, U-Net has as many outputs as inputs by construction so providing the concatenated images should suffice.