How do I return a dict + an image from a FastAPI endpoint?

Question

I am trying to use FastAPI to allow my (dockerized) server to respond to API calls returning

• an image image, and
• a dictionary additional_dict

(for a machine learning example, this could be classification labels from a classifier and a saliency map).

In my case, I think it makes sense to make use the same endpoint to get the two objects, because these get generated with the same computations.

I can successfully return an image as binary using something along the lines of https://stackoverflow.com/a/55905051/4240413:

from PIL import Image
from fastapi import FastAPI, File
import tempfile
from starlette.responses import FileResponse

@app.post("/getstuff")
async def get_image_and_data(file: bytes = File(...)):
    
    image, additional_dict = process_image(image)

    with tempfile.NamedTemporaryFile(mode="w+b", suffix=".png", delete=False) as outfile:
        image.save(outfile)
        return FileResponse(outfile.name, media_type="image/png")

This allows me even to see the image response when I invoke the service through the swagger UI on localhost:8000/docs.

However, I don't know how to get the image binary and the dictionary together.

I tried to replace my return statement with:

return FileResponse(outfile.name, media_type="image/png"), additional_dict

but this doesn't really work (when trying on swagger at localhost:8000/docs, I just get the json reply below, with a path to the temp file created)

[{
"path": "/tmp/tmpldwe_79d.png",
"status_code": 200,
"filename": null,
"send_header_only": false,
"media_type": "image/png",
"background": null,
"raw_headers": [
    [
    "content-type",
    "image/png"
    ]
],
"stat_result": null
},
{
"foo": 1,
"bar": 2
}]

It is possible to get in the response the binary image and the additional dict, from the same endpoint? If so, what's the best way to do it?
Is it possible to have the image rendered in the Swagger UI /docs and read the dict values there?

Answer 1

You can encode binary data (it would be in image in your case) with base64 and send the encoded string via dictionary.

   import base64

   with open("image.png", "rb") as image_file:
       encoded_image_string = base64.b64encode(image_file.read())

   payload = {
       "mime" : "image/png",
       "image": encoded_image_string,
       "some_other_data": None
   }

So this dictionary will contain the base64 encoded image and any other data fields.

On the front-end you may decode an image from base64 string back to bytes and present it to the user.