How to get rid of the CORS error when sending email with attachments with Gmail API?

Question

I know there are plenty of articles on the topic but none of them helped me solve this issue.

I have a website where users can connect their Gmail account and manage their inbox. I make requests (to get, send, trash etc. messages) from the browser in javascript using an access token which I got from my Server.

When sending an Email with attachments, I always get:

Access to fetch at 'https://gmail.googleapis.com/upload/gmail/v1/users/me/messages/send?uploadType=multipart' from origin 'http://localhost:8000' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

Along with a:

POST https://gmail.googleapis.com/upload/gmail/v1/users/me/messages/send?uploadType=multipart net::ERR_FAILED 200

But the message is sent successfully anyway !

In the Google Cloud Console, under APIs & Services > Credentials, in the settings of my "OAuth 2.0 Client ID", I added "http://localhost:8000" in the "Authorized JavaScript origins", waited 5 minutes and cleared my cache. But still get the CORS error.

I installed this Chrome extension https://chrome.google.com/webstore/detail/allow-cors-access-control/lhobafahddgcelffkeicbaginigeejlf and it removes the error but I obviously can't ask my user to do that.

I thought maybe it could be due to an error in my request headers or body but the email is being sent so... Any ideas how to get rid of that error ?