Why does Google Cloud Storage freeze when I try to upload a large folder (2.5GB of images)?

Question

After getting frustrated with Azure, I decided to try GCP. I wanted to try training a deep learning image classification model using GCP. To start off, I went to Cloud Storage through the Google Cloud Console UI and made a bucket to upload my images and... it freezes when I try to upload my image folder. Not a good first experience.

I've tried half a dozen times. I always get the little popup "upload started," but then it freezes. If I close it and reopen, the bucket will have made the folder and downloaded 3 or so images, but even if I let it run all night, it never makes progress beyond that. It just stops uploading and becomes nonresponsive. I have about 38,000 images, in total the folder size that I'm trying to upload is 2.5 GB.

There is no error message, and no sign that anything went wrong when I reload aside from the fact that only 3 or so images downloaded. Nothing else on my computer freezes. If I have other tabs open in the browser, they do not freeze. The tab with Google Console in it simply silently freezes. I tried Data Transfer to move the files to Google directly from Azure Blob Storage, but Azure wouldn't give Google permission to access the storage.

Can someone recommend a possible work-around to this? Should I be uploading my files some other way?

Answer 1

Whether or not a large number of files can be uploaded via the Console in-browser appears to be a bit sporadic. In my particular case, attempting to upload a large folder (not large files in a folder, a large NUMBER of files in a folder) at once results in the browser freezing due to memory problems in GCP's Cloud Storage AND in AWS's S3 when I try to upload it through their browser portals. I have tried this on Windows 10, on Linux Mint, and tried on both Google Chrome and Firefox to no avail.

Some GCP and AWS users report freezing or crashes when the number of files in the folder is large, but not necessarily when the size of the folder itself is large. This is my experience exactly--I can upload a large file or a folder with a few hundred files through Cloud Console, but not a folder with many thousands of files.

The only stable solution I've found is to surrender and abandon trying to upload a massive number of files in a folder via browser portals and instead upload via CLI. It isn't the solution I'd hoped for, but it is the only one I've found.

EDIT:

Since this correct (and only working) solution was downvoted for some reason,

See the comments to Kainax's answer to this question for multiple AWS users experiencing this problem.

See the question and answers to this question for the same issue I had in GCP.

Since this is an issue and not a feature, there is no official documentation on it.

Neither I nor anyone else as yet could solve this through the Cloud Console and this has been a recurring issue for years with some users. The only clear workaround is to avoid using the Console UI altogether when uploading a folder with a large number of files.

Answer 2

Tried to recreate image folder with 2G+ size and upload it on bucket using console. Below steps are working on my end.

1. Create Image files up to 2G+
2. Cloud Storage Browser > Select Bucket
3. Upload Folder > Select Folder > Upload
4. On/Off Show Deleted data on Bucket Details Page
5. Validate on Cloud Shell via gsutil command.

gsutil ls
gsutil du -h -a gs://$BUCKETNAME
gsutil ls gs://$BUCKETNAME/* | wc -l

At first upon uploading, not responding pop up appear. Tried to open new tab and try to check my bucket on console. Also I can see only 3 files upon checking the folder on the bucket. I give it some time for about 15 mins and tried to refresh still 3 Files on the folder. Tried to on and off the Show deleted data (Middle Right part of Bucket Details) and some of the files are showing. I Give it a little more time like 30 - 45 mins and it was completed.

==> standard input <==
3.02 KiB     gs://testupload/testimage/download (992).jpeg#1651637939860508
3.02 KiB     gs://testupload/testimage/download (993).jpeg#1651637579558398
3.02 KiB     gs://testupload/testimage/download (994).jpeg#1651637761564156
3.02 KiB     gs://testupload/testimage/download (995).jpeg#1651637080592200
3.02 KiB     gs://testupload/testimage/download (996).jpeg#1651637475578587
3.02 KiB     gs://testupload/testimage/download (997).jpeg#1651637634800927
3.02 KiB     gs://testupload/testimage/download (998).jpeg#1651637897219133
3.02 KiB     gs://testupload/testimage/download (999).jpeg#1651638427607588
3.02 KiB     gs://testupload/testimage/download.jpeg#1651636931991159
2.26 GiB     gs://testupload/testimage/

XXXXXXXXX@cloudshell:~ (XXXXXXX)$ gsutil ls gs://testupload/* | wc -l
4564

Source File image

Uploaded Files on Console