'Databricks delta live tables stuck when ingest file from S3
I'm new to databricks and just created a delta live tables to ingest 60 millions json file from S3. However the input rate (the number of files that it read from S3) is stuck at around 8 records/s, which is very low IMO. I have increased the number of worker/core in my delta live tables but the input rate stays the same.
Is there any config that I have to add to increase the input rate for my pipeline?
amazon-s3Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|