'How to avoid file overwrites when multiple ECS tasks mount the same EFS
This is a typical auto-scale configuration for AWS ECS (Fargate):
- A docker image with our JavaEE app deployed on an application server, producing a couple of log files (rotated every X Mb of size).
- The image is deployed on ECS (Fargate) with an auto-scale configuration. We have a minimum of two tasks running at all times.
- Tasks are logging to stdout (using
tail
) so all logs end up in Cloudwatch and then we process the streams to reach their final destination (ELK)
We are now trying to bypass Cloudwatch (and associated costs) and instead mount an EFS to the tasks so all files end up there. We can then process the files.
This is where the problem comes: All tasks mount the same EFS at some mount point and all tasks use the same path to place log files. So, if we proceed with this solution, we will end up with multiple tasks (2 plus auto-scale) writing over the same mounted files (i.e. to /mnt/efs/<task name>/logs/server.log
).
What are our options to differentiate each task (using the same task definition!) so it writes to a different file or different path? For example, are there any environmental variables set for each ECS task (task ID?) I could use? Or the only option would be to generate a GUID and then log to /mnt/efs/logs/<task name>/<GUID>/server.log
?
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|