Some files missing when Ubuntu-based image is deployed to newer k8s and Azure, but not with Alpine, or not with onprem older k8s

Question

I work in a large enterprise that mostly deploys Alpine-based images running Java SpringBoot services, to k8s v1.13.12 on prem.

We are starting to move to Azure, running k8s v1.20.7, and as a separate initiative, changing the base image to Ubuntu.

We have a build system that runs on top of Jenkins, where I can simply specify a service to deploy, which branch, and what cluster to deploy to (onprem or Azure).

We have one service that we're trying the Ubuntu base image with. When I deploy this onprem, it works fine. When I deploy other similar services running the Alpine image to either Azure or onprem, they work fine. However, when I deploy the Ubuntu image to Azure, there is one directory that ends up being empty in the pod container. That directory contains the jar files containing the service code, so that is pretty fatal.

By itself, the Ubuntu base image isn't a problem. It runs onprem fine. By itself, the combined factor of Azure and the newer k8s isn't a problem. We are successfully running similar services there. However, the combination of the Ubuntu base image, running on Azure with the newer k8s, results in that directory being empty.

When I searched the web for this, I actually found another similar incident, although there was no reported solution in that case. Someone mentioned that the behavior is similar to an "emptydir" volume, but I see no indication that this is in play here.

The container runs a shell script at startup, and I've been adding debugging to this in the image so I can compare between onprem and Azure. I don't see any obvious clues from those comparisons. The directory is definitely empty in the bad case, and I don't see any other files or directories that are missing.

I don't expect a response here that solves my problem, but I could use some ideas for what else I can look at that might provide some clues.

Update:

I figured out what fixes the problem, but I don't understand it yet.

I simply changed the Dockerfile, adding a VOLUME statement for the bin directory in question. When the container starts up, the bin directory is not empty, and it works fine.

Remember that the original image worked fine with k8s 1.13.12, so this appears to be something that happened between that version and 1.20.7.

I always thought that k8s ignored the declared VOLUMEs in the Dockerfile. At least in this newer version, it is paying at least some attention to them.

I'd appreciate any known history or background on this.