How to speed up building .net solution in docker

Question

My current setup:

App.sln
src\
----BaseProject\
-------------Dockerfile
-------------BaseProject.csproj
----ProjectA\
-------------Dockerfile
-------------ProjectA.csproj (references BaseProject)
----ProjectB\
-------------Dockerfile
-------------ProjectB.csproj (references BaseProject)
----ProjectC\
-------------Dockerfile
-------------ProjectC.csproj (references BaseProject)

Right now each project has its own dockerfile, and the containers are built in parallel, in the cloud. I wonder if it is possible to reuse build output to speed up build process.

I'm building whole solution using docker compose build --parallel, and publishing each project as separate container:

FROM sdk_image as build
RUN dotnet restore
RUN dotnet build --no-restore
RUN dotnet publish --no-build --no-restore  -o output

FROM runtime_image as runtime
COPY build/ouput .
RUN dotnet run myapp.dll

I've done some attempts to cache whole build in single layer but apparently output assemblies are so big that copying them takes more time than actual compilation.

I wonder if it's possible to speedup such build, maybe using docker volumes for nuget or build or similar mechanisms, or maybe deleting unnecessary files from build output?