Tensorflow on Docker Engine Error Code 132

Question

I am using Docker and Docker-Compose on Ubuntu 20. The application I am deploying on container is using Tensorflow. Docker-Compose build is able to be executed but not the Docker-Compose up because is failing with Error Code 132. This error can be translated to this: "Illegal instruction (core dumped)". This error reveals some missing cpu instructions that tensorflow is using.

Searching for several hours found several articles over the internet and I have tried all of them, here are some more details.

1. Host: Windows 10 Pro 64-bit i7 1185G7 (Clean Install)
2. Guest: VBox v6.1.30 / Ubuntu 20 64-bit (Clean Install)
3. Tensorflow >= 2.2.0
4. Host supports AVX and AVX2 instructions

mmx sse sse2 sse3 ssse3 sse4.1 sse4.2 avx avx2 avx-512 bmi2 abm fma adx dep smep smap em64t eist tm1 tm2 htt turbo sstaes-ni rdrand rdseed sha

5. Guest does not show AVX and AVX2 instructions

pu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid pni ssse3 cx16 pcid sse4_1 sse4_2 lahf_lm invpcid_single ibrs_enhanced fsgsbase invpcid md_clear flush_l1d arch_capabilities

6. Guest is able to import and execute Tensorflow on Spyder

Tensorflow module imported

7. Guest is not able to start containers with Tensorflow using Docker Engine

Container exited with code 132

Tried already the following things using commandline and powershell without any sucess:

• Disable Hyper-V on Host

Disable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All

DISM /Online /Disable-Feature /All /FeatureName:Microsoft-Hyper-V

• Enable Hyper-V on Host

Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V -All

DISM /Online /Enable-Feature /All /FeatureName:Microsoft-Hyper-V

• Enable AVX and AVX2 instructions on Guest using VBOXManage

VBoxManage setextradata "Ubuntu 20 Tls" VBoxInternal/CPUM/IsaExts/AVX 1

VBoxManage setextradata "Ubuntu 20 Tls" VBoxInternal/CPUM/IsaExts/AVX2 1

• Disable AVX and AVX2 instructions on Guest using VBOXManage

VBoxManage setextradata "Ubuntu 20 Tls" VBoxInternal/CPUM/IsaExts/AVX 0

VBoxManage setextradata "Ubuntu 20 Tls" VBoxInternal/CPUM/IsaExts/AVX2 0

Here is the dockerfile

FROM python:3.8

RUN pip install fastapi==0.70.0
RUN pip install motor==2.5.1
RUN pip install uvicorn==0.11.8
RUN pip install python-multipart==0.0.5
RUN pip install python-dotenv==0.14.0
RUN pip install gunicorn==20.0.4
RUN pip install databases==0.3.2
RUN pip install pandas==1.1.4
RUN pip install numpy==1.19.2
RUN pip install tensorflow==2.7.0
RUN pip install sklearn==0.0
RUN pip install Pillow==8.2.0
RUN pip install pydicom==2.1.2

COPY ./api /app

WORKDIR /app

CMD ["uvicorn", "api:app", "--host", "0.0.0.0", "--port", "80"]

================ UPDATE ================

On the same host, without any change using the Hyper-V Manager and attaching the same vhd disk of Guest, the instructions are available and docker-compose up is working. In summary, VBox Guest VM is not able to run tensorflow inside the docker-engine while HyperV Guest VM is able to run.

Any ideas?

Answer 1

After some days while using different things here are my findings:

Seems that installing VirtualBox on a machine which has HyperV already maybe need additional configuration to make a passthrough of AVX and AVX2 instructions.

In addition, deploying the containers in Synology NAS 920+, I am experiencing the same problem with tensorflow. In this case maybe is normal as this model uses Intel Celeron J4125 which does not support AVX and AVX2 instructions.

Finally older tensorflow versions (<=1.6) does not require AVX and AVX2 instructions and more recent versions of tensorflow (>=2.7.0) are displaying the issue more nicely saying that your CPU does not have the instructions instead of the generic 132 exit code.

In any case there are unofficial tensorflow builds which are compiled without AVX and AVX2 instructions. Just select your proper wheel based on your python version. I confirmed already while now i am running light tensorflow processes on my Synology NAS!

I found also a repository with ready tensorflow compiled builds: https://tf.novaal.de/