'Conda: How to prevent mkl to openblas switch
Conda wants to downgrade my blas, lapack etc. packages from an mkl to an openblas version. I understand that conda juggling with mkl versus openblas seems not an uncommon issue. Yet, I have not found a solution to do the job for me. I have these packages installed
blas 2.113 mkl conda-forge
blas-devel 3.9.0 13_linux64_mkl conda-forge
libblas 3.9.0 13_linux64_mkl conda-forge
libcblas 3.9.0 13_linux64_mkl conda-forge
liblapack 3.9.0 13_linux64_mkl conda-forge
liblapacke 3.9.0 13_linux64_mkl conda-forge
mkl 2022.0.1 h06a4308_117
mkl-devel 2022.0.1 h66538d2_117
mkl-include 2022.0.1 h06a4308_117
mkl-service 2.4.0 py39h404a4ab_0 conda-forge
mkl_fft 1.3.1 py39h6964271_2 conda-forge
mkl_random 1.2.2 py39h8b66066_1 conda-forge
and I have a .condarc (on linux) containing
channels:
- conda-forge
- defaults
dependencies:
- python>=3.6
- numpy>=1.13
- scipy>=0.18
- cython>=0.29
- mkl
- mkl-devel
- libblas=*=*mkl
- bottleneck
- pip
- setuptools>=30.3.0
- h5py
- pyyaml
- pytest
ssl_verify: true
auto_activate_base: false
Moreover in the conda-meta directories I have a pinned file, containing the line libblas=*=*mkl. Yet, upon conda update --all this is suggested:
The following packages will be DOWNGRADED:
... other pkgs ...
libblas 3.9.0-13_linux64_mkl --> 3.9.0-13_linux64_openblas
libcblas 3.9.0-13_linux64_mkl --> 3.9.0-13_linux64_openblas
liblapack 3.9.0-13_linux64_mkl --> 3.9.0-13_linux64_openblas
Why, despite of the the .condarc and pinned files, am I getting this switch from mkl to openblas, and what else can I do to prevent it?
Solution 1:[1]
I'm posting a quirky workaround for my own OP. This finally worked for me.
First, I could not convince conda to respect the channel ordering in .condarc nor the content in the pinned files before running the update by any other suggestion, including those found outside of stackoverflow .
Second, I stored a conda list | grep mkl, a conda list | grep intel, and a conda list | grep open away for later reference. Then I "gave in" and let the "upgrade" happen, running conda update --all. No need to mention that after that, my environment indeed showed the unwanted replacement of all mkl-type libraries with openblas stuff.
Third, and within the openblas-infested environment I "re-installed" mkl
conda install blas=*=*mkl
conda install libblas=*=*mkl
conda update numpy
conda update scipy
conda install intel-openmp # the "update" had also removed this ...
Also make sure that no openblas-stuff remains by doing a conda remove on whatever related package. (I'm not claiming that really all of the above commands are necessary to reach the original state of the environment regarding mkl. But that's what I did.)
Fourth, comparing with the reference notes from the preceding second step I checked that at this point my environment claimed to be back to "all-mkl". Moreover, using this extremely helpful site http://markus-beuckelmann.de/blog/boosting-numpy-blas.html I also checked, that this was indeed true regarding typical mkl timings to be expected.
On the side, there is a really weird and confusing issue, which may not be related to the OP but which I stumbled across in this context: On the WWW one finds many many many quotes stating, that for numpy or scipy actually "using" mkl, one has to have this kind of output
In []: numpy.show_config()
blas_mkl_info:
libraries = ['mkl_rt', 'pthread']
library_dirs = ['/home/abcd/efgh..../lib']
from numpy/scipy.show_confi(). This seems not true in general. In fact when one gets
In []: numpy.show_config()
blas_info:
libraries = ['cblas', 'blas', 'cblas', 'blas']
library_dirs = ['/home/abcd/efgh..../lib']
this is no cause for panic as long as in /home/abcd/efgh..../lib one finds everything linked as
liblapacke.so -> libmkl_rt.so.2*
libblas.so -> libmkl_rt.so.2*
...a.s.o.
which I do.
(conda is just soo painful. sigh)
Solution 2:[2]
The package management can be a mess whenever the Intel MKL package has been updated, but the numpy package has not. I may end up with Intel MKL being installed, but numpy is using the openblas. To get this working in my setup, I often do this when creating a new environment:
conda create -y --name test python=3 numpy mkl cmake blas=*=*mkl
so that the MKL is installed at the same time as numpy, and numpy will use the MKL. This often results in an older version of MKL, since numpy was linked against an older version of the MKL.
Without the blas=*=*mkl, I will often end up with an openblas based numpy installed with the newest version of the Intel MKL not being used by anything.
conda create -n test33 numpy mkl blas=*=*mkl
.
.
.
The following NEW packages will be INSTALLED:
blas pkgs/main/osx-64::blas-1.0-mkl
.
numpy pkgs/main/osx-64::numpy-1.21.5-py39h2e5f0a9_1
.
mkl pkgs/main/osx-64::mkl-2021.4.0-hecd8cb5_637
.
# results in:
blas_opt_info:
libraries = ['mkl_rt', 'pthread']
conda create -n test33 numpy mkl
.
.
.
The following NEW packages will be INSTALLED:
blas pkgs/main/osx-64::blas-1.0-openblas
.
numpy pkgs/main/osx-64::numpy-1.21.5-py39h9c3cb84_1
.
mkl pkgs/main/osx-64::mkl-2022.0.0-hecd8cb5_105
.
# results in:
blas_opt_info:
libraries = ['openblas', 'openblas']
Unfortunately, numpy does not appear to show the blas library it is linked against in its version id. Oftentimes, you will end up with both openblas and mkl in your environment.
Solution 3:[3]
See the note section at the end of https://conda-forge.org/docs/maintainer/knowledge_base.html#switching-blas-implementation
If you want to commit to a specific blas implementation, you can prevent conda from switching back by pinning the blas implementation in your environment. To commit to mkl, add
blas=*=mklto<conda-root>/envs/<env-name>/conda-meta/pinned, as described in the conda-docs.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Mark |
| Solution 2 | |
| Solution 3 | isuruf |