'Listing packages in MWAA with a broken scheduler
So, I'm currently working with an Airflow installation via MWAA. I'm having this issue with a broken dependency, specifically:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
apache-airflow-providers-amazon 1.4.0 requires watchtower~=0.7.3, but you have watchtower 2.0.1 which is incompatible.
I want to get a list of the currently installed packages to do some debugging, but the broken install also seems to break the scheduler. Also, the worker logs suggest that it just keeps trying and failing to install what is listed in requirements.txt.
Besides just removing the package with a broken dependency, is there any way of listing the currently installed packages outside of just creating a DAG to do it? I know the DAG shouldn't be difficult to write, It's just that it will never be executed anyways since the scheduler is preoccupied
here is the current requirements.txt:
--trusted-host <A private repository>
--index-url <A private repository>
gnupg
snowflake-connector-python
snowflake-sqlalchemy
apache-airflow-providers-snowflake
EDIT: I tried adding "apache-airflow-providers-amazon" from our internal repo and got some strange results. No errors reported in the log, except for a failure to connect to the standard pypi repo (this error also occurs with working versions of requirements.txt however). However, the scheduler still appears to be broken.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|