'Maximum memory size for an XCOM in Airflow
I was wondering if there is any memory size limit for an XCOM in airflow ?
Solution 1:[1]
After looking at the source code it looks there is none, the type is a large binary in SQLAlchemy. Code So according to the documentation is an unlengthed binary type for the target platform, such as BLOB on MySQL and BYTEA for PostgreSQL.
Solution 2:[2]
According to the source code check this source code link, maximum XCOM size is 48KB.
Solution 3:[3]
Airflow is NOT a processing framework. It is not Spark, neither Flink. Airflow is an orchestrator, and it the best orchestrator. There is no optimisations to process big data in Airflow neither a way to distribute it (maybe with one executor, but this is another topic). If you try to exchange big data between your tasks, you will end up with a memory overflow error! Oh, and do you know the xcom limit size in Airflow?
It depends on the database you use:
- SQLite: 2 GB
- Postgres: 1 GB
- MySQL: 64 KB
Yes, 64 Kilobytes for MySQL! Again, use XComs only for sharing small amount of data.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | nono |
| Solution 2 | Datageek |
| Solution 3 | Dheemanth Bhat |