'Google cloud composer-airflow throws error: Broken DAG: File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py
I am trying to created a DAG in Cloud Composer. When importing, I'm getting the following error:
Broken DAG: [/home/airflow/gcs/dags/branch_name/persoflow/dags/test_dag.py] Traceback (most recent call last):
File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 628, in download_arrow_row_iterator
yield _row_iterator_page_to_arrow(page, column_names, arrow_types)
File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 601, in _row_iterator_page_to_arrow
arrays.append(pyarrow.array(page._columns[column_index], type=arrow_type))
AttributeError: 'NoneType' object has no attribute 'array'
Version used :
composer-1.17.9-airflow-2.1.4
To dig deeper I checked the python file pandas_helpers.py.
In the code we are using bigquery client to get the bigquery query result to dataframe like this
client = bigquery.Client(project=bq_project)
job_config = bigquery.QueryJobConfig()
client.query(query, job_config=job_config).to_dataframe(['list1'].values.tolist()[0]
Any help is much appreciated
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
Solution | Source |
---|