'Google cloud composer-airflow throws error: Broken DAG: File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py

I am trying to created a DAG in Cloud Composer. When importing, I'm getting the following error:

Broken DAG: [/home/airflow/gcs/dags/branch_name/persoflow/dags/test_dag.py] Traceback (most recent call last):
  File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 628, in download_arrow_row_iterator
    yield _row_iterator_page_to_arrow(page, column_names, arrow_types)
  File "/opt/python3.8/lib/python3.8/site-packages/google/cloud/bigquery/_pandas_helpers.py", line 601, in _row_iterator_page_to_arrow
    arrays.append(pyarrow.array(page._columns[column_index], type=arrow_type))
AttributeError: 'NoneType' object has no attribute 'array'

Version used :

composer-1.17.9-airflow-2.1.4

To dig deeper I checked the python file pandas_helpers.py.

In the code we are using bigquery client to get the bigquery query result to dataframe like this

client = bigquery.Client(project=bq_project) 
job_config = bigquery.QueryJobConfig()
client.query(query, job_config=job_config).to_dataframe(['list1'].values.tolist()[0]

Any help is much appreciated



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source