How do I call scrapy from airflow dag?

Question

My scrapy project runs perfectly well with 'scrapy crawl spider_1' command. How to trigger it (or call the scrappy command) from airflow dag?

with DAG(<args>) as dag:
     scrapy_task = PythonOperator(
          task_id='scrapy',
          python_callable= ?)
     task_2 = ()
     task_3 = ()
   ....
scrapy_task >> [task_2, task_3, ...]

Answer 1

Run with BashOperator

• https://airflow.apache.org/docs/apache-airflow/stable/howto/operator/bash.html

with DAG(<args>) as dag:
     scrapy_task = BashOperator(
          task_id='scrapy',
          bash_command='scrapy crawl spider_1')

• If you're using virtualenv, you may use VirtualEnvOperator
  • or to use existing environment, you can use source activate venv && scrapy crawl spider_1

Run with PythonOperator

• From scrapy documentation: https://docs.scrapy.org/en/latest/topics/practices.html#run-scrapy-from-a-script

from scrapy.crawler import CrawlerProcess
from scrapy.utils.project import get_project_settings

process = CrawlerProcess(get_project_settings())
process.crawl('spider_1')
process.start() # the script will block here until the crawling is finished