Many tasks at once in Celery?

Question

If we use celery beat and run about 1000 tasks by same crontab schedule, will tasks run one by one or some tasks will not run (cause of out of time)? redis as MQ

Answer 1

celery beats only trigger those 1000 tasks (by the crontab schedule), not run them.

If you want to run 1000 tasks in parallel, you should have enough celery workers available to run those tasks.

Keep in mind that workers can run concurrent, so if --concurrency=4 in each worker, only 250 workers to run all tasks at once.

If you have only 125 workers, your 1,000 tasks will run in two bulks of 500 each (assuming all tasks take the same time to run).