starting celery worker from python script results in error - "click.exceptions.UsageError: No such command" using celery==5.1.2

Question

Directory structure:

Here is my cw_manage_integration/psa_integration/api_service/sync_config/init.py:

from celery import Celery
from kombu import Queue

from psa_integration.celery_config import QUEUE, USER, MAX_PRIORITIES_SUPPORT_AT_TIME

BROKER = "amqp://{0}:{1}@{2}/xyz".format("abc", "pqrst", "x.x.x.x)
APP = Celery(
"sync service",
broker=BROKER,
backend='rpc://',
include=["psa_integration.sync_service.alert_sync.alert",
"psa_integration.sync_service.tenant_sync.tenant",
"psa_integration.sync_service.alert_sync.update_status"]
)
APP.conf.task_queues = [
Queue(QUEUE, queue_arguments={'x-max-priority': MAX_PRIORITIES_SUPPORT_AT_TIME}),
]

The below is the cw_manage_integration/start_service.py:

"""Scrip to start Sync service via Celery."""
from psa_integration.utils.logger import *
from psa_integration import sync_service
from psa_integration.celery_config import CELERY_CONCURRENCY

APP = sync_service.APP

try:
    APP.start(["__init__.py", "worker", "-c", str(CELERY_CONCURRENCY)])
except Exception as scheduler_exception:
    logging.exception("Exception occurred while starting services. Exception = {}".format(scheduler_exception))

When I run the command python3 start_service.py using celery version celery==4.4.5, it just works fine by starting celery workers.

But when the same start_service.py is run using celery==5.1.2, it is throwing the below error:

>python3 start_service.py

MainProcess INFO 2021-07-07 16:27:42,725 all_logs
79 : started MainProcess INFO 2021-07-07 16:27:42,725 all_logs
80 : log file name: /home/sdodmane/PycharmProjects/cw_manage_integration1/cw_manage_integration/psa_integration/logs/worker_2021-07-07.log MainProcess INFO 2021-07-07 16:27:42,725 all_logs
81 : Level: 4 Traceback (most recent call last): File "/home/sdodmane/.local/lib/python3.8/site-packages/click_didyoumean/init.py", line 34, in resolve_command return super(DYMMixin, self).resolve_command(ctx, args) File "/usr/lib/python3/dist-packages/click/core.py", line 1188, in resolve_command ctx.fail('No such command "%s".' % original_cmd_name) File "/usr/lib/python3/dist-packages/click/core.py", line 496, in fail raise UsageError(message, self) click.exceptions.UsageError: No such command "init.py".

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "start_service.py", line 10, in APP.start(["init.py", "worker", "-c", str(CELERY_CONCURRENCY)]) File "/usr/local/lib/python3.8/dist-packages/celery/app/base.py", line 371, in start celery.main(args=argv, standalone_mode=False) File "/usr/lib/python3/dist-packages/click/core.py", line 717, in main rv = self.invoke(ctx) File "/usr/lib/python3/dist-packages/click/core.py", line 1132, in invoke cmd_name, cmd, args = self.resolve_command(ctx, args) File "/home/sdodmane/.local/lib/python3.8/site-packages/click_didyoumean/init.py", line 42, in resolve_command raise click.exceptions.UsageError(error_msg, error.ctx) click.exceptions.UsageError: No such command "init.py".

Not able to differentiate between celery==4.4.5 and celery==5.1.2 in this context. Please help me in solving this problem.

Answer 1

Currently, the start method is broken in the current(5.1.2) release. It has been fixed (https://github.com/celery/celery/pull/6825/files) but has not been released yet. Hopefully, the next release v5.1.3 will fix this issue.

Answer 2

I had a similar problem and was able to fix with the following change:

# celery==5.1.1
APP.start()

# celery==5.2.6
import sys
APP.start(argv=sys.argv[1:])

For you, that may mean removing the __init__.py in your list of args:

APP.start(["worker", "-c", str(CELERY_CONCURRENCY)])