Implementing threading with flask with application context

Question

I am trying to implement multiprocessing or threading in my flask __init__.py to implement a sub-process that will handle some back-end work for me. However, I can't seem to pass the application context to the sub-process, and exisitng libraries are focused on requests - I need this process to start running in parallel when flask is run, no later.

Here is my __init__.py code (excluding setup for irrelevant pages, I only use one implementation at a time ofc):

import os
from multiprocessing import Process
from threading import Thread
from flask_executor import Executor
from flask import Flask
from . import backend

def create_app():
    app = Flask(__name__, instance_relative_config=True)

    # Multiprocessing Implementation:
    p = Process(target=backend.start)
    p.start()

    # Threading Implementation:
    thread = Thread(target=backend.start)
    thread.daemon = True
    thread.start()

    # Flask Executor Implementation:
    executor = Executor(app)
    executor.submit(backend.start)

    return app

And here is my backend.py, called by the subprocess:

from datetime import datetime
from flask import g, request, session
from flaskr.db import log

def start(app=None):
    print("\nBackend Started\n")
    log("INFO","Backend Started")
    while True:
        pass

The backend code calls a logging function which works when called from a request inside my normal flask process.

My multiprocessing and threading implementation do not work, as I cannot pass the application context to the sub-process. Process(target=backend.start, args=app) or Thread(target=backend.start, args=app) gives me an error, TypeError: 'Flask' object is not iterable. I cannot add @with_appcontext flags to the start function, as it is not a request.

My Flask Executor passes the application context to the sub-process, but it cannot succeed either, as it is not called from a request:

Traceback (most recent call last):
  File "c:\...\python\python39\lib\runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\...\python\python39\lib\runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "C:\...\Python\Python39\Scripts\flask.exe\__main__.py", line 7, in <module>
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 988, in main
    cli.main()
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 579, in main
    return super().main(*args, **kwargs)
  File "c:\...\python\python39\lib\site-packages\click\core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "c:\...\python\python39\lib\site-packages\click\core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "c:\...\python\python39\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "c:\...\python\python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "c:\...\python\python39\lib\site-packages\click\decorators.py", line 84, in new_func
    return ctx.invoke(f, obj, *args, **kwargs)
  File "c:\...\python\python39\lib\site-packages\click\core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 850, in run_command
    app = DispatchingApp(info.load_app, use_eager_loading=eager_loading)
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 299, in __init__
    self._load_unlocked()
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 333, in _load_unlocked
    self._app = rv = self.loader()
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 389, in load_app
    app = locate_app(import_name, name)
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 251, in locate_app
    return find_best_app(module)
  File "c:\...\python\python39\lib\site-packages\flask\cli.py", line 77, in find_best_app
    app = app_factory()
  File "C:\...\flaskr\__init__.py", line 51, in create_app
    executor.submit(backend.start)
  File "c:\...\python\python39\lib\site-packages\flask_executor\executor.py", line 162, in submit
    fn = self._prepare_fn(fn)
  File "c:\...\python\python39\lib\site-packages\flask_executor\executor.py", line 122, in _prepare_fn
    fn = copy_current_request_context(fn)
  File "c:\...\python\python39\lib\site-packages\flask\ctx.py", line 172, in copy_current_request_context
    raise RuntimeError(
RuntimeError: This decorator can only be used when a request context is active, such as within a view function.

How can I implement this properly?

Answer 1

I found a solution to this issue. I added this code to my backend.py:

from flask import Blueprint
from threading import Thread

bp = Blueprint('backend', __name__)

def backend(app):
    thread = Thread(target=start, args=(app,))
    thread.daemon = True
    thread.start()

I then added backend.backend(app) to my create_app() function in __init__.py, right before the end of my function. This will call my backend() function from backend.py and pass the application context, and this function starts my sub-process.