Trouble transitioning to multiprocessing python code to cython

Question

I'm running a program that takes real time data on a Raspberry pi 4. Because the program is quite demanding, the different functionalities are separated into the different cores with Multiprocessing using Pool, each process being an MQTT client.

I have been trying to transition the code to Cython but I have been running into trouble with multiprocessing. From posts I have seen on the internet, prange is used. But when trying to pass mqtt clients to prange I get errors concerning with nogil. Is there a way to use Pool with Cython or use multiple processes that require gil to run in parallel with Cython?

Sorry that it might sound like a newbie question, also I apologize but I can't show the code.

Thanks in advance for your help.

Edit: Currently I'm stuck on the map section of Pool. Here is a code segment of it:

def clie1(num):
  global id1
  print("Clie1")
  mqttBroker="mqtt.eclipseprojects.io"
  id1 = os.getpid()
  client.on_connect = on_connect
  client.on_message = on_message
  client.connect(mqttBroker, 1883, 60)
  #client.connect("127.0.0.1", 1883, 60)
  client.loop_forever()


def clie2(num):
  global id2
  print("Clie2")
  mqttBroker="mqtt.eclipseprojects.io"
  id2 = os.getpid()
  client2.on_connect = on_connect2
  client2.on_message = on_message2
  client2.connect(mqttBroker, 1883, 60)
  #client2.connect("127.0.0.1", 1883, 60)
  client2.loop_forever()

def smap(f):
  return f()

client = mqtt.Client()
client2 = mqtt.Client()


def main():
  fc1 = functools.partial(clie1, 1)
  fc2 = functools.partial(clie2, 2)
  print("Pool")
  with Pool() as pool:
      res = pool.map(smap, [fc1,fc2])
        

main() 

Terminal Result:

My setup file is very simple and maybe it requires something additional to work:

from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules=cythonize("MyFile.pyx"),
)

Answer 1

So we've established the issue is that: to run it is a .py file you need

if __name__=="__main__":
    main()

However a Cython file can't be run "as a script" so __name__ is never "__main__". (In principle this works with embedded Cython but I don't think that'll help you).

The solution is to write a tiny Python script that imports your Cython module and calls its main() function:

# Do NOT compile this script with Cython!
import my_cython_mod
if __name__ == "__main__":
    my_cython_mod.main()

When you do that a cut-down version of your example runs fine for me.