Having multiple sockets for same Context() and same port in ZMQ

Question

My current system takes input stream from cameras, each camera in a separate instance, and apply Computer Vision models on each camera (Object Detection, Object Tracking and Personnel Recognition), and then pass the results to a sink/master process that performs the rest of functionality over those results and I'm using ZMQ as an inter-process communication.

What I implemented now is that each worker connects to a different port, and then the sink subscribes to these ports independently, but this solution is not scalable, as we might have 3 or 4 cameras/worker, and I felt that it won't be efficient to keep opening ports like that.

Multiple Ports Implementation

That's when I tried to implement Multi-Pub/Single-Sub module, where all workers will connect to one port and the sink will subscribe to that port only.

Single Port Implementation

The problem I faced is that I no longer can distinguish between different cameras since I'm receiving different footages in the same port which causes a problem in streaming them later, that's why I'm thinking about the possibility of having multiple sockets for each context, while each socket subscribes to a different IP, is that possible?

Note: I've seen this answer but it has different ports for different sockets which does not really serve my case.

Answer 1

Q : " ... I no longer can distinguish between different cameras ... "

A :
Yet, there are ZeroMQ tools to do so - check details about :

• .setsockopt( zmq.METADATA, "X-key:value" )
• .setsockopt( zmq.ROUTING_ID, Id )

As you see, PUB/SUB-archetype is the worst one to be used here ( you pay all the costs of TOPIC-filter based subscription-management, yet receive nothing for doing that ).

Using better matching archetypes is the way to go.

Given not performance details were posted, the capacity may soon get over-saturated, so may use more specific steps to flatten the workload and protect smooth-flow of the service :

• .setsockopt( zmq.TOS, aTransportPath_TOS )
• .setsockopt( zmq.MAXMSGSIZE, aBLOB_limit_to_save_RAM )

Given a streaming could block on many "old"-frames not having got through the e2e-pipeline in due time, it might make sense to also set this :

• .setsockopt( zmq.CONFLATE, 1 )

As you can see, there are many smart details in the configuration space of the ZeroMQ, plus once scaling is to grow larger and larger, your design shall also fine-tune the Context()-engine performance once instantiating :

• .Context( aNumOfContextIOthreads2use )