Best practices in block based or flow based system programming [closed]

Question

I'm trying to implement a Labview-like system (in Java) where some blocks are capable of
- Sourcing
- Sinking
- Passing-through
the data.

Every block has a number of inputs (>= 0) and a number of outputs (>= 0) and is devoted to a function on inputting, outputting and transforming data. Data flow is represented as samples, taken from field measures coming from various devices (thermometers, wind-speed meters, irradiation sensors, power-meters and so on).

Every sample is related to an "epoch", that is an instant where all values are captured, starting from a trigger (source blocks should have a special trigger input), so a sample group coming from an epoch should be processed together with other samples from the same one.

A typical block-chain, for example, may be represented by a source (temp sensor), followed by an averager block (pass-through) and finally followed by a sink able to log data into a database (sink block).

I've implemented a push-like policy, where each block owns the "next-block" reference and calls a method on it passing the data to process. This policy works perfectly until it encounters a situation where a block (like a comparer block, with two float inputs and a boolean output), needs to output a comparation value based on the same sample epoch.

The problem is that I'm unable to find and apply a pattern that abstracts this problem. Maybe this is something similar to how Labview or Simulink blocks are working internally but I've still no success finding a "best-practice". Major problems with my "push-policy" is that, depending on chain sizes and depths, blocks with more than one input finish to work on samples belonging to different epochs (imagine a comparer block comparing two samples coming from two different measurement istants).

Thanks in advance for all advices, tips, best-practices and everything that will be able to solve my more-phylosophical-than-programming problem.

Answer 1

Fun question!

I would suggest separating the ideas of "the block" (unit of functionality represented in a box), the topology (the definition of how blocks are connected), and control (instantiates the blocks, arbitrates message passing and does flow control causing blocks to execute at the right times etc..). The objective is to eliminate coupling between blocks (blocks should have absolutely no knowledge of each other) and create an arbitrarily configurable system.

Here are my thoughts on the three major components:

Topology

Topology Definition: The definition of the topology typically occurs in a resource file that is not a code file (should be user configurable right). The resource file should contain information about what blocks, their inputs (outputs can usually be derived from knowledge of inputs) and define any properties a block might have. For example a resource JSON file might look like the following:

[{ uniqueId:"1",
   type:"Number",
   val:0.3
 },
 { uniqueId:"2",
   type:"Sine",
   stepSize:0.1,
   initialOutput=0
 },
 { uniqueId:"3",
   type:"Multiplier",
   inputs:["1","2"],
   initialOutput=0
 },
 { uniqueId:"4",
   type:"Plotter",
   inputs:["3"],
 }]

Control

The control mechanism starts by reading the topology definition (don't forget validation) and then uses this information to instantiate the block components and create the messaging infrastructure that ties the block components together using knowledge of the inputs each block has (treat each input as a separate message queue). In particular the controller needs to determine order of execution of the blocks in each time step. At each time step the control does several steps, first starting at the beginning of your block chain push outputs to the input queues that are "listening". Execute the next layer of blocks. Push outputs to next layer of blocks, etc.. (Or some other strategy like that.)

Block

The blocks basically only know how to take the inputs given and return a suitable output.

---

As an aside another analogous system you could study are circuit simulation programs (like Spice) used by electrical engineers. These typically consist circuit components (your blocks), a node list file (the topology definition), and a simulation runtime of some sort.

---

Anyways thats my idea..

Answer 2

This kind of problem can be solved by having synchronous inputs.

In a LabVIEW VI, the inputs are all defined when the VI starts. They therefore all have the same epoch. Differences in how long it takes to work through different paths then becomes moot.

In cases when several VIs work on a data set there is a VI that collects all the data from the source once per cycle. The code has three stages. Collect data. Process data. Output data. This means all the source data has the same epoch.

For control applications, this synchronisation is vital to make the control system math stable.