Loading TensorFlow model to manipulate an audio stream with C++

Question

I want to load a machine learning model created with TensorFlow into my C++ Audio Application made with JUCE6. In order to use TensorFlow inside C++, I am using the TensorFlow wrapper CppFlow. I have the problem, that I don't know how to load the model for use in an audio stream.

Tensorflow models are loaded with cppFlow like this:

cppflow::model model("path_to_model");

Then I can use the model inside my audio processing block like this.

auto output = model(input);

Here is an example: https://github.com/serizba/cppflow/blob/master/examples/load_model/main.cpp

If I implement it like this inside my AudioProcessingBlock, the Application calls abort() without an error code and says: Debug Error! Probably due to a CPU overflow, because the model is loaded with every sample -> 44k times per second. If I implement the model loading inside my prepareToPlay method (called once), where I would place it anyway, the application runs just fine, but I cannot access the model inside my AudioProcessingBlock. Therefore I am not able to call

auto output = model(input);

.

the cppflow::model::model is an inline function:

inline model::model(const std::string &filename){
this-> graph = {TF_NewGraph(), TF_DeleteGraph()};
...

The complete implementation can be found here, starting line 46: https://github.com/serizba/cppflow/blob/master/include/cppflow/model.h

How could I save the model inside a private variable inside my class? So I can instantiate the model once, and use it inside the AudioProcessingBlock while my application is running.

Answer 1

As far as I understand you want to use the model multiple times and loading the model each time causes overload. Either you load the model in global scope (outside of any class or main). The other option is to create a unique pointer which will be loaded once when first called.

Declare unique pointer:

std::unique_ptr<cppflow::model> model;

When you first use the model or in a constructor or init function call:

model = std::make_unique<cppflow::model>(const std::string &filename);

Then use the model to make predictions:

auto prediction = (*model)(input)