Spring Cloud Data Flow - Task Properties

Question

I'm using SCDF and i was wondering if there was any way to configure default properties for one application? I got a task application registered in SCDF and this application gets some JDBC properties to access business database :

app.foo.export.datasource.url=jdbc:db2://blablabla
app.foo.export.datasource.username=testuser
app.foo.export.datasource.password=**************
app.foo.export.datasource.driverClassName=com.ibm.db2.jcc.DB2Driver

Do i really need to put this prop in a property file like this : (it's bit weird to define them during the launch)

task launch fooTask --propertiesFile aaa.properties

Also, we cannot use the rest API, credentials would appear in the url.

Or is there another way/place to define default business props for an application ? These props will be only used by this task.

The purpose is to have one place where OPS team can configure url and credentials without playing with the launch command.

Thank you.

Answer 1

Yeah, SCDF feels a bit weird in the configuration area.

As you wrote, you can register an application and create tasks, but all the configuration is passed at the first launch of the task. Speaking other way round, you can't fully install/configure a task without running it.

As soon as a task has run once, you can relaunch it without any configuration and it uses the configuration from before. The whole config is saved in the SCDF database.

However, if you try to overwrite an existing configuration property with a new value, SCDF seems to ignore the new value and continue to use the old one. No idea if this is intended by design or a bug or if we are doing something wrong.

Because we run SCDF tasks on Kubernetes and we are used to configure all infrastructure in YAML files, the best option we found was to write our own Operator for SCDF.

This operator works against the REST interface of SCDF and also compensates the weird configuration issues mentioned above. For example the overwrite issue is solved by first deleting the configuration and recreate it with the new values.

With this operator we have reached what you are looking for: all our SCDF configuration is in a git repository and all changes are done through merge requests. Thanks to CI/CD, on the next launch, the new configuration is used.

However, a Kubernetes operator should be part of the product. Without it, SCDF on Kubernetes feels quite "alien".