'How to update SDK version for dataflow job
I created a dataflow job using a template (Datastream to BigQuery). All is running fine but when I open the Dataflow job page, in the lateral job info pane, I am reminded that:
A newer version of the SDK family exists and updating is recommended
How do I go on doing that? Do I need to create a clone of the existing job or is there an easier way?
regards, Matteo
Solution 1:[1]
This is a Google-provided template, which gets updated to the latest Beam version by Google as soon as a new Beam version is available, both in the source code and the version that you can run from the Cloud Console UI.
Though there is typically a 1-2 week delay between the latest Beam version is available in Dataflow and the templates get updated. In this case, 2022-05-04-00_RC00 release of templates included the Beam 2.38 upgrade (see the release notes), and the rollout making it available in the Cloud Console UI finished yesterday, May 11th.
So if you running the job from the UI, you can simply restart it (as of today) and it will pick up the new Beam version. If you doing something like building the template from the source code and running it as a custom template, you'll have to rebuild it using from the 2022-05-04-00_RC00 or later version.
Solution 2:[2]
Whoever created the template would need to do that by re-creating the template against a newer version of the SDK.
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Nick |
| Solution 2 | robertwb |