Correct way to define an apache beam pipepline

Question

I am new to Beam and struggling to find many good guides and resources to learn best practices.

One thing I have noticed is there are two ways pipelines are defined:

with beam.Pipeline() as p:
# pipeline code in here

Or

p = beam.Pipeline()
# pipeline code in here
result = p.run()
result.wait_until_finish()

Are there specific situations in which each method is preferred?

Answer 1

From code snippets, I see the main difference is if you care about pipeline result or not. If you want to use PipelineResult to monitor pipeline status or or cancel your pipeline by your code, you can go to the second style.

Answer 2

I think functional wise they are equivalent since the __exit__ function for pipeline context manager is executing the same code. https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline.py#L426

Answer 3

As pointed out by Yichi Zhang, Pipeline.__exit__ set .result, so you can do:

with beam.Pipeline() as p:
  ...

result = p.result

The contextmanager version is cleaner as it can correctly cleanup when error are raised inside the contextmanager.