Is there an example of PyFlink SQL unit testing in a self-contained repo?

Question

Is there an example of a self-contained repository showing how to perform SQL unit testing of PyFlink (specifically 1.13.x if possible)?

There is a related SO question here, where it is suggested to use some of the tests from PyFlink itself. The issue I'm running into is that the PyFlink repo assumes that a bunch of things are on the Java classpath and that some Python utility classes are available (they're not distributed via PyPi apache-flink).

I have done the following:

1. Copied test_case_utils.py and source_sink_utils.py from PyFlink into my project.
2. Copy an example unit test (this one as suggested by the related SO question.

When I try to run the test, I get an error because the test case cannot determine what version of Avro jars to download (download_apache_avro() fails, because this code tries to evaluate the value of avro.version by running mvn help:evaluate -Dexpression=avro.version)

I then added a dummy pom.xml defining a Maven property of avro.version (with a value of 1.10.0) and my unit test case is loaded.

I now get a new error and my test is skipped:

'flink-table-planner*-tests.jar' is not available. Will skip the related tests.

I don't know how to fix this. I've tried adding flink-table-planner and flink-table-planner-blink dependencies with <type>test-jar</type> to my dummy pom.xml, but it still fails.

This is starting to feel like a real pain to do something that should be trivial: basic TDD of a PyFlink project. Is there a real-world example of a Python project that shows how to set up a testing environment for unit testing SQL with PyFlink?

Answer 1

You can refer to https://github.com/dianfu/pyflink-faq/tree/main/testing which gives an example on how to write unit tests in an external project.