Testing a Generator in Python

Question

I programmed a very simple generator in Python 3.4:

class Gen:
  def __init__(self, xml_lines, attribs):
    #...

  def _get_values(self, xml_line):
    # get only specific "attribs" from a line in a "xml"

  def values(self):
    for line in self.xml_lines:
      yield self._get_values(line)

The code works when I use a for loop to consume the generated values:

reader = Gen(...):
for v in reader.values():
   print(v)

But I'm now trying to create a unit test, and for that, I'd need to get each value at a time (outside a loop). I'm trying something like this:

import unittest
#...
reader = Gen(...):
v = reader.values()
self.assertIsNone(v)

When I try that, I always get a

AssertionError: <generator object next_values at 0x7fe2a5f9e3f0> is not None

So, when I call the values myself, it return something (is it a pointer?) instead of the generated value.

As I'm considering I'm using a basic generator pattern, my question is a little broader then my own code: what's the proper way to test a Python Generator?

Answer 1

A generator is meant to be iterated through. You can use the next() function to get the next value of the generator. Be aware that if your generator has exhausted it's values, it will raise a StopIteration exception.

reader = Gen(...):
values = reader.values()
v = next(values)
self.assertIsNone(v)

Answer 2

I needed to unit-test a generator myself, so I created this helper function:

def generator_tester(generator_iterator_to_test, expected_values):   
    range_index = 0
    for actual in generator_iterator_to_test:
        assert range_index + 1 <= len(expected_values), 'Too many values returned from range'
        assert expected_values[range_index] == actual
        range_index += 1

    assert range_index == len(expected_values), 'Too few values returned from range'

Here's an example of using it:

generator_tester(
    Gen(...),
    [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
)

Answer 3

I needed to check that the value produced by the return in the generator was valid, and I needed to know how the yielded values differed from the expected ones when the lenght was different. I ended up using a pretty big helper function, which reads up to 6 values that are produced but not expected, or the other way around, and displays them. Here it is:

from unittest import TestCase


class MISSING:
    pass


    def _generator_tester(
        self,
        iterator,
        expected_value_list,
        expected_returned_value=MISSING,
    ):
        """
        Helper method to test a generator.

        Args:
            iterator (Iterator):
                generator iterator to test
            expected_value_list (list):
                list of the values the generator should produce
            expected_returned_value (any):
                the final values returned by the generator
        """
        expected_value_iterator = iter(expected_value_list)

        try:
            for expected in expected_value_iterator:
                actual = next(iterator)
                self.assertEqual(actual, expected)
        except StopIteration:
            # Generator iterator is too short.
            #
            # len(iterator) < len(expected_value_iterator)
            #
            # Let's see what values were expected
            remaining_expected_values = [expected]
            for value in iterator:
                remaining_expected_values.append(value)
                if len(remaining_expected_values) == 8:
                    remaining_expected_values[-1] = Ellipsis
                    break
            self.fail(f"Too few values. Expected: {remaining_expected_values}")

        try:
            remaining_values = [next(iterator)]
        except StopIteration as stop_iteration:
            # Generator iterator is the right length
            #
            # len(iterator) == len(expected_value_iterator)
            #
            # Let's check the return value of the iterator
            if expected_returned_value != MISSING:
                self.assertEqual(stop_iteration.value, expected_returned_value)
        else:
            # Generator iterator is too long
            #
            # len(iterator) > len(expected_value_iterator)
            #
            # Let's see what are the extra values of the generator
            for value in iterator:
                remaining_values.append(value)
                if len(remaining_values) == 8:
                    remaining_values[-1] = Ellipsis
                    break
            self.fail(f"Too many values: {remaining_values}")

Answer 4

Given that you can only test a finite number of outputs from a generator, you may just turn the iterator to a list by:

self.assertEqual(list(your_generator_function_call(your_inputs)), list_of_exepcted_outputs)