Mypy Function "lxml.etree.ElementTree" is not valid as a type but why?

Question

I am using Mypy for this function

import funcy as fu
from lxml import etree
from lxml.etree import Element, ElementTree

def find_nodes(tree: ElementTree, paths: Iterable[str]) -> Iterable[Element]:
    """Yield all non-zero nodes for the given paths in the tree."""
    return fu.remove(fu.isnone, map(tree.find, paths))

checking this file with mypy (0.942) and lxml-stubs gives me this strange error:

xxx\layout\treeutils.py:9: error: Function "lxml.etree.ElementTree" is not valid as a type
xxx\layout\treeutils.py:9: note: Perhaps you need "Callable[...]" or a callback protocol?

How can I fix that?

Answer 1

In lxml-stubs, ElementTree is indeed declared as a function. It returns an object of type _ElementTree, which is declared as a class. Something similar is happening for Element and _Element.

Looking into lxml itself, it turns out that this is right: _ElementTree is defined as a Cython class, and ElementTree is defined as a function. I don't know enough about Cython to say why it's implemented this way.

So this works:

from lxml.etree import Element, ElementTree, _Element, _ElementTree

def find_nodes(tree: _ElementTree, paths: Iterable[str]) -> Iterable[_Element]:

I wouldn't worry too much about the underscore here. A single underscore is merely a convention to indicate protected types, and doesn't have a special meaning to the Python interpreter. If lxml chooses to expose such types as return values in its public API, chances are they're not going anywhere in future versions of lxml. Your choice is to use these types and enjoy some type safety, or resort to Any and have no type safety at all.