How to get an abstract syntax tree in python?

Question

I have a grammar defined. For sake of example expr = a | expr *expr.
I want to get an ast.
Is it possible to do this in python?
I have looked at ast module in python, but I believe it works only for python code, and not a generic grammar.

Answer 1

The ast module is specifically designed for the Python AST.

And, in fact, any good ast library will be specifically designed around the AST design for the target language, which in turn will be based on the requirements of the language processor (i.e., the compiler or whatever).

That's different from a concrete parse tree, which is what is usually built by parser generators which feature tree building. You can use a parse tree instead of an AST, but unless you're writing a source-to-source processor (like a syntax highlighter or code formatter), it's not usually very convenient, since it contains lots of irrelevant details which complicate the tree walks. Of course, you can build an AST from the parse tree if you find that more convenient than building it as you go. Sometimes it is.

What Python's ast module does give you is a reasonably well-thought-out model for how to design an AST, so you might want to play around with it a bit. Think about what your specific needs are, and see how similar ones could be implemented (or not) in the Python framework. That might help you design your AST.

Even so, I can tell you from experience that you're unlikely to get it right the first time round. I'm sure you'll build something which can be used to meet your immediate needs, but you'll likely also run into requirements which require awkward workarounds. The next one you build will be a lot better.