Processing arguments for subprocesses using argparse: "Expected one argument"

Question

First of all, I am sorry about the title that does not give a perfect idea of what my problem is, I kinda struggled with it, so I am going to illustrate it more clearly with a reproducible example:

I have the following code:

example.py:

import argparse


def parse_args():
    """Command line argument parser

    Returns:
        The extra arguments
    """

    parser = argparse.ArgumentParser(description="Arguments for test runner")
    parser.add_argument('-e', '--extra_args', type=str, default="", help="Extra arguments. Enter as a single line encapsulated by quotes. Example: -e \"--repeat=0 otherstuff -x anotherpositional\"")

    arguments = parser.parse_args()

    return arguments


if __name__ == "__main__":
    args = parse_args()
    print(args.extra_args)

The --extra_args argument is then going to be used as a positional argument for a subprocess call.

The argument I need to pass is the following one: --repeat=0. It does not work, I got the following results:

python example.py -e "toto" # This print toto
python example.py -e "--repeat = 0" # This print --repeat = 0
python example.py -e "--repeat" # error: argument -e/--extra_args: expected one argument
python example.py -e "--repeat=0" # error: argument -e/--extra_args: expected one argument

What I understand is that the parser process the --repeat as an argument, find no value for it, and breaks. But unfortunately I have no choice than to write '--repeat=0' all attached, because of the software receiving it.

Would you know any workaround around it ?

Answer 1

I found an ugly workaround but if it exit, I would enjoy a better solution.

I added the following function:

def __workaround_for_extra_utr_args():
    """This workaround is necessary because utr can receives args at \"--repeat=0\" which are not processable by arg parse, even as a string"""
    import sys

    index_of_extra_args = [sys.argv.index(v) for v in ['-e', '--extra_utr_args'] if v in sys.argv]
    space = " "

    if index_of_extra_args:
        index_of_extra_args = index_of_extra_args[0]

        if space not in sys.argv[index_of_extra_args + 1]:
            sys.argv[index_of_extra_args + 1] = sys.argv[index_of_extra_args + 1] + space

Call it before calling the parser, and it will be able to return the string, it is then possible to remove the extra space after the parsing.

Answer 2

It seems there is a problem with the way parse_args inteprets the double dash at the beginning of the arg string.

If you add a prefix, it could be a single character, like a space within the string, this could be a workaround.

python example.py -e " --repeat=0" # OK
python example.py -e ":--repeat=0" # OK
python example.py -e " --repeat=0 --foo=1 --boo=2" # OK

It would be easy then in your code to strip the extra_args of the first character or whatever prefix.

import argparse


def parse_args():
    """Command line argument parser

    Returns:
        A tuple containing the toolset version and a force flag
    """

    parser = argparse.ArgumentParser(description="Arguments for test runner")
    parser.add_argument('-e', '--extra_args', type=str, default="", help="Extra arguments. Enter as a single line encapsulated by quotes. Example: -e \"--repeat=0 otherstuff -x anotherpositional\"")

    arguments = parser.parse_args()
    arguments.extra_args = arguments.extra_args[1:]

    return arguments


if __name__ == "__main__":
    args = parse_args()
    print(args.extra_args)

Not ideal but an easy workaround.

Answer 3

Handling values that start with a '--' is a challenge with argparse. The idea that it marks a flag (as with your --extra_args) is so integral to argparse, that it's hard to work around it.

Simply adding quotes doesn't help. They control how the shell handles the strings, but doesn't otherwise change what's in sys.argv.

https://docs.python.org/3/library/argparse.html#arguments-containing

describes the use of pseudo-argument "--", but that only works for positional arguments. (I'd have to look at the code again to see why it doesn't do anything with optionals.)

So with:

parser = argparse.ArgumentParser(description="Arguments for test runner")
parser.add_argument('extra_args', nargs='*')
arguments = parser.parse_known_args()
print(arguments)

I get:

919:~/mypy$ python3 stack72129874.py -- --repeat
argv ['stack72129874.py', '--', '--repeat']
(Namespace(extra_args=['--repeat']), [])

0919:~/mypy$ python3 stack72129874.py -- --repeat=0
argv ['stack72129874.py', '--', '--repeat=0']
(Namespace(extra_args=['--repeat=0']), [])

0920:~/mypy$ python3 stack72129874.py -- --repeat = 0
argv ['stack72129874.py', '--', '--repeat', '=', '0']
(Namespace(extra_args=['--repeat', '=', '0']), [])

0920:~/mypy$ python3 stack72129874.py -- "--repeat = 0"
argv ['stack72129874.py', '--', '--repeat = 0']
(Namespace(extra_args=['--repeat = 0']), [])

There is a nargs=argparse.REMAINDER that works like the pseudo-argument, but again only for positionals. And there are some problem with using it, so it's now undocumented.

Adding spaces such as in " --repeat = 0" has been mentioned in previous SO about the topic.

And if you don't really need the rest of the argparse machinery (help, error handling, other arguments), doing your own parsing of sys.argv is a perfectly find solution.