How do I run the main script of a Python package directly?

Question

I have an issue on a large set of scripts I've put into a package, and setup a test repo package_test to get things working, as shown below. I'm using Python 3.7.4 on Windows 10, with VS Code as my IDE.

package_test/
-- package_test/
---- __init__.py
---- __main__.py
---- package_test.py
---- module_1.py
-- setup.py

I have gotten things to work so that I can run this as a module, using python -m package_test from the root of this directory. However, if I try to run the package_test.py module directly (such as having VS Code launch it, or to use the debugger), I get an error.

The problem appears to be with imports. Why can't I run the package_test.py script directly?

---

Here are the relevant files:

__init__.py

from .module1 import *

__main__.py

import package_test.package_test

def main():
    package_test.package_test.main()

if __name__ == '__main__':
    main()

package_test.py

import package_test
from package_test.module1 import *

def main():
    package_test.module1.main()

if __name__ == '__main__':
    main()

module1.py

import package_test
from .module1 import *

def textfx():
    print('Hello textfx!!')

def main():
    package_test.module1.textfx()

if __name__ == '__main__':
    main()

---

The error when running directly is:

USER@PC MINGW64 /c/Code/python/package_test (master)
$ C:/apps/Python37/python.exe c:/Code/python/package_test/package_test/package_test.py
Traceback (most recent call last):
  File "c:/Code/python/package_test/package_test/package_test.py", line 1, in <module>
    import package_test
  File "c:\Code\python\package_test\package_test\package_test.py", line 2, in <module>
    from package_test.module1 import *
ModuleNotFoundError: No module named 'package_test.module1'; 'package_test' is not a package

But, when I run this as a module, the result is:

USER@PC MINGW64 /c/Code/python/package_test (master)
$ py -m package_test
Hello textfx!!

Answer 1

As can be seen from the documentation of sys.path:

As initialized upon program startup, the first item of this list, path[0], is the directory containing the script that was used to invoke the Python interpreter. [...]

Since you are running package_test$ python package_test/package_test.py the first place where Python will look for modules in your example is package_test/package_test. Here it finds the module package_test.py which you import via import package_test. Now this module is cached in sys.modules. When you do from package_test.module1 import * it fetches package_test from the module cache and reports back that this isn't a package and thus it can't perform the import.

You should rename that package_test.py script to something else. Why does it exist in the first place when all it does is importing from another module and __main__ just imports from that script. Why can't you run __main__.py and have it import from module1 directly?

You can place this code at the top of package_test.py and inspect the output:

import sys
print(sys.path)

import package_test
print(sys.modules)
print(package_test)

Answer 2

Two Options

Based on a_guest's answer and further investigation, there are two ways to resolve this as I understand it:

---

#1: Do not have a module name that matches a package name

• This is basically a_guest's answer
• i.e. if package_test is the package, do not have a package_test.py. Name it something like run.py or other functional verb instead of a noun.

This will allow run and debug directly on the script by VS Code, and probably other IDEs as well.

---

#2: Always launch from an script outside the package

Launch the program from a script outside of that package:

• create a run/launch script in the project root and always run from there.
  • this will allow run/debug by VS Code, by running the external script.
  • e.g. Create a ./run_package_test.py script in the project root directory and run a function in /package_test/package_test.py from that script.
• launch using an installed script from setuptools, Poetry, or another package manager (i.e. through a function entry-point). This will work even if you are referencing the problematic module with the same name as the package, because it is installing a separate mini launch script that points to the main script and imports the package reference first.

  • This method, would allow running and debugging by VS Code, but you would have to run the installed script that is located in the virtual environment.

  • For example, if you use the following in the pyproject.toml file with Poetry package manager:

    [tool.poetry.scripts]
    run = "package_test.run_package:main"
    

    To create a script called run when the project is installed (i.e. through poetry install), it will create a script in the virtual environment simply called run (with a corresponding run.cmd for windows). If you create local virtual environments, this would be located at:

    • ./.venv/Scripts/run

    You could open this file (it is a Python file, even though it doesn't have the *.py extension), and run or debug it with VS Code.

I believe #2 works because you are forced to import your package_test.py script as a part of the package from outside the package, and cannot directly import the module.

This means the package gets imported first (or the module gets imported as a nested symbol under the package anyway). This gives us access to the package in the sys.modules cache, rather than just the module.