How to set preferred encoding in WSGI to UTF-8

Question

Feeling a bit crazy here. I've got Apache set up with mod_wsgi, but I can't get the encoding to work properly. I have:

• tested that mod_wsgi is running in daemon mode
• read Graham Dumpleton's blog post about setting up the lang and locale settings for the WSGIDaemonProcess directive.
• created a minimal test that seems to demonstrate the issue

# I recompiled the mod_wsgi file to get the Python version correct
sys.version = '3.8.6 (default, Sep 24 2020, 21:54:23) \n[GCC 8.3.0]'
sys.prefix = '/usr/local'
sys.path = ['/usr/local/lib/python38.zip', '/usr/local/lib/python3.8', '/usr/local/lib/python3.8/lib-dynload', '/usr/local/lib/python3.8/site-packages', '/usr/local/src/scorched']

# This seems to be a timing thing? Not sure, but possibly problematic
locale.getlocale() = (None, None)
# This was fixed by setting lang or locale (not sure which)
locale.getdefaultlocale() = ('en_US', 'UTF-8')
sys.getdefaultencoding() = 'utf-8'

# These seem like a problem...
sys.getfilesystemencoding() = 'ascii'
locale.getpreferredencoding(False): 'ANSI_X3.4-1968'

# It's daemon mode
mod_wsgi.process_group = 'cl'

My WSGI configs look like this:

    WSGIScriptAlias / /opt/courtlistener/docker/apache/wsgi-configs/python_version_test.py
    WSGIDaemonProcess cl \
      threads=10 \
      processes=64 \
      python-path=/usr/local/lib/python3.8/site-packages/ \
      lang='en_US.UTF-8' \
      locale='en_US.UTF-8'
    WSGIProcessGroup cl
    WSGIApplicationGroup %{GLOBAL}
    WSGIPassAuthorization On

When I log into the server and start python in the terminal, this line works fine, but it fails when it runs via mod_wsgi:

from reporters_db import REPORTERS

All that line does is import a json file that has some utf-8 content in it. Here's the code behind that import:

db_root = os.path.dirname(os.path.realpath(__file__))
with open(os.path.join(db_root, "data", "reporters.json")) as f:
    REPORTERS = json.load(f, object_hook=datetime_parser)

Since the json call above doesn't have the encoding specified, it uses ASCII and fails:

 Traceback (most recent call last):
   File "/opt/courtlistener/docker/apache/wsgi-configs/python_version_test.py", line 6, in <module>
     from reporters_db import REPORTERS
   File "/usr/local/lib/python3.8/site-packages/reporters_db/__init__.py", line 22, in <module>
     REPORTERS = json.load(f, object_hook=datetime_parser)
   File "/usr/local/lib/python3.8/json/__init__.py", line 293, in load
     return loads(fp.read(),
   File "/usr/local/lib/python3.8/encodings/ascii.py", line 26, in decode
     return codecs.ascii_decode(input, self.errors)[0]
 UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 441720: ordinal not in range(128)

How can I tell it (and the rest of my codebase) to use utf-8 like sane adults?

---

Edit 1

Perhaps it is important to mention that I'm running apache with the following command:

exec apache2ctl -D FOREGROUND "$@"

I thought that would source the /etc/apache2/envvars file, so I appended the following to that file:

export LANG="en_US.UTF-8"

And I tried tweaking my startup command to:

LANG="en_US.UTF-8" exec apache2ctl -D FOREGROUND "$@"

I was hopeful, but no. Still no progress.

Answer 1

Well, I finally figured this out by searching for every time Graham Dumpleton mentioned the word "lang" on the Internet. That eventually turned up this thread, which mentioned that it was possible to not have a locale installed. I was able to check that by running locale -a inside my Ubuntu Docker image, which revealed:

locale -a
C
C.UTF-8
POSIX

So that's the issue! mod_wsgi doesn't know what I'm asking for when I ask for en_US.utf-8, and it doesn't throw an error either. Swapping my settings to instead be set to C.UTF-8 fixed this immediately.

I'm running a slim docker image, so that must be why I lack locales. I also don't have a file at /etc/default/locale that a lot of other answers in this general area refer to.

I've filed this as a bug.

Answer 2

I had a similar UnicodeDecodeError issue when parsing a yaml file containing Unicode characters on Debian 11, Apache2, mod_wsgi.

It was enough to set WSGIDaemonProcess locale to C.UTF-8, then the error went. This single line changed in my /etc/apache2/sites-available/000-default.conf

     WSGIDaemonProcess my_app locale='C.UTF-8'

In the question, mlissner mentioned a bunch of settings tried, but those were not needed for me.