'Glue 3.0 has Psycopg2 but "No module named 'psycopg2._psycopg'"?
I'm using AWS Glue 3.0 and am trying to connect to Redshift using Psycopg2. At first I was uploading a whl file version of it and it would give me the error above. I then read that psycopg2 is included in Python 3 so I ran print(help('modules')) and see psycopg2 in the list of modules. I removed external references (whl file) and it still fails. More recently I tried the parameter --additional-python-modules with a value of psycopg2-binary={I don't recall the version} and that failed.
I don't understand why this particular library won't load. I can import mail, numpy, matplotlib etc with no issue... don't know why this one is stubborn. Am I missing something to get psycopg2 that is provided in AWS to load?
Python version 3.7.10 (default, Jun 3 2021, 00:02:01) [GCC 7.3.1 20180712 (Red Hat 7.3.1-13)]
Module listing
Code:
import sys
import traceback, boto3
print("Python version")
print (sys.version)
print (help('modules'))
import psycopg2
exit()
Solution 1:[1]
Psycopg2 did not show up using print(help('modules')) in a my Glue 3.0 Job.
However, adding a Job Parameter did allow me to import it
psycopg2 2.9.3 (dt dec pq3 ext lo64)
Sources
This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.
Source: Stack Overflow
| Solution | Source |
|---|---|
| Solution 1 | Bob Haffner |