Mailgun only sends partial zip file in Python script when using zipfile zlib python and mailgun

Question

EDIT: I tried to implement gzip but couldn't get it to work with the zipfile library. The documentation seems to indicate zlib is the only library compatible. I also attempted to reinstall zlib.

I have run into a curious issue with a Python script I have created. The point of the script is to pull some data from SQL server, dump it into a CSV, zip it up (compressed) and email it through the Mailgun API.

if I zip the file without zlib compression, mailgun picks up the entire zipfile (14mb with test data) and emails it off successfully. I am able to download the zipfile from the email and my local filesystem and open the csv file within without issue.

But as soon as I add the zipfile.ZIP_DEFLATED compression method, I encounter issues. The generated zip gets down to 365kb on my local filesystem and opens up just fine. But when I look at the generated email, the zip files size is severely truncated and is about 865bytes. if I attempt to download it and open it, I get "this is not a valid zip file" error from Windows.

Why does Mailgun send the full uncompressed zip, but not the compressed zip even though it opens fine on my filesystem? One crazy thing I tried even the code is obviously synchronous, was adding pauses between the function calls in case the compression needed more time. This didn't help.

Code is below, clipped for brevity.

import pyodbc
import unicodecsv
import time
import requests
import zipfile
import zlib

now = time.localtime(time.time())
today = time.strftime("%Y-%m-%d %H%M%p", now)

filename_sql1 = "SCORM Courses " + today
filename_sql2 = "Non-SCORM Courses " + today

print "ready...."

class ODBCtoCSV(object):

    def __init__(self):

    def getEncodedData (self, doubleList):
        data=struct.pack(len(doubleList) * 'd', *doubleList)
        return base64.b64encode(data)

    def dump(self, sql, filename, include_headers=True):
        f = unicodecsv.writer(file(filename + ".csv", 'wb'))

        cnxn = pyodbc.connect(self.connect_string)
        c = cnxn.cursor()
        c.execute(sql)

        if include_headers:
            f.writerow([d[0] for d in c.description])


            f.writerows(c.fetchall())
            cnxn.close()

def generatereport(s, filename):

    print "Processing %s" % (filename)
    if __name__ == '__main__':

        query = ODBCtoCSV()
        query.dump(s, filename)
        f = open(filename + ".csv", 'rb')


def zippy(filename):
    zf = zipfile.ZipFile(filename + ".zip", "w", zipfile.ZIP_DEFLATED)
    zf.write(filename + ".csv")
    zf.close()


def send_notification_message(filename):
    return requests.post(
        "xxxxxxx",
        auth=("api", "xxxx"),
        files=[("attachment", open(filename + ".zip"))],
        data={"from": "xxxxxxx",
              "to": ["xxxxxx"],
              "subject": " Reports are Available",
              "text": "This is an automated message. The reports have run successfully and are available via attachment.",
              "html": "<p><b>This is an automated message.</b></p><p>The reports have run successfully and are available via attachment.</p>",
              "o:tag": "xxxxxx"})        

generatereport(sql1, filename_sql1)
zippy(filename_sql1)
print filename_sql1 + "Succesfully generated"
generatereport(sql2, filename_sql2)
zippy(filename_sql2)
print filename_sql2 + "Succesfully generated"
send_notification_message(filename_sql1)
send_notification_message(filename_sql2)

exit

Answer 1

I resolved this by changing the mode file IO mode to'rb' in the Mailgun call.

I am not sure why it worked for uncompressed zip files, and not compressed zipfiles still. Revised code is below.

files=[("attachment", open(filename + ".zip", "rb"))],