'Integer out of range when inserting large number of rows to postgress

I have tried multiple solutions and way around to solve this issue, probably something is still I am missing.

I want to insert a list of values to my database. Here is what I am doing -

import psycopg2
import pandas as pd


Region = [   
    "Region1",
    "Region2",

]

qa  = "endpoint1"
def insert_many(data_list):
    """Add data to the table."""
    insert_query = """INSERT INTO pbi_forecast(a,b,c,d,e,f,g,h,i,j,k,l,m,n,o,p,q,r,s,t,u) 
            VALUES (%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s,%s)"""

    conn = None
    try:
        conn = psycopg2.connect(
            database='db',
            user='user',
            host='host1',
            port=5432,
            password=pw
)
        cur = conn.cursor()
        cur.executemany(insert_query, data_list)
        conn.commit()
        cur.close()
    except(Exception, psycopg2.DatabaseError) as error:
        print(error)
    finally:
        if conn is not None:
            conn.close()


# connect prod2
conn_prod2 = psycopg2.connect(
    database='db',
    user='user',
    host='host2',
    port=5432,
    password=pw
)
cur_prod2 = conn_prod2.cursor()

for re in region:

    sql_prod2_select = f"select * from pbi_forecast  where \
                        run_date >= '2022-04-20 00:00:00'\
                        and run_date <= '2022-04-22 00:00:00'\
                        and region = '{re}' ;"

    cur_prod2.execute(sql_prod2_select)
    forecast = pd.DataFrame(cur_prod2.fetchall())
    data_list = [list(row) for row in forecast.itertuples(index=False)]
    insert_many(data_list)

I am getting integer out of range error when running it. If I restrict the insert record to somewhere 50 records it works but when running it without any limit it throws this error.

enter image description here

Thanks in advance.



Sources

This article follows the attribution requirements of Stack Overflow and is licensed under CC BY-SA 3.0.

Source: Stack Overflow

Solution Source