Problem using IMF data API for a large number of countries

Question

I am trying to download national account data from the API of the International Financial Statistics from the International Monetary Fund.

I don't have any trouble with downloading a few selected countries. As you can check in the following code, I am able to turn some json from the API into a pandas dataframe.

import requests
import pandas as pd

# Nominal domestic variables
dcn_sa = [
    'NINV_SA_XDC',
    'NGDP_D_SA_IX', 
    'NGDP_SA_XDC', 
    'NCPHI_SA_XDC',
    'NCPHN_SA_XDC',
    'NCGG_SA_XDC',
    'NFI_SA_XDC',
    'NX_SA_XDC',
    'NM_SA_XDC'
]
# A bunch of countries
countries = [
    'MX',
    'FR',
    'US',
    'GB'
]

# API calling
base = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/IFS/'

period = 'Q'
#countries = codes['Alpha2'].values
time = '?startPeriod=1960&endPeriod=2021'

# Get data from the above URL using the requests package
url = f"{base}{period}.{'+'.join(countries)}.{'+'.join(dcn_sa)}.{time}"
rq = requests.get(url)

response = rq.json()

series = response['CompactData']['DataSet']['Series']

N = len(series)

temp_imf = pd.DataFrame()
dum_df = pd.DataFrame()
for n in range(0, N-1):
    temp_dic = series[n].get('Obs')
    temp_df = pd.DataFrame.from_dict(
        temp_dic
        ).rename(
            columns = {
            '@OBS_VALUE':'Value',
            }
        )
    temp_df['Code'] = series[n].get('@REF_AREA')
    temp_df['Variable'] = series[n].get('@INDICATOR')
    temp_df['Period'] = pd.to_datetime(
            [row.replace('-', '') for row in temp_df['@TIME_PERIOD']]
        )
    temp_imf = temp_imf.append(temp_df)
    
imf = pd.pivot(
    temp_imf,
    values='Value', 
    index=['Code', 'Period'],
    columns='Variable'
)

Nevertheless, as I increase de selection of countries to, for example:

countries = [
    'AF', 'AX', 'AL', 'DZ', 'AS', 'AD', 'AO',
    'AI', 'AQ', 'AG', 'AR', 'AM', 'AW', 'AU',
    'AT', 'AZ', 'BS', 'BH', 'BD', 'BB', 'BY',
    'BE', 'BZ', 'BJ', 'BM', 'BT', 'BO', 'BQ',
    'BA', 'BW', 'BV', 'BR', 'IO', 'BN', 'BG',
    'BF', 'BI', 'CV', 'KH', 'CM', 'CA', 'KY',
    'CF', 'TD', 'CL', 'CN', 'CX', 'CC', 'CO',
    'KM', 'CD', 'CG', 'CK', 'CR', 'CI', 'HR',
    'CU', 'CW', 'CY', 'CZ', 'DK', 'DJ', 'DM',
    'DO', 'EC', 'EG', 'SV', 'GQ', 'ER', 'EE',
    'SZ', 'ET', 'FK', 'FO', 'FJ', 'FI', 'FR',
    'GF', 'PF', 'TF', 'GA', 'GM', 'GE', 'DE', 
    'GH', 'GI', 'GR', 'GL', 'GD', 'GP', 'GU',
    'GT', 'GG', 'GN', 'GW', 'GY', 'HT', 'HM',
    'VA', 'HN', 'HK', 'HU', 'IS', 'IN', 'ID'
]

I get the following error

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

I thought that, perhaps empty elements of the json files could be causing trouble so I re-wrote previous code, but iterating country by country

base = 'http://dataservices.imf.org/REST/SDMX_JSON.svc/CompactData/IFS/'
period = 'Q'
time = '?startPeriod=1960&endPeriod=2021'
countries = codes['Alpha2']
temp_imf = pd.DataFrame()
for cont in countries:
    url = f"{base}{period}.{cont}.{'+'.join(dcn_sa)}.{time}"
    rq = requests.get(url)
    if rq == 200:
        response = rq.json()
        series = response['CompactData']['DataSet']['Series']
        N = len(series)
        for n in range(0, N-1):
            temp_dic = series[n].get('Obs')
            temp_df = pd.DataFrame.from_dict(
                temp_dic
            ).rename(
                columns = {
                '@OBS_VALUE':'Value',
            }
            )
            temp_df['Code'] = series[n].get('@REF_AREA')
            temp_df['Variable'] = series[n].get('@INDICATOR')
            temp_df['Period'] = pd.to_datetime(
                [row.replace('-', '') for row in temp_df['@TIME_PERIOD']]
            )
            temp_imf = temp_imf.append(temp_df)
        print(response)
   # print([cont, rq])
        
imf = pd.pivot(
    temp_imf,
    values='Value', 
    index=['Code', 'Period'],
    columns='Variable'
)

but no dataframe is created as when I select only a few countries.

My question is how to avoid these errors and write some code that can handle all the countries.

I hope you can help.

Thanks

Answer 1

I have also seen issues with this API. In that you give the same inputs , but it will error out sometimes , you can handle the request code. But have also seen it work , but there be no data , so you need to resolve if the series is there. On the documentation is says there are limits on how many requests can be made in given time. So introducing a pause might also help

makeRequest = requests.get(url)
if makeRequest.status_code == 200:
     Data = makeRequest.json()
     # Call was a success
     # Look to see if you have a series
     if 'Series' in Data['CompactData']['DataSet']:
           #### You Got Some Data 
     import time
     time.sleep(1)