PerformanceWarning: DataFrame is highly fragmented. How to convert in to a more efficient way via pd.concat with designated column name

Question

I got following warning while running under python 3.8 with the newest pandas.

PerformanceWarning: DataFrame is highly fragmented.

this is the place where I compile my data into one single dataframe, and also where the problem pops up.

def get_all_score():
    df = pd.DataFrame()
    for name, code in get_code().items():
        global count
        count += 1
        print("ticker：" + name, "trade_code：" + code, "The {} data updated".format(count))
        try:
            df[name] = indicator_score(code)['total']
            time.sleep(0.33334)
        except:
            continue
    return df

I tried to look up in the forum, but I can't figure out how to manipulate with two variables, df[name]is my column name, and indicator_score(code)['total'] is my column output data, all the fractured dataframes are added horizontally, shown as bellow:

     a     b     c     <<<   zz
1    30    40    10          21
2    41    50    11          33
3    44    66    20          29
4    51    71    19          10
5    31    88    31          60
6    60    95    40          70
.
.
.

what would be a neat way to use pd.concat() to solve my issue? thanks.

Answer 1

This is my workaround on this issue, but it seems not that reliable, one little glitch can totally ruin the past process. Here are my code:

def get_all_score():
    df = pd.DataFrame()
    name_list = []
    for name, code in get_code().items():
        global count
        count += 1
        print("ticker?" + name, "trade_code?" + code, "The {} data updated".format(count))
        try:
            name_list.append(name)
            df = pd.concat([df, indicator_score(code)['??']], axis=1)
            # df[name] = indicator_score(code)['??']
            # time.sleep(0.33334)
        except:
            name_list.remove(name)
            continue
    df.columns = name_list
    return df

I tried to replace name for column name before concat process, however I failed to do so. I only figured out how to replace column name after the concat process. This is such a pain. Does anyone have a better way to do so?