Replacing negative values in specific columns of a dataframe

Question

This is driving me crazy! I want to replace all negative values in columns containing string "_p" with the value multiplied by -0.5. Here is the code, where Tdf is a dataframe.

    L=list(Tdf.filter(regex='_p').columns)
    Tdf[L]=Tdf[L].astype(float)
    Tdf[Tdf[L]<0]= Tdf[Tdf[L]<0]*-.5  

I get the following error:

"TypeError: Cannot do inplace boolean setting on mixed-types with a non np.nan value"

I variefied that all columns in Tdf[L] are type float64.

Even more confusing is that when I run a code, essentially the same except looping through multiple dataframes, it works:

csv_names=['Full','Missing','MinusMissing']

for DF in csv_names:
    L=list(vars()[DF].iloc[:,1:])
    vars()[DF][L]=vars()[DF][L].astype(float)
    vars()[DF][vars()[DF][L]<0]= vars()[DF][vars()[DF][L]<0]*-.5  

What am I missing?

Answer 1

Please clarify your question. If your question is about the error,

Tdf[Tdf[L]<0]= Tdf[Tdf[L]<0]*-.5 

likely fails to non np.nan null values, as the error describes.

If your question is instead:"How do I multiply negative values by -0.5 in columns with "_p" in the column name?"

for col in Tdf.filter(regex='_p').columns.tolist():
    Tdf[col] = Tdf.apply((lambda Tdf: Tdf[col]*-.5 if Tdf[col] < 0 else Tdf[col], axis =1)

Answer 2

Updated: You just need to filter out the string column which is type object. And then work on the data that is left. You can disable the slice warning if you want.

import pandas as pd
import numpy as np


Tdf = pd.DataFrame(columns=["name", "a_p", "b_p", "c"],
                   data=[["a", -1, -2, -3],
                         ["b", 1, -2, 3],
                         ["c", 1, np.NaN, 3]])

# get only non object columns
sub_Tdf = Tdf.select_dtypes(exclude='object')

# then work on slice
L = list(sub_Tdf.filter(regex='_p').columns)
sub_Tdf[L] = sub_Tdf[L].astype(float)
sub_Tdf[sub_Tdf[L] < 0] = sub_Tdf[sub_Tdf[L] < 0] * -.5

# see results were applied correctly
print(Tdf)

Output:

  name  a_p  b_p  c
0    a   -1 -2.0 -3
1    b    1 -2.0  3
2    c    1  NaN  3

This will trigger the setting value on slice warning. You can disable it with.

import pandas as pd
pd.options.mode.chained_assignment = None