How to use data files of sub-directories and perform iterative operation in python

Question

I have my jupyter notebook (python script) in current directory. In current directory, I have two subfolders, namely a and b. In both directories a and b I have equal number of .dat files with same names. For example, directory a contains files, namely x1-x1-val_1, x1-x1-val_5, x1-x1-val_11...x1-x1-val_86 and x1-x2-val_1, x1-x2-val_5, x1-x2-val_11...x1-x2-val_86, i.e. values are in range(1,90,5). Likewise I have files in directory b.

I want to use my python script to access files in a and b to perform iterative operations on .dat files. My present code works only if I keep files of directory a or b in current directory. For example, my script uses following function.

def get_info(test):     
    my_dict = {'test':test}   
    c = []    
    
    for i in range(1,90,5):    
        x_val = 'x_val_'+test+'-val_'+str(i)
        y_val = 'y_val_'+test+'-val_'+str(i)
        
        my_dict[x_val],my_dict[y_val]= np.loadtxt(test+'-val_'+str(i)+'.dat'
                                                        ,usecols= (1,2),unpack=True)
        dw = compute_yy(my_dict[x_val],my_dict[y_val],test)
        c.append(dw)
    
    my_dict.update({test+'_c'+:np.array(c)})

    return my_dict

I call get_info() by using following:

tests = ['x1-x1', 'x1-x2']

new_dict = {}
for i in tests:
     new_dict.update({i:get_info(i)})

How can I use my code to access files in either directory a and/or b? I know its about providing correct path, but I am unsure how can I do so. One way I thought is following;

ext = '.dat'
for files in os.listdir(path_to_dir):
    if files.endswith(ext):
        print(files)   # do operations

Alternative could be to make use of os.path.join(). However, I am unable to solve it such that I can use same python script (with minimum changes perhaps) that can use files and iterate on them which are in subfolders a and b. Thanks for your feedback in advance!

Answer 1

If you want to run get_info() on every folder separatelly then you have two methods:

First: described by @medium-dimensional in comment

You can use os.chdir(folder) to change Current Working Directory. And then code will run with files in this folder

You can see current working directory with print( os.getcwd() )

os.chdir("a")
get_info(i)
os.chdir("..") # move back to parent folder

os.chdir("b")
get_info(i)
os.chdir("..") # move back to parent folder

chdir() (similar to command cd in console) can use relative path (r"a") full path (r"C:\full\path\to\a") and .. to move to parent folder (r"a\..\b")

If files can be in nested folders then .. may not go back you can use getcwd()

cwd = os.getcwd()

os.chdir("folder1/folder2/a")
get_info(i)
os.chdir(cwd) # move back to previous folder

os.chdir("folder1/folder2/b")
get_info(i)
os.chdir(cwd) # move back to previous folder

(BTW: in console on Linux you can use cd - to move back to previous folder)

Second: use folder when you open file

Every command which gets filename can also get path with folder\filename (it can be relative path, full path, and path with ..) like

• r"a\filename.dat"
• r"C:\full\path\to\b\filename.dat"
• r"a\..\b\filename.dat"

So you could define function with extra option folder

def get_info(text, folder):

and use this folder when you read file

loadtxt(folder + r'\' + test+'-val_'+str(i)+'.dat', ...)

or more readable with f-string

loadtxt(rf'{folder}\{test}-val_{i}.dat', ...)

And later you run it as

get_info(i, "a")

get_info(i, "b")