My program has two for loops. I generate a df in each looping. I want to append this result. For each iteration of inner loop, 1 row and 24 columns data is generated. For each iteration of outer loop, it generates 8 rows 24 columns data. I am having issues in appending in the right way so the final dataframe has 8 rows and 24 columns. My code:

biglist = []
# The actual code is bigger. Below is representation of it.
for i in range (x1,...,x8):tem_list = []for j in range ([y1,y2,y3],[y4,..]...[y22,y23,y24]):tem_df = pd.DataFrame({'y1':[value1],'y2':[value2],'y3':[value3]},index=i)tem_list.append(tem_df)biglist.append(tem_list)
# convert listss of lists in biglist to a simple list of dfs
biglist1 = [item for sublist in biglist for item in sublist]
df = pd.concat(biglist1)
print(df)

Present output:

# below is actual output of my dataframe: Pmpp_loss Pmpp_delmt  ... Rsh_delmt Rsh_desen
s1    17.0326    42.5349  ...       NaN       NaN
s2        NaN        NaN  ...       NaN       NaN
s3        NaN        NaN  ...       NaN       NaN
s4        NaN        NaN  ...       NaN       NaN
s5        NaN        NaN  ...       NaN       NaN
s6        NaN        NaN  ...       NaN       NaN
s7        NaN        NaN  ...       NaN       NaN
s8        NaN        NaN  ...   92.1853 -0.444959[8 rows x 192 columns]

In the above, 8 rows is correct. But I got 192 columns, instead of 24. Here, 24 columns got repeated 8 times. That is the reason we see many NaNs here.

Try:

• Change this biglist.append(tem_list) to this: biglist.append(pd.concat(tem_list)).

• Remove this line: biglist1 = [item for sublist in biglist for item in sublist]

• Modify this one df = pd.concat(biglist1) to df = pd.concat(biglist)

If you have defined column names, you can also create an empty DataFrame outside your looping scope, and append the data directly on it from your inner loop:

# Before loop
colnames = ['y1', 'y2', 'y3']
df = pd.DataFrame(data=None, columns=colnames)

chaging your append lines to a single one inside your inner loop:

df = df.append(tem_df)

Not needed the use of biglist, tem_list or pd.concat.

Edit after user comments:

biglist = []
for i in range (x1,...,x8):for j in range ([y1,y2,y3],[y4,..]...[y22,y23,y24]):tem_df = pd.DataFrame({'y1':[value1],'y2':[value2],'y3':[value3]},index=i)biglist.append(pd.concat(tem_df),axis=1)
df = pd.concat(biglist)
print(df)