Why can't add a row to pandas dataframe?
<p>I have a 370864*493 dataset, I want to add a new row at the tail of the data set. I have tried Dataframe.loc[370864]=.... and Dataframe.append(). Both methods didn't work as my expectation.But the same code can work on a smaller data set, just 20 thousand rows. I hope to know the reason. The size of dataset is 1.6G. I use pandas and My IDE is spyder. The image shows more detail. The data source is UCSC cancer browser, LUAD methylation data.<a href="http://i.stack.imgur.com/1apUH.png" rel="nofollow">The tail of the dataframe</a></p>

<pre><code>import pandas as pd
import numpy as np
from sklearn.cross_validation import train_test_split
from sklearn.preprocessing import MinMaxScaler

"""
get clinical information and count number of M0 and M1
"""
def get_Metastasis(sampleID_list,df_clinical):
    num_M0=0
    num_M1=0
    list_Metastasis=[]
    list_Metastasis.append('Metastasis')
    for ID in sampleID_list:
        row_number=df_clinical.loc[df_clinical.sampleID==ID].index[0]
        list_Metastasis.append(df_clinical.loc[row_number,'pathologic_M'])
    for i in range(1,len(list_Metastasis)):
        if list_Metastasis[i]!="M0" and isinstance(list_Metastasis[i],str):
            list_Metastasis[i]="M1"
            num_M1+=1
            list_Metastasis[i]=1
        elif list_Metastasis[i]=="M0":
            num_M0+=1
            list_Metastasis[i]=0
#        else:
#            list_Metastasis[i]=None

    return list_Metastasis, num_M1, num_M0




"""
read Data
"""
path_for_clinical_data="clinical_data"                            
path_for_genomicMatrix="genomicMatrix"
df_clinical = pd.read_table(path_for_clinical_data)
df_genomicMatrix = pd.read_table(path_for_genomicMatrix)
df_genomicMatrix=df_genomicMatrix.dropna(axis=0)      ##get rid of row include nan
"""
Add metastasis information
"""
sampleID_list=list(df_genomicMatrix.columns.values)
sampleID_list=sampleID_list[1:]
list_M=[]

list_M,num_M1,num_M0=get_Metastasis(sampleID_list,df_clinical)
df_genomicMatrix.loc[len(df_genomicMatrix)]=list_M             ## Here is the problem.
</code></pre>

<p>Here is the result:</p>

<pre><code>        sample  TCGA-44-4112-01  TCGA-NJ-A4YP-01  TCGA-86-8278-01  \
485566  cg15678817          0.02110           0.0961          -0.1652   
485567  cg14483317         -0.41520          -0.4051          -0.4117   
485573  cg10230711         -0.42750          -0.3067          -0.4182   
485574  cg16651827          0.22345           0.2358           0.2007   
485576  cg07883722          0.36660           0.3932           0.4155  
</code></pre>

<p>Try changing the last line that you gave us from </p>

<pre><code>df_genomicMatrix.loc[370864]=list_M
</code></pre>

<p>to</p>

<pre><code>df_genomicMatrix.loc[len(df_genomicMatrix)]=list_M
</code></pre>

<p><code>len(df_genomicMatrix)</code> returns the length of the dataframe. Using this instead of a static number should make the code more generic. You included a line in your code to remove rows with NA values, which could have changed the size of the dataframe and making <code>df_genomicMatrix.loc[370864]=list_M</code> inaccurate.</p>

<p>This should really be a comment, but I don't have enough rep to comment on other people's posts (welp).</p>

python pandas