Error ValueError: cannot reindex from a duplicate axis because of concatenating dataframes
<p>I implemented experimental environment in my project. </p>

<p>This component is based on Scikit learn. </p>

<p>In this compnent I read the given CSV into pandas dataframe. 
After that I selected the best features and reduced the dimensions of the given dataframe from 100 to 5. 
After that I added to this reduced dataframe the removed ID column for future use. This coloumn was dropped by the dimension reduction process. </p>

<p>Everything works fine until I changed my code to read all CSV files and return one union dataframe:</p>

<p>Please look on the next code:
Reading all CSV:</p>

<pre><code>dataframes = []

from os import listdir
from os.path import isfile, join
files_names = [f for f in listdir(full_path_directory_files) if   isfile(join(full_path_directory_files, f))]
for file_name in files_names:
    full_path_file = full_path_directory_files + file_name

    data_frame = pd.read_csv(full_path_file, index_col=None, compression="infer")
dataframes.append(dataframe)
</code></pre>

<p>After that I made concatenation between the dataframes</p>

<pre><code>features_dataframe = pd.concat(dataframes, axis=0)
</code></pre>

<p>I also checked it. 
I created two different dataframes with shape = (200, 100)
and after concatenating it turned to (400, 100)</p>

<p>After that the dataframe was sent into the following method:</p>

<pre><code> def _reduce_dimensions_by_num_of_features(self, features_dataframe, truth_dataframe, num_of_features):
    print("Create dataframe with the {0} best features".format(num_of_features))

## In those functions I got the ids and their class

    ids, id_series = self._create_ids_by_dataframe(features_dataframe)
    features_dataframe_truth_class = self._extract_truth_class_by_truth_dataframe(truth_dataframe, ids)


    k_best_classifier = SelectKBest(score_func=f_classif, k=num_of_features)
    k_best_features = k_best_classifier.fit_transform(features_dataframe, features_dataframe_truth_class)

    reduced_dataframe_column_names = self._get_k_best_feature_names(k_best_classifier, features_dataframe)


    reduced_dataframe = pd.DataFrame(k_best_features, columns=reduced_dataframe_column_names)
</code></pre>

<p>Now I retrieved the ID column:</p>

<pre><code>    reduced_dataframe["Id"] = id_series
</code></pre>

<p>The software it failed on the message:</p>

<pre><code>ValueError: cannot reindex from a duplicate axis
</code></pre>

<p>This is occurred only after the concation of the dataframes.</p>

<p>How can I add the column of the IDs into the dataframe without getting error??</p>

<p>I found the problem:</p>

<p>After the concatenation of the dataframes, the index is changed and when we add the row :</p>

<pre><code>reduced_dataframe["Id"] = id_series
</code></pre>

<p>We got an error.</p>

<p>The solution is to reset the index : </p>

<pre><code>features_dataframe = pd.concat(dataframes, axis=0)
features_dataframe.reset_index(drop=True, inplace=True)
</code></pre>

python csv scikit-learn