pandas dataframe issue with special characters
<p>I am struggling with the following issue with pandas data frame
Python 2.7.12
pandas 0.18.1</p>

<pre><code>df = pd.read_csv(file_name, encoding='utf-16', header=0, index_col=False, error_bad_lines=False,
                     names=['Package_Name','Crash_Report_Date_And_Time','Crash_Report_Millis_Since_Epoch','Device', 'Android_OS_Version','App_Version_Name','App_Version_Code','Exception_Class_Name','Exception_Message','Throwing_File_Name','Throwing_Class_Name','Throwing_Method_Name','Throwing_Line_Number','Native_Crash_Library','Native_Crash_Location','Crash_Link'])
</code></pre>

<p>I debug the code and found following data is not inserting properly in the dataframe. </p>

<p>There are some special characters in the <code>Exception_Message</code> field which is telling pandas to move rest of the data on that row to the next row.  </p>

<p>Some how Pandas is not reading the file properly. </p>

<p>Following is the output for both the rows. 137 and 138 are the row numbers.</p>

<pre><code>Package_Name Crash_Report_Date_And_Time  \
137  com.vcast.mediamanager        2016-09-05 14:54:13   
138                     NaN                 Class.java   

    Crash_Report_Millis_Since_Epoch        Device  Android_OS_Version  \
137                   1473087253130       victara                  22   
138                 java.lang.Class  classForName                  -2   

    App_Version_Name  App_Version_Code  \
137          14.3.34      1.503050e+09   
138              NaN               NaN   

                                  Exception_Class_Name  \
137                   java.lang.ClassNotFoundException   
138  https://play.google.com/apps/publish?dev_acc=0...   

                     Exception_Message Throwing_File_Name Throwing_Class_Name  \
137  Invalid name: com.strumsoft.appen                NaN                 NaN   
138                                NaN                NaN                 NaN   

    Throwing_Method_Name  Throwing_Line_Number Native_Crash_Library  \
137                  NaN                   NaN                  NaN   
138                  NaN                   NaN                  NaN   

    Native_Crash_Location Crash_Link account_id  
137                   NaN        NaN       NONE  
138      

         NaN        NaN       NONE  
</code></pre>

<p>Row 138 is created erroneously with some data from row 137. <code>Exception Message</code> field in 137 has some value which is breaking that row to the next row.  Which is wrong.</p>

<p>I tried different encoding nothing helped. Can anyone please help?</p>

<p>So the solution was really simple. In Pandas 0.18 I had to specify the <code>lineterminator='n'</code></p>

<pre><code>df = pd.read_csv(file_name,lineterminator='\n', encoding='utf-16', delimiter=',', header=0, index_col=False, error_bad_lines=False,...
</code></pre>

<p>This simple flag fixed my issue. </p>

python pandas