Pyspark: spark data frame column width configuration in Jupyter Notebook
<p>I have the following code in Jupyter Notebook:</p>

<pre><code>import pandas as pd
pd.set_option('display.max_colwidth', 80)
my_df.select('field_1','field_2').show()
</code></pre>

<p>I want to increase the column width so I could see the full value of <code>field_1</code> and <code>field_2</code>. I know we can use  <code>pd.set_option('display.max_colwidth', 80)</code> for pandas data frame, but it doesn't seem to work for spark data frame. </p>

<p>Is there a way to increase the column width for the spark data frame like what we did for pandas data frame? Thanks!</p>

<p>I don't think you can set a specific width, but this will show ensure your data is not cutoff no matter the size</p>

<pre><code>my_df.select('field_1','field_2').show(10, truncate = False)
</code></pre>

python apache-spark pyspark spark-dataframe jupyter-notebook