How to install psycopg2 for use in HDInsight PySpark Jupyter notebook
<p>I need to "import psycopg2" as part of my PySpark script.</p>

<p>Per the <a href="https://notebooks.azure.com/faq#upload_data" rel="nofollow">documentation</a>:</p>

<blockquote>
  <p>!pip install
  psycopg2</p>
</blockquote>

<p>but that results in a syntax error (as any ! command does).</p>

<p>I was able to install in an SSH session for /usr/bin/anaconda/bin/python, but it appears Jupyter may be using a different different environment? </p>

<p>I even tried forcing </p>

<blockquote>
  <p>os.environ["PYTHONPATH"]</p>
</blockquote>

<p>in the notebook but no luck, by which I mean from a cell in the notebook:</p>

<ul>
<li>import psycopg2 results in an error that module can't be found</li>
<li>help("modules") does not show psycopg2</li>
<li>help("modules psycopg2") results in the following error:</li>
</ul>

<blockquote>
<pre><code>Here is a list of matching modules.  Enter any module name to get more help.

Failed to write user configuration file.
Please submit a bug report.
sqlalchemy.dialects.postgresql.psycopg2 - .. dialect:: postgresql+psycopg2
sqlalchemy.dialects.postgresql.psycopg2cffi - .. dialect:: postgresql+psycopg2cffi
</code></pre>
</blockquote>

<p>Did you make a typo? Your text was 'pyscopg2' not 'psycopg2'</p>

<pre><code>!pip install psycopg2
</code></pre>

<p>For Python 3.5 and Python 2.7 is installed already on Azure Notebooks.</p>

python azure psycopg2 jupyter hdinsight