Save HDFS To MongoDB using Spark-DataFrame
<p>I'm trying to save a Spark-DataFrame using PyMongo connecter.
Following is my code, but every-time I run the code I get an error: </p>

<pre><code>java.io.IOException: No FileSystem for scheme: mongodb
</code></pre>

<p>following is my code:</p>

<pre><code>import pymongo
import pymongo_spark
pymongo_spark.activate()
from pyspark import SparkContext, SparkConf
from pyspark.sql import SQLContext
conf = SparkConf()
sc = SparkContext(conf=conf)
sqlContext=SQLContext(sc)
from pyspark.sql import SparkSession
from pyspark.sql import SparkSession
path = "hdfs://localhost:9000/home/hadoop/h_data/sales_ord_univ.csv"
df=sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load(path)
collections=df.collect()
df.write.format('mongodb://localhost:27017/test.sales_order_2').save()
</code></pre>

<p>I have a pretty naive code since Im a newbie to this, but any help towards this would be greatly appreciated. Im using Spark- 2.0.0, Python 2.7.6,MongoDB: 3.2.9</p>

<blockquote>
  <p>I'm trying to save a Spark-DataFrame using PyMongo connector</p>
</blockquote>

<p>You can try to use <a href="https://docs.mongodb.com/spark-connector/" rel="nofollow">MongoDB Connector for Spark</a>. Using your setup environment of <a href="https://spark.apache.org/releases/spark-release-2-0-0.html" rel="nofollow">Apache Spark v2.0.x</a>, Python v2.7.x, and <a href="https://www.mongodb.com/" rel="nofollow">MongoDB</a> v3.2.x, you can do something like below: </p>



<pre><code>from pyspark.sql import SparkSession
spark = SparkSession.builder.appName("Application Name").getOrCreate()
dataframe = spark.read.csv("path/to/file.csv", header=True, mode="DROPMALFORMED")
dataframe.write.format("com.mongodb.spark.sql.DefaultSource")\
               .option("spark.mongodb.output.uri", "mongodb://localhost:27017/database.collection")\
               .save()
</code></pre>

<p>The full version of Python example file can be found on <a href="https://github.com/sindbach/mongodb-pyspark-docker/blob/master/pyspark/files/examples.py#L57" rel="nofollow">MongoDB PySpark Docker: examples.py</a> . Which includes an example to use <a href="https://docs.mongodb.com/manual/aggregation/" rel="nofollow">MongoDB Aggregation</a> in Spark, and <a href="https://docs.mongodb.com/spark-connector/spark-sql/" rel="nofollow">Spark SQL</a>.</p>

<p>If you are familiar with <a href="https://www.docker.com/" rel="nofollow">docker</a>, you can execute the git project <a href="https://github.com/sindbach/mongodb-pyspark-docker" rel="nofollow">MongoDB PySpark Docker</a> using <a href="https://docs.docker.com/compose/" rel="nofollow">docker-compose</a> and run some of the PySpark examples.</p>

<p>You may find the following resources useful:</p>

<ul>
<li><a href="https://docs.mongodb.com/spark-connector/python-api/" rel="nofollow">Spark Connector Python API</a></li>
<li><a href="https://university.mongodb.com/courses/M233/about" rel="nofollow">Free online course M233: Getting Started with Spark and MongoDB</a></li>
</ul>

python mongodb csv hadoop apache-spark