Read and Write file from MS Azure using Python
<p>I am newbie to Python and Spark, I am trying to load file from Azure to table. Below is my simple code. </p>

<p><div class="snippet" data-lang="js" data-hide="false" data-console="true" data-babel="false">
<div class="snippet-code">
<pre class="snippet-code-js lang-js prettyprint-override"><code>import os
import sys
os.environ['SPARK_HOME'] = "C:\spark-2.0.0-bin-hadoop2.74"
sys.path.append("C:\spark-2.0.0-bin-hadoop2.7\python")
sys.path.append("C:\spark-2.0.0-bin-hadoop2.7\python\lib\py4j-0.10.1-src.zip")
from pyspark import SparkContext
from pyspark import SparkConf
from pyspark.sql.types import *
from pyspark.sql import *
sc = SparkContext("local", "Simple App")


def loadFile(path, rowDelimeter, columnDelimeter, firstHeaderColName):
 
  
    loadedFile = sc.newAPIHadoopFile(path, "org.apache.hadoop.mapreduce.lib.input.TextInputFormat",
                                      "org.apache.hadoop.io.LongWritable", "org.apache.hadoop.io.Text",
                                      conf={"textinputformat.record.delimiter": rowDelimeter})
 
    
    rddData = loadedFile.map(lambda l:l[1].split(columnDelimeter)).filter(lambda f: f[0] != firstHeaderColName)
        
    return rddData


Schema= StructType([
    
    StructField("Column1", StringType(), True),
    StructField("Column2", StringType(), True),
    StructField("Column3", StringType(), True),
    StructField("Column4", StringType(), True)
        
        

])

rData= loadFile("wasbs://Storagename@Accountname.blob.core.windows.net/File.txt",
                     '\r\n',"#|#","Column1")
DF = sc.createDataFrame(Data,Schema)
DF.write.saveAsTable("Table1")</code></pre>
</div>
</div>
</p>

<p>I am getting error like FileNotFoundError: [WinError 2] The system cannot find the file specified</p>

<p>@Miruthan,
As far as I know, If we'd like to read data from WASB into Spark, the URL syntax is as following :</p>

<pre><code>wasb[s]://&lt;containername&gt;@&lt;accountname&gt;.blob.core.windows.net/&lt;path&gt;
</code></pre>

<p>Meanwhile, due to Azure Storage Blob (WASB) is used as the storage account associated with an HDInsight cluster.Could you please double check it? Any update, please let me know. </p>

python azure apache-spark pyspark