Convert csv to parquet file using python : <p>I am trying to convert a .csv file to a .parquet file.<br>
The csv file (<code>Temp.csv</code>) has the following format    </p>

<pre><code>1,Jon,Doe,Denver
</code></pre>

<p>I am using the following python code to convert it into parquet</p>

<pre><code>from pyspark import SparkContext
from pyspark.sql import SQLContext
from pyspark.sql.types import *
import os

if __name__ == "__main__":
    sc = SparkContext(appName="CSV2Parquet")
    sqlContext = SQLContext(sc)

    schema = StructType([
            StructField("col1", IntegerType(), True),
            StructField("col2", StringType(), True),
            StructField("col3", StringType(), True),
            StructField("col4", StringType(), True)])
    dirname = os.path.dirname(os.path.abspath(__file__))
    csvfilename = os.path.join(dirname,'Temp.csv')    
    rdd = sc.textFile(csvfilename).map(lambda line: line.split(","))
    df = sqlContext.createDataFrame(rdd, schema)
    parquetfilename = os.path.join(dirname,'output.parquet')    
    df.write.mode('overwrite').parquet(parquetfilename)
</code></pre>

<p>The result is only a folder named, <code>output.parquet</code> and not a parquet file that I'm looking for, followed by the following error on the console.  </p>

<p><a href="https://i.stack.imgur.com/WWDID.png" rel="noreferrer"><img src="https://i.stack.imgur.com/WWDID.png" alt="CSV to Parquet Error"></a></p>

<p>I have also tried running the following code to face a similar issue.  </p>

<pre><code>from pyspark.sql import SparkSession
import os

spark = SparkSession \
    .builder \
    .appName("Protob Conversion to Parquet") \
    .config("spark.some.config.option", "some-value") \
    .getOrCreate()

# read csv
dirname = os.path.dirname(os.path.abspath(__file__))
csvfilename = os.path.join(dirname,'Temp.csv')    
df = spark.read.csv(csvfilename)

# Displays the content of the DataFrame to stdout
df.show()
parquetfilename = os.path.join(dirname,'output.parquet')    
df.write.mode('overwrite').parquet(parquetfilename)
</code></pre>

<p>How to best do it? Using windows, python 2.7.</p>
