Spark 2.0 deprecates 'DirectParquetOutputCommitter', how to live without it? : <p>Recently we migrated from "EMR on HDFS" --> "EMR on S3" (EMRFS with consistent view enabled) and we realized the Spark 'SaveAsTable' (parquet format) writes to S3 were ~4x slower as compared to HDFS but we found a workaround of using the DirectParquetOutputCommitter -[1] w/ Spark 1.6.</p>

<p>Reason for S3 slowness - We had to pay the so called Parquet tax-[2] where the default output committer writes to a temporary table and renames it later where the rename operation in S3 is very expensive</p>

<p>Also we do understand the risk of using 'DirectParquetOutputCommitter' which is possibility of data corruption w/ speculative tasks enabled.</p>

<p>Now w/ Spark 2.0 this class has been deprecated and we're wondering what options do we have on the table so that we don't get to bear the ~4x slower writes when we upgrade to Spark 2.0. Any Thoughts/suggestions/recommendations would be highly appreciated.</p>

<p>One workaround that we can think of is - Save on HDFS and then copy it to S3 via s3DistCp (any thoughts on how can this be done in sane way as our Hive metadata-store points to S3?)</p>

<p>Also looks like NetFlix has fixed this -[3], any idea on when they're planning to open source it?</p>

<p>Thanks.</p>

<p>[1] - <a href="https://github.com/apache/spark/blob/21d5ca128bf3afd5c2d4c7fcc56240e28443474f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/DirectParquetOutputCommitter.scala" rel="noreferrer">https://github.com/apache/spark/blob/21d5ca128bf3afd5c2d4c7fcc56240e28443474f/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/DirectParquetOutputCommitter.scala</a></p>

<p>[2] - 
<a href="https://www.appsflyer.com/blog/the-bleeding-edge-spark-parquet-and-s3/" rel="noreferrer">https://www.appsflyer.com/blog/the-bleeding-edge-spark-parquet-and-s3/</a></p>

<p>[3] -
<a href="https://www.youtube.com/watch?v=85sew9OFaYc&amp;feature=youtu.be&amp;t=8m39s" rel="noreferrer">https://www.youtube.com/watch?v=85sew9OFaYc&amp;feature=youtu.be&amp;t=8m39s</a> 
<a href="http://www.slideshare.net/AmazonWebServices/bdt303-running-spark-and-presto-on-the-netflix-big-data-platform" rel="noreferrer">http://www.slideshare.net/AmazonWebServices/bdt303-running-spark-and-presto-on-the-netflix-big-data-platform</a></p>
