Fast Parquet row count in Spark : <p>The Parquet files contain a per-block row count field. Spark seems to read it at some point (<a href="https://github.com/apache/spark/blob/v2.0.2/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/SpecificParquetRecordReaderBase.java#L151" rel="noreferrer"><code>SpecificParquetRecordReaderBase.java#L151</code></a>).</p>

<p>I tried this in <code>spark-shell</code>:</p>

<pre><code>sqlContext.read.load("x.parquet").count
</code></pre>

<p>And Spark ran two stages, showing various aggregation steps in the DAG. I figure this means it reads through the file normally instead of using the row counts. (I could be wrong.)</p>

<p>The question is: Is Spark already using the row count fields when I run <code>count</code>? Is there another API to use those fields? Is relying on those fields a bad idea for some reason?</p>
