Spark: Find Each Partition Size for RDD : <p>What's the best way of finding each partition size for a given RDD. I'm trying to debug a skewed Partition issue, I've tried this:</p>

<pre><code>l = builder.rdd.glom().map(len).collect()  # get length of each partition
print('Min Parition Size: ',min(l),'. Max Parition Size: ', max(l),'. Avg Parition Size: ', sum(l)/len(l),'. Total Partitions: ', len(l))
</code></pre>

<p>It works fine for small RDDs, but for bigger RDDs, it is giving OOM error. My idea is that <code>glom()</code> is causing this to happen. But anyway, just wanted to know if there is any better way to do it?</p>
