Spark DataFrame mapPartitions : <p>I need to proceed distributed calculation on Spark DataFrame invoking some arbitrary (not SQL) logic on chunks of DataFrame. 
I did:</p>

<pre><code>def some_func(df_chunk):
    pan_df = df_chunk.toPandas()
    #whatever logic here

df = sqlContext.read.parquet(...)
result = df.mapPartitions(some_func)
</code></pre>

<p>Unfortunatelly it leads to:</p>

<blockquote>
  <p>AttributeError: 'itertools.chain' object has no attribute 'toPandas'</p>
</blockquote>

<p>I expected to have spark DataFrame object within each map invocation, instead I got 'itertools.chain'. Why? And how to overcome this?</p>
