spark dataframe drop duplicates and keep first : <p>Question: in pandas when dropping duplicates you can specify which columns to keep. Is there an equivalent in Spark Dataframes?</p>

<p>Pandas:</p>

<pre><code>df.sort_values('actual_datetime', ascending=False).drop_duplicates(subset=['scheduled_datetime', 'flt_flightnumber'], keep='first')
</code></pre>

<p>Spark dataframe (I use Spark 1.6.0) doesn't have the keep option</p>

<pre><code>df.orderBy(['actual_datetime']).dropDuplicates(subset=['scheduled_datetime', 'flt_flightnumber'])
</code></pre>

<p>Imagine 'scheduled_datetime' and 'flt_flightnumber' are columns 6 ,17. By creating keys based on the values of these columns we can also deduplicate </p>

<pre><code>def get_key(x):
    return "{0}{1}".format(x[6],x[17])

df= df.map(lambda x: (get_key(x),x)).reduceByKey(lambda x,y: (x))
</code></pre>

<p>but how to specify <strong>to keep the first row</strong> and get rid of the other duplicates ? What about the last row ? </p>
