how to sort partition in apache spark : Hi i have a used the dataframe which contains the query


df : Dataframe =spark.sql(s"show Partitions $yourtablename")

now the number of partition changes everyday as it runs everyday

the main concern is i need to fetch the latest partition. 

supoose i get the partition for a random table for a particular day
like  
year=2019/month=1/day=1

year=2019/month=1/day=10

year=2019/month=1/day=2

year=2019/month=1/day=21

year=2019/month=1/day=22

year=2019/month=1/day=23

year=2019/month=1/day=24

year=2019/month=1/day=25

year=2019/month=1/day=26

year=2019/month=2/day=27

year=2019/month=2/day=3


Now of you can see have as the functionality that it sorts the partition after day=1 comes day=10  so this creates a problem as need to fetch the latest partition.



I have managed to get the partition by using 
val df =dff.orderby(col("partition").desc.limit(1)

but this gives me the tail -1 partition and not the latest partition .

How can i get the latest partition from the tables overcoming hives's limitation of arranging partition.

so suppose in the above example  
  

     i need to pickup
     year=2019/month=2/day=27  
     and not year=2019/month=2/day=3 
     which is the last partition in the table