How to merge two dataframes in Spark Hadoop? : I am trying to join two dataframes.

data: DataFrame[_1: bigint, _2: vector]

cluster: DataFrame[cluster: bigint]

    result = data.join(broadcast(cluster))

The strange thing is, that all the executors are failing on the joining step.

I have no idea what I could do.

The data file is 2.8 gb on HDFS and the cluster data only 5 mb.
The files are read using Parquet.




