Fastest way to get a distributed spark cluster to parallelize a massive compressed local directory? : How can I get a distributed spark cluster to parallelize a massive compressed directory (1 terabyte uncompressed) of small text files, if that file is on one local machine? Ideally, this entire document would already be on a shared file system like HDFS or Hive, but it's currently on a local machine. sanitizing and placing each file in this directory onto a distributed data store will take on the order of days if done locally. is there any way for remote spark workers to partition this zipped file, and just start batch processing their chunks and dumping their results in a distributed store? if so, what is the fastest store i should be using? 