How to use s3 with Apache spark 2.2 in the Spark shell : <p>I'm trying to load data from an Amazon AWS S3 bucket, while in the Spark shell.</p>

<p>I have consulted the following resources:</p>

<p><a href="https://stackoverflow.com/questions/43654319/parsing-files-from-amazon-s3-with-apache-spark">Parsing files from Amazon S3 with Apache Spark</a></p>

<p><a href="https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark">How to access s3a:// files from Apache Spark?</a></p>

<p><a href="https://community.hortonworks.com/articles/25523/hdp-240-and-spark-160-connecting-to-aws-s3-buckets.html" rel="noreferrer">Hortonworks Spark 1.6 and S3</a></p>

<p><a href="https://www.cloudera.com/documentation/enterprise/5-5-x/topics/spark_s3.html" rel="noreferrer">Cloudera</a></p>

<p><a href="https://gist.github.com/tobilg/e03dbc474ba976b9f235" rel="noreferrer">Custom s3 endpoints</a></p>

<p>I have downloaded and unzipped <a href="https://spark.apache.org/downloads.html" rel="noreferrer">Apache Spark 2.2.0</a>. In <code>conf/spark-defaults</code> I have the following (note I replaced <code>access-key</code> and <code>secret-key</code>):</p>

<pre><code>spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.access.key=access-key 
spark.hadoop.fs.s3a.secret.key=secret-key
</code></pre>

<p>I have downloaded <code>hadoop-aws-2.8.1.jar</code> and <code>aws-java-sdk-1.11.179.jar</code> from <a href="https://mvnrepository.com" rel="noreferrer">mvnrepository</a>, and placed them in the <code>jars/</code> directory. I then start the Spark shell:</p>

<pre><code>bin/spark-shell --jars jars/hadoop-aws-2.8.1.jar,jars/aws-java-sdk-1.11.179.jar
</code></pre>

<p>In the shell, here is how I try to load data from the S3 bucket:</p>

<pre><code>val p = spark.read.textFile("s3a://sparkcookbook/person")
</code></pre>

<p>And here is the error that results:</p>

<pre><code>java.lang.NoClassDefFoundError: org/apache/hadoop/fs/GlobalStorageStatistics$StorageStatisticsProvider
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:348)
  at org.apache.hadoop.conf.Configuration.getClassByNameOrNull(Configuration.java:2134)
  at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2099)
  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
</code></pre>

<p>When I instead try to start the Spark shell as follows:</p>

<pre><code>bin/spark-shell --packages org.apache.hadoop:hadoop-aws:2.8.1
</code></pre>

<p>Then I get two errors: one when the interperter starts, and another when I try to load the data. Here is the first:</p>

<pre><code>:: problems summary ::
:::: ERRORS
    unknown resolver null

    unknown resolver null

    unknown resolver null

    unknown resolver null

    unknown resolver null

    unknown resolver null


:: USE VERBOSE OR DEBUG MESSAGE LEVEL FOR MORE DETAILS
</code></pre>

<p>And here is the second:</p>

<pre><code>val p = spark.read.textFile("s3a://sparkcookbook/person")
java.lang.IllegalAccessError: tried to access method org.apache.hadoop.metrics2.lib.MutableCounterLong.&lt;init&gt;(Lorg/apache/hadoop/metrics2/MetricsInfo;J)V from class org.apache.hadoop.fs.s3a.S3AInstrumentation
  at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(S3AInstrumentation.java:195)
  at org.apache.hadoop.fs.s3a.S3AInstrumentation.streamCounter(S3AInstrumentation.java:216)
  at org.apache.hadoop.fs.s3a.S3AInstrumentation.&lt;init&gt;(S3AInstrumentation.java:139)
  at org.apache.hadoop.fs.s3a.S3AFileSystem.initialize(S3AFileSystem.java:174)
  at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2669)
  at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
  at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
  at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
  at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
  at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
  at org.apache.spark.sql.execution.datasources.DataSource.hasMetadata(DataSource.scala:301)
  at org.apache.spark.sql.execution.datasources.DataSource.resolveRelation(DataSource.scala:344)
  at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:152)
  at org.apache.spark.sql.DataFrameReader.text(DataFrameReader.scala:506)
  at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:542)
  at org.apache.spark.sql.DataFrameReader.textFile(DataFrameReader.scala:515)
</code></pre>

<p>Could someone suggest how to get this working? Thanks.</p>
