Spark executors crash due to netty memory leak : <p>when running spark streaming app that consumes data from kafka topic with 100 partitions, and the streaming runs with 10 executors, 5 cores and 20GB RAM per executor, the executors crash with the following log:</p>

<blockquote>
  <p>ERROR ResourceLeakDetector: LEAK: ByteBuf.release() was not called
  before it's garbage-collected. Enable advanced leak reporting to find
  out where the leak occurred.</p>
  
  <p>ERROR YarnClusterScheduler: Lost executor 18 on worker23.oct.com:
  Slave lost</p>
  
  <p>ERROR ApplicationMaster: RECEIVED SIGNAL TERM</p>
</blockquote>

<p>this exception appears in spark JIRA:</p>

<p><a href="https://issues.apache.org/jira/browse/SPARK-17380" rel="noreferrer">https://issues.apache.org/jira/browse/SPARK-17380</a></p>

<p>and someone wrote that after upgrading to spark 2.0.2 the problem was solved. however we use spark 2.1 as part of HDP 2.6. so I guess this bug wasn't solved in spark 2.1.</p>

<p>there's also someone who encountered this bug and wrote about it in spark user list but got no answer:</p>

<p><a href="http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Receiver-Resource-Leak-td27857.html" rel="noreferrer">http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Receiver-Resource-Leak-td27857.html</a></p>

<p>BTW - the streaming app doesn't call <code>cache()</code> or <code>persist()</code>, so no caching is involved whatsoever.</p>

<p>Did anyone encounter a streaming app that crashed on such bug?</p>
