Broadcast a dictionary to rdd in PySpark : <p>I am just getting the hang of Spark, and I have function that needs to be mapped to an <code>rdd</code>, but uses a global dictionary:</p>

<pre><code>from pyspark import SparkContext

sc = SparkContext('local[*]', 'pyspark')

my_dict = {"a": 1, "b": 2, "c": 3, "d": 4} # at no point will be modified
my_list = ["a", "d", "c", "b"]

def my_func(letter):
    return my_dict[letter]

my_list_rdd = sc.parallelize(my_list)

result = my_list_rdd.map(lambda x: my_func(x)).collect()

print result
</code></pre>

<p>The above gives the expected result; however, I am really not sure about my use of the global variable <code>my_dict</code>. It seems that a copy of the dictionary is made with every partition. And it just does not feel right..</p>

<p>It looked like <a href="http://spark.apache.org/docs/latest/programming-guide.html#broadcast-variables">broadcast</a> is what I am looking for. However, when I try to use it:</p>

<pre><code>my_dict_bc = sc.broadcast(my_dict)

def my_func(letter):
    return my_dict_bc[letter] 
</code></pre>

<p>I get the following error:</p>

<pre><code>TypeError: 'Broadcast' object has no attribute '__getitem__
</code></pre>

<p>This seems to imply that I cannot broadcast a dictionary.</p>

<p>My question: If I have a function that uses a global dictionary, that needs to be mapped to <code>rdd</code>, what is the proper way to do it?</p>

<p>My example is very simple, but in reality <code>my_dict</code> and <code>my_list</code> are much larger, and <code>my_func</code> is more complicated.</p>
