Spark: Convert column of string to an array : <p>How to convert a column that has been read as a string into a column of arrays?
i.e. convert from below schema</p>

<pre><code>scala&gt; test.printSchema
root
 |-- a: long (nullable = true)
 |-- b: string (nullable = true)

+---+---+
|  a|  b|
+---+---+
|  1|2,3|
+---+---+
|  2|4,5|
+---+---+
</code></pre>

<p>To: </p>

<pre><code>scala&gt; test1.printSchema
root
 |-- a: long (nullable = true)
 |-- b: array (nullable = true)
 |    |-- element: long (containsNull = true)

+---+-----+
|  a|  b  |
+---+-----+
|  1|[2,3]|
+---+-----+
|  2|[4,5]|
+---+-----+
</code></pre>

<p>Please share both scala and python implementation if possible. 
On a related note, how do I take care of it while reading from the file itself?
I have data with ~450 columns and few of them I want to specify in this format.
Currently I am reading in pyspark as below:</p>

<pre><code>df = spark.read.format('com.databricks.spark.csv').options(
    header='true', inferschema='true', delimiter='|').load(input_file)
</code></pre>

<p>Thanks.</p>
