Spark ML VectorAssembler returns strange output : <p>I am experiencing a very strange behaviour from <code>VectorAssembler</code> and I was wondering if anyone else has seen this.</p>

<p>My scenario is pretty straightforward. I parse data from a <code>CSV</code> file where I have some standard <code>Int</code> and <code>Double</code> fields and I also calculate some extra columns. My parsing function returns this: </p>

<pre><code>val joined = countPerChannel ++ countPerSource //two arrays of Doubles joined
(label, orderNo, pageNo, Vectors.dense(joinedCounts))
</code></pre>

<p>My main function uses the parsing function like this:</p>

<pre><code>val parsedData = rawData.filter(row =&gt; row != header).map(parseLine)
val data = sqlContext.createDataFrame(parsedData).toDF("label", "orderNo", "pageNo","joinedCounts")
</code></pre>

<p>I then use a <code>VectorAssembler</code> like this:</p>

<pre><code>val assembler = new VectorAssembler()
                           .setInputCols(Array("orderNo", "pageNo", "joinedCounts"))
                           .setOutputCol("features")

val assemblerData = assembler.transform(data)
</code></pre>

<p>So when I print a row of my data before it goes into the <code>VectorAssembler</code> it looks like this:</p>

<pre><code>[3.2,17.0,15.0,[0.0,0.0,0.0,0.0,3.0,0.0,0.0,1.0,0.0,0.0,0.0,0.0,4.0,0.0,0.0,2.0]]
</code></pre>

<p>After the transform function of VectorAssembler I print the same row of data and get this:</p>

<pre><code>[3.2,(18,[0,1,6,9,14,17],[17.0,15.0,3.0,1.0,4.0,2.0])]
</code></pre>

<p>What on earth is going on? What has the <code>VectorAssembler</code> done? I 've double checked all the calculations and even followed the simple Spark examples and cannot see what is wrong with my code. Can you? </p>
