PCA on word2vec embeddings : <p>I am trying to reproduce the results of this paper: <a href="https://arxiv.org/pdf/1607.06520.pdf" rel="noreferrer">https://arxiv.org/pdf/1607.06520.pdf</a></p>

<p>Specifically this part:</p>

<blockquote>
  <p>To identify the gender subspace, we took the ten gender pair difference vectors and computed its principal components (PCs). As Figure 6 shows, there is a single direction that explains the majority of variance in these vectors. The first eigenvalue is significantly larger than the rest.</p>
</blockquote>

<p><a href="https://i.stack.imgur.com/EOJJK.png" rel="noreferrer"><img src="https://i.stack.imgur.com/EOJJK.png" alt="enter image description here"></a></p>

<p>I am using the same set of word vectors as the authors (Google News Corpus, 300 dimensions), which I load into word2vec. </p>

<p>The 'ten gender pair difference vectors' the authors refer to are computed from the following word pairs:</p>

<p><a href="https://i.stack.imgur.com/7b6Dj.png" rel="noreferrer"><img src="https://i.stack.imgur.com/7b6Dj.png" alt="enter image description here"></a></p>

<p>I've computed the differences between each normalized vector in the following way:</p>

<pre><code>model = gensim.models.KeyedVectors.load_word2vec_format('GoogleNews-vectors-
negative300.bin', binary = True)
model.init_sims()

pairs = [('she', 'he'),
('her', 'his'),
('woman', 'man'),
('Mary', 'John'),
('herself', 'himself'),
('daughter', 'son'),
('mother', 'father'),
('gal', 'guy'),
('girl', 'boy'),
('female', 'male')]

difference_matrix = np.array([model.word_vec(a[0], use_norm=True) - model.word_vec(a[1], use_norm=True) for a in pairs])
</code></pre>

<p>I then perform PCA on the resulting matrix, with 10 components, as per the paper:</p>

<pre><code>from sklearn.decomposition import PCA
pca = PCA(n_components=10)
pca.fit(difference_matrix)
</code></pre>

<p>However I get very different results when I look at <code>pca.explained_variance_ratio_</code> :</p>

<pre><code>array([  2.83391436e-01,   2.48616155e-01,   1.90642492e-01,
         9.98411858e-02,   5.61260498e-02,   5.29706681e-02,
         2.75670634e-02,   2.21957722e-02,   1.86491774e-02,
         1.99108478e-32])
</code></pre>

<p>or with a chart:</p>

<p><a href="https://i.stack.imgur.com/RuNEi.png" rel="noreferrer"><img src="https://i.stack.imgur.com/RuNEi.png" alt="enter image description here"></a></p>

<p>The first component accounts for less than 30% of the variance when it should be above 60%! </p>

<p>The results I get are similar to what I get when I try to do the PCA on randomly selected vectors, so I must be doing something wrong, but I can't figure out what.</p>

<p>Note: I've tried without normalizing the vectors, but I get the same results.</p>
