Fastest approach to read thousands of images into one big numpy array : <p>I'm trying to find the fastest approach to read a bunch of images from a directory into a numpy array. My end goal is to compute statistics such as the max, min, and nth percentile of the pixels from all these images. This is straightforward and fast when the pixels from all the images are in one big numpy array, since I can use the inbuilt array methods such as <code>.max</code> and <code>.min</code>, and the <code>np.percentile</code> function.</p>

<p>Below are a few example timings with 25 tiff-images (512x512 pixels). These benchmarks are from using <code>%%timit</code> in a jupyter-notebook. The differences are too small to have any practical implications for just 25 images, but I am intending to read thousands of images in the future.</p>

<pre><code># Imports
import os
import skimage.io as io
import numpy as np
</code></pre>

<ol>
<li><p>Appending to a list</p>

<pre><code>%%timeit
imgs = []    
img_path = '/path/to/imgs/'
for img in os.listdir(img_path):    
    imgs.append(io.imread(os.path.join(img_path, img)))    
## 32.2 ms ± 355 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre></li>
<li><p>Using a dictionary</p>

<pre><code>%%timeit    
imgs = {}    
img_path = '/path/to/imgs/'    
for img in os.listdir(img_path):    
    imgs[num] = io.imread(os.path.join(img_path, img))    
## 33.3 ms ± 402 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre></li>
</ol>

<p>For the list and dictionary approaches above, I tried replacing the loop with a the respective comprehension with similar results time-wise. I also tried preallocating the dictionary keys with no significant difference in the time taken. To get the images from a list to a big array, I would use <code>np.concatenate(imgs)</code>, which only takes ~1 ms.</p>

<ol start="3">
<li><p>Preallocating a numpy array along the first dimension</p>

<pre><code>%%timeit    
imgs = np.ndarray((512*25,512), dtype='uint16')    
img_path = '/path/to/imgs/'    
for num, img in enumerate(os.listdir(img_path)):    
    imgs[num*512:(num+1)*512, :] = io.imread(os.path.join(img_path, img))    
## 33.5 ms ± 804 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre></li>
<li><p>Preallocating a numpy along the third dimension</p>

<pre><code>%%timeit    
imgs = np.ndarray((512,512,25), dtype='uint16')    
img_path = '/path/to/imgs/'    
for num, img in enumerate(os.listdir(img_path)):    
    imgs[:, :, num] = io.imread(os.path.join(img_path, img))    
## 71.2 ms ± 2.22 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
</code></pre></li>
</ol>

<p>I initially thought the numpy preallocation approaches would be faster, since there is no dynamic variable expansion in the loop, but this does not seem to be the case. The approach that I find the most intuitive is the last one, where each image occupies a separate dimensions along the third axis of the array, but this is also the slowest. The additional time taken is not due to the preallocation itself, which only takes ~ 1 ms.</p>

<p>I have three question regarding this:</p>

<ol>
<li>Why is the numpy preallocation approaches not faster than the dictionary and list solutions?</li>
<li>Which is the fastest way to read in thousands of images into one big numpy array?</li>
<li>Could I benefit from looking outside numpy and scikit-image, for an even faster module for reading in images? I tried <code>plt.imread()</code>, but the <code>scikit-image.io</code> module is faster.</li>
</ol>
