How to read data into TensorFlow batches from example queue? : <p>How do I get TensorFlow example queues into proper batches for training?</p>

<p>I've got some images and labels:</p>

<pre><code>IMG_6642.JPG 1
IMG_6643.JPG 2
</code></pre>

<p>(feel free to suggest another label format; I think I may need another dense to sparse step...)</p>

<p>I've read through quite a few tutorials but don't quite have it all together yet.
Here's what I have, with comments indicating the steps required from TensorFlow's <a href="https://www.tensorflow.org/versions/r0.8/how_tos/reading_data/index.html#reading-from-files" rel="noreferrer">Reading Data</a> page.</p>

<ol>
<li>The list of filenames
(optional steps removed for the sake of simplicity)</li>
<li>Filename queue</li>
<li>A Reader for the file format</li>
<li>A decoder for a record read by the reader</li>
<li>Example queue</li>
</ol>

<p>And after the example queue I need to get this queue into batches for training; that's where I'm stuck...</p>

<p><strong>1. List of filenames</strong></p>

<p><code>files = tf.train.match_filenames_once('*.JPG')</code></p>

<p><strong>4. Filename queue</strong></p>

<p><code>filename_queue = tf.train.string_input_producer(files, num_epochs=None, shuffle=True, seed=None, shared_name=None, name=None)</code></p>

<p><strong>5. A reader</strong></p>

<p><code>reader = tf.TextLineReader()
key, value = reader.read(filename_queue)</code></p>

<p><strong>6. A decoder</strong></p>

<p><code>record_defaults = [[""], [1]]
col1, col2 = tf.decode_csv(value, record_defaults=record_defaults)
</code>
(I don't think I need this step below because I already have my label in a tensor but I include it anyways)</p>

<p><code>features = tf.pack([col2])</code></p>

<p>The documentation page has an example to run one image, not get the images and labels into batches:</p>

<p><code>
for i in range(1200):
    # Retrieve a single instance:
    example, label = sess.run([features, col5])
</code></p>

<p>And then below it has a batching section:</p>

<pre><code>def read_my_file_format(filename_queue):
  reader = tf.SomeReader()
  key, record_string = reader.read(filename_queue)
  example, label = tf.some_decoder(record_string)
  processed_example = some_processing(example)
  return processed_example, label

def input_pipeline(filenames, batch_size, num_epochs=None):
  filename_queue = tf.train.string_input_producer(
  filenames, num_epochs=num_epochs, shuffle=True)
  example, label = read_my_file_format(filename_queue)
  # min_after_dequeue defines how big a buffer we will randomly sample
  #   from -- bigger means better shuffling but slower start up and more
  #   memory used.
  # capacity must be larger than min_after_dequeue and the amount larger
  #   determines the maximum we will prefetch.  Recommendation:
  #   min_after_dequeue + (num_threads + a small safety margin) *              batch_size
  min_after_dequeue = 10000
  capacity = min_after_dequeue + 3 * batch_size
  example_batch, label_batch = tf.train.shuffle_batch(
  [example, label], batch_size=batch_size, capacity=capacity,
  min_after_dequeue=min_after_dequeue)
  return example_batch, label_batch
</code></pre>

<p>My question is: <strong>how do I use the above example code with the code I have above?</strong>  I need <em>batches</em> to work with, and most of the tutorials come with mnist batches already.</p>

<pre><code>with tf.Session() as sess:
  sess.run(init)

  # Training cycle
for epoch in range(training_epochs):
    total_batch = int(mnist.train.num_examples/batch_size)
    # Loop over all batches
    for i in range(total_batch):
        batch_xs, batch_ys = mnist.train.next_batch(batch_size)
</code></pre>
