How to efficiently save a Pandas Dataframe into one/more TFRecord file? : <p>First I want to quickly give some background. What I want to achieve eventually is to train a fully connected neural network for a multi-class classification problem under tensorflow framework.</p>

<p>The challenge of the problem is that the size of training data is huge (~ 2 TB). In order for the training to work under limited memory, I want to save training set into small files and use mini-batch gradient descent algorithm to train the model. (Each time only one or a few files are loaded into the memory).</p>

<p>Say now I already have two data frames with processed data, one with X_train (7 million entries * 200 features with column names) and one with training_y (7 million entries * 1 label). <strong>How can I efficiently save this into TFrecord files, keeping column names, row index, etc, and I may want to have each file to contain say 100,000 entries?</strong> I know that with everything under TFrecord I can utilize some of the neat shuffling and batching functionalities implemented in tensorflow. I probably need a very efficient way to write such records because later on I will need to write 2TB of data into this file format. </p>

<p>I tried to search "How to write pandas data frame to TFRecords" on Google but didn't get any luck on good examples. Most examples ask me to create a <code>tf.train.Example</code> column by column, row by row and write to tfrecord file using <code>tf.python_io.TFRecordWriter</code>. Just want to confirm this is best of what I can get here.</p>

<p>If you have other suggestions for the problem I am trying to solve, it will be much appreciated too! </p>
