Concatenating multiple csv files into a single csv with the same header - Python : <p>I am currently using the below code to import 6,000 csv files (with headers) and export them into a single csv file (with a single header row).</p>

<pre><code>#import csv files from folder
path =r'data/US/market/merged_data'
allFiles = glob.glob(path + "/*.csv")
stockstats_data = pd.DataFrame()
list_ = []

for file_ in allFiles:
    df = pd.read_csv(file_,index_col=None,)
    list_.append(df)
    stockstats_data = pd.concat(list_)
    print(file_ + " has been imported.")
</code></pre>

<p>This code works fine, but it is slow. It can take up to 2 days to process. </p>

<p>I was given a single line script for Terminal command line that does the same (but with no headers). This script takes 20 seconds.</p>

<pre><code> for f in *.csv; do cat "`pwd`/$f" | tail -n +2 &gt;&gt; merged.csv; done 
</code></pre>

<p>Does anyone know how I can speed up the first Python script? To cut the time down, I have thought about not importing it into a DataFrame and just concatenating the CSVs, but I cannot figure it out. </p>

<p>Thanks.</p>
