read, compare, and save 2 files with shell script (little more then what it sounds like) : <p>So I didn't quite get the answer I was looking for before, so here we go again;</p>

<p>I'm creating a script that searches IPs from a.csv log file against pre-defined blacked IP lists. </p>

<p>It first imports the log file, then parses IPs from it, then searches the parsed IPs against pre-defined blacked IP list, finally it needs to ask user (if any result found) to save the results to the original log file that was imported. </p>

<p>File 1 is a example of IP-output.csv in the code.</p>

<p>File 2 is a example of $filename in the code (original imported .csv).</p>

<p>File 1:</p>

<pre><code>107.147.166.60 ,SUSPICIOUS IP
107.147.167.26 ,SUSPICIOUS IP
108.48.185.186 ,SUSPICIOUS IP
108.51.114.130 ,SUSPICIOUS IP
142.255.102.68 ,SUSPICIOUS IP
</code></pre>

<p>File 2:</p>

<pre><code>outlook.office365.com ,174.203.0.118 ,UserLoginFailed
outlook.office365.com ,107.147.166.60 ,UserLoginFailed
outlook.office365.com ,107.147.167.26 ,UserLoginFailed
outlook.office365.com ,174.205.17.24 ,UserLoginFailed
outlook.office365.com ,108.48.185.186 ,UserLoginFailed
outlook.office365.com ,174.226.15.21 ,UserLoginFailed
outlook.office365.com ,108.51.114.130 ,UserLoginFailed
outlook.office365.com ,67.180.23.93 ,UserLoginFailed
outlook.office365.com ,142.255.102.68 ,UserLoginFailed
outlook.office365.com ,164.106.75.235 ,UserLoginFailed
</code></pre>

<p>I wanna change File 2 to this:</p>

<pre><code>outlook.office365.com ,174.203.0.118 ,UserLoginFailed
outlook.office365.com ,107.147.166.60 ,UserLoginFailed ,SUSPICIOUS IP
outlook.office365.com ,107.147.167.26 ,UserLoginFailed ,SUSPICIOUS IP
outlook.office365.com ,174.205.17.24 ,UserLoginFailed
outlook.office365.com ,108.48.185.186 ,UserLoginFailed ,SUSPICIOUS IP
outlook.office365.com ,174.226.15.21 ,UserLoginFailed
outlook.office365.com ,108.51.114.130 ,UserLoginFailed ,SUSPICIOUS IP
outlook.office365.com ,67.180.23.93 ,UserLoginFailed
outlook.office365.com ,142.255.102.68 ,UserLoginFailed ,SUSPICIOUS IP
outlook.office365.com ,164.106.75.235 ,UserLoginFailed
</code></pre>

<p>This is the script I created:</p>

<pre><code>#!/bin/bash
#
# IP Blacklist Checker
#Import .csv (File within working directory)
echo "Please import a .csv log file to parse/search the IP(s) and UserAgents: "
read filename
#Parsing IPs from .csv log file
echo "Parsing IP(s) from imported log file..."
grep -Eo '([0-9]{1,3}\.){3}[0-9]{1,3}' $filename | sort | uniq &gt; IP-list.txt
echo 'Done'
awk 'END {print NR,"IP(s) Found in imported log file"}' IP-list.txt
echo 'IPs found in imported log file:'
cat IP-list.txt
#searches parsed ip's against blacked ip lists
echo 'Searching parsed IP(s) from pre-defined Blacked IP List Databases...'
fgrep -w -f "IP-list.txt" "IPlist.txt" &gt; IP-output.txt
awk 'END {print NR,"IP(s) Found Blacked IP List Databases"}' IP-output.txt
echo 'Suspicious IPs found in Blacked IP List Databases:'
cat IP-output.txt
while true; do
read -p "Do you want to add results to log file?" yn
case $yn in
    [Yy]* ) grep -Ff IP-output.txt $filename | sed 's/$/ ,SUSPICIOUS IP/' &gt; IP-output.csv &amp;&amp; awk 'FNR==NR {m[$1]=$0; next} {for (i in m) {match($0,i); val=substr($0, RSTART, RLENGTH); if (val) {sub(val, m[i]); print; next}};} 1' IP-output.csv $filename &gt; $filename; break;;
    [Nn]* ) break;;
    * ) echo "Please answer yes or no.";;
esac
done
echo "Finished searching parsed IP(s) from pre-defined Blacked IP List Databases."
rm IP-list.txt IP-output.csv IP-output.txt 
</code></pre>

<p>The log file I'm importing is really long with 15-20 columns, and the IPlist.txt (blacked IPs) has over 15000 IPs in it. After saving the results to the same log file, .csv file gets empty, and if I save it under a different name, all the columns go out of order, and the ", SUSPICIOUS IP" column appears next to the IP column, I need it instead to be at the last column (end of the line).</p>

<p>I also don't know how to prompt to save for a file only if anything was found, if not only prompt nothing found!</p>

<p>The results i'm getting:</p>

<pre><code> outlook.office365.com ,174.203.0.118 ,UserLoginFailed
 outlook.office365.com ,107.147.166.60 ,SUSPICIOUS IP ,UserLoginFailed
 outlook.office365.com ,107.147.167.26 ,SUSPICIOUS IP ,UserLoginFailed
 outlook.office365.com ,174.205.17.24 ,UserLoginFailed
 outlook.office365.com ,108.48.185.186 ,SUSPICIOUS IP ,UserLoginFailed
 outlook.office365.com ,174.226.15.21 ,UserLoginFailed
 outlook.office365.com ,108.51.114.130 ,SUSPICIOUS IP ,UserLoginFailed
 outlook.office365.com ,67.180.23.93 ,UserLoginFailed
 outlook.office365.com ,142.255.102.68 ,SUSPICIOUS IP ,UserLoginFailed
 outlook.office365.com ,164.106.75.235 ,UserLoginFailed
</code></pre>
