pyspark dataframe add a column if it doesn't exist : <p>I have json data in various json files And the keys could be different in lines, for eg</p>

<pre><code>{"a":1 , "b":"abc", "c":"abc2", "d":"abc3"}
{"a":1 , "b":"abc2", "d":"abc"}
{"a":1 ,"b":"abc", "c":"abc2", "d":"abc3"}
</code></pre>

<p>I want to aggreagate data on column 'b','c','d' and 'f' which is not present in the given json file but could be present in the other files. SO as column 'f' is not present we can take empty string for that column.</p>

<p>I am reading the input file and aggregating the data like this</p>

<pre><code>import pyspark.sql.functions as f
df =  spark.read.json(inputfile)
df2 =df.groupby("b","c","d","f").agg(f.sum(df["a"]))
</code></pre>

<p>This is the final output I want</p>

<pre><code>{"a":2 , "b":"abc", "c":"abc2", "d":"abc3","f":"" }
{"a":1 , "b":"abc2", "c":"" ,"d":"abc","f":""}
</code></pre>

<p>Can anyone please Help? Thanks in advance!</p>
