I am trying to overwrite a Spark dataframe using the following option in PySpark but I am not successful

spark_df.write.format('com.databricks.spark.csv').option("header", "true",mode='overwrite').save(self.output_file_path)

the mode=overwrite command is not successful

Try:

spark_df.write.format('com.databricks.spark.csv') \.mode('overwrite').option("header", "true").save(self.output_file_path)