SPOT-212: [Ingest] [Proxy] bluecoat.py parser not able to save into H…

…ive table Replaced saveAsTable syntax (deprecated for Hive tables in Spark 2.0) with insertInto
rabarona · Aug 1, 2017 · 1e9833e · 1e9833e
1 parent 781ccf7
commit 1e9833e
Showing 1 changed file with 1 addition and 1 deletion.
diff --git a/spot-ingest/pipelines/proxy/bluecoat.py b/spot-ingest/pipelines/proxy/bluecoat.py
@@ -121,7 +121,7 @@ def save_data(rdd,sqc,db,db_table,topic):
         sqc.setConf("hive.exec.dynamic.partition", "true")
         sqc.setConf("hive.exec.dynamic.partition.mode", "nonstrict")
         hive_table = "{0}.{1}".format(db,db_table)
-        df.write.saveAsTable(hive_table,format="parquet",mode="append",partitionBy=('y','m','d','h'))
+        df.write.format("parquet").mode("append").insertInto(hive_table)
 
     else:
         print("------------------------LISTENING KAFKA TOPIC:{0}------------------------".format(topic))