Spark additional configs and enabling log

Spark additional configuration after installation:
spark.dynamicAllocation.enabled=true
spark.dynamicAllocation.executorIdleTimeout=300s
spark.dynamicAllocation.initialExecutors=5
spark.dynamicAllocation.maxExecutors=25
spark.dynamicAllocation.minExecutors=5
spark.shuffle.service.enabled=true
Configure spark log after installing spark
Change following properties to spark-defaults.conf by using Ambari 
(Ambari UI -> Spark -> Configs)
spark.history.fs.logDirectory=hdfs://<namenode>:8020/spark2-history/
spark.eventLog.dir=hdfs://<namenode>:8020/spark2-history/
spark.eventLog.enabled=true
Also in custom-spark2 defaults add:
spark.eventLog.compress=true
Or if HDFS HA is enabled:
spark.history.fs.logDirectory=hdfs://<name_service_id>/spark/applicationHistory
spark.eventLog.dir=hdfs://<name_service_id>/spark/applicationHistory
spark.eventLog.enabled=true
Update spark.history.provider in spark-default.conf using Ambari 
(Ambari UI -> Spark -> Configs -> Advanced spark-defaults)
spark.history.provider = org.apache.spark.deploy.history.FsHistoryProvider
Restart Spark History Server.
0 Comments

There are no comments yet

Leave a comment

Your email address will not be published. Required fields are marked *