Skip to content

Commit

Permalink
[SPARK-22683][DOC] document tasksPerExecutorSlot parameter
Browse files Browse the repository at this point in the history
  • Loading branch information
jcuquemelle committed Dec 5, 2017
1 parent 997fa42 commit e480d69
Showing 1 changed file with 14 additions and 0 deletions.
14 changes: 14 additions & 0 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -1584,6 +1584,7 @@ Apart from these, the following properties are also available, and may be useful
<code>spark.dynamicAllocation.minExecutors</code>,
<code>spark.dynamicAllocation.maxExecutors</code>, and
<code>spark.dynamicAllocation.initialExecutors</code>
<code>spark.dynamicAllocation.tasksPerExecutorSlots</code>
</td>
</tr>
<tr>
Expand Down Expand Up @@ -1628,6 +1629,19 @@ Apart from these, the following properties are also available, and may be useful
Lower bound for the number of executors if dynamic allocation is enabled.
</td>
</tr>
<tr>
<td><code>spark.dynamicAllocation.tasksPerSlot</code></td>
<td>1</td>
<td>
Each executor can process a certain number of tasks in parallel (task slots).
The number of task slots per executor is: executor.cores / task.cpus.
The ExecutorAllocationManager will set a target number of running executors equal to:
nbCurrentTask / (taskSlots * tasksPerSlot), with nbCurrentTask being the total number
of running and backlogged tasks. With the default value of 1, each available task slot
will compute a single task in average, which gives the best latency. With small tasks
however, this setting wastes a lot of resources due to executor allocation overhead
</td>
</tr>
<tr>
<td><code>spark.dynamicAllocation.schedulerBacklogTimeout</code></td>
<td>1s</td>
Expand Down

0 comments on commit e480d69

Please sign in to comment.