Skip to content

Commit

Permalink
Address Patrick's comments
Browse files Browse the repository at this point in the history
  • Loading branch information
aarondav committed Dec 17, 2014
1 parent 3b1f84e commit 8a8b373
Showing 1 changed file with 2 additions and 20 deletions.
22 changes: 2 additions & 20 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -866,33 +866,15 @@ Apart from these, the following properties are also available, and may be useful
<td>1</td>
<td>
(Netty only) Connections between hosts are reused in order to reduce connection buildup for
large clusters. For small clusters with many hard disks, this may result in insufficient
large clusters. For clusters with many hard disks and few hosts, this may result in insufficient
concurrency to saturate all disks, and so users may consider increasing this value.
</td>
</tr>
<tr>
<td><code>spark.shuffle.io.serverThreads</code></td>
<td>min(num-cores, 8)</td>
<td>
(Netty only) Size of fixed pool of threads to make requests. As each thread gets 16 MB of
off-heap buffer space initially, the default is limited to 8 cores to avoid excessive off-heap
allocation. It has been experimentally determined that this default should be sufficient to
saturate a 10Gb/s network, but in some cases this may have to be adjusted.
</td>
</tr>
<tr>
<td><code>spark.shuffle.io.clientThreads</code></td>
<td>min(num-cores, 8)</td>
<td>
(Netty only) Size of fixed pool of threads to make requests. This has the same properties as
<code>spark.shuffle.io.serverThreads</code>, above.
</td>
</tr>
<tr>
<td><code>spark.shuffle.io.maxRetries</code></td>
<td>3</td>
<td>
(Netty only) Fetches that fail due to IO-related exceptions are automatically retried if this is
(Netty only) Fetches that fail due to IO-related exceptions are automatically retried if this is
set to a non-zero value. This retry logic helps stabilize large shuffles in the face of long GC
pauses or transient network connectivity issues.
</td>
Expand Down

0 comments on commit 8a8b373

Please sign in to comment.