[SPARK-28366][CORE][FOLLOW-UP] Improve the conf IO_WARNING_LARGEFILET…

…HRESHOLD ### What changes were proposed in this pull request? Improve conf `IO_WARNING_LARGEFILETHRESHOLD` (a.k.a `spark.io.warning.largeFileThreshold`): * reword documentation * change type from `long` to `bytes` ### Why are the changes needed? Improvements according to apache#25134 (comment) & apache#25134 (comment). ### Does this PR introduce any user-facing change? No. ### How was this patch tested? Pass Jenkins. Closes apache#26691 from Ngone51/SPARK-28366-followup. Authored-by: wuyi <[email protected]> Signed-off-by: Wenchen Fan <[email protected]>
attilapiros · Dec 6, 2019 · 58a01ab · 58a01ab
1 parent d5899cd
commit 58a01ab
Showing 1 changed file with 3 additions and 3 deletions.
diff --git a/core/src/main/scala/org/apache/spark/internal/config/package.scala b/core/src/main/scala/org/apache/spark/internal/config/package.scala
@@ -1325,9 +1325,9 @@ package object config {
   private[spark] val IO_WARNING_LARGEFILETHRESHOLD =
     ConfigBuilder("spark.io.warning.largeFileThreshold")
       .internal()
-      .doc("When spark loading one single large file, if file size exceed this " +
-        "threshold, then log warning with possible reasons.")
-      .longConf
+      .doc("If the size in bytes of a file loaded by Spark exceeds this threshold, " +
+        "a warning is logged with the possible reasons.")
+      .bytesConf(ByteUnit.BYTE)
       .createWithDefault(1024 * 1024 * 1024)
 
   private[spark] val EVENT_LOG_COMPRESSION_CODEC =