[SPARK-22862] Docs on lazy elimination of columns missing from an enc…

…oder This behavior has confused some users, so lets clarify it. Author: Michael Armbrust <[email protected]> Closes #20048 from marmbrus/datasetAsDocs. (cherry picked from commit 8df1da3) Signed-off-by: gatorsmile <[email protected]>
apache · Dec 22, 2017 · 1cf3e3a · 1cf3e3a
1 parent 1e4cca0
commit 1cf3e3a
Showing 1 changed file with 4 additions and 0 deletions.
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala b/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
@@ -368,6 +368,10 @@ class Dataset[T] private[sql](
    * If the schema of the Dataset does not match the desired `U` type, you can use `select`
    * along with `alias` or `as` to rearrange or rename as required.
    *
+   * Note that `as[]` only changes the view of the data that is passed into typed operations,
+   * such as `map()`, and does not eagerly project away any columns that are not present in
+   * the specified class.
+   *
    * @group basic
    * @since 1.6.0
    */