Code

Resync from upstream master#307

Code

Merged

dansanduleac merged 60 commits intomasterpalantir/spark:masterfrom ds/sync-upstreampalantir/spark:ds/sync-upstreamCopy head branch name to clipboardFeb 6, 2018

+2,272-936

Commits

Commits on Jan 30, 2018

• [SPARK-23260][SPARK-23262][SQL] several data source v2 naming cleanup

  Show description for 0a9ac02
  cloud-fan

  committed
  0a9ac02

  Copy full SHA for 0a9ac02

• [SPARK-23261][PYSPARK] Rename Pandas UDFs

  Show description for 7a2ada2
  

  gatorsmile

  authored and

  HyukjinKwon

  committed
  7a2ada2

  Copy full SHA for 7a2ada2

• [SPARK-23222][SQL] Make DataFrameRangeSuite not flaky

  Show description for 84bcf9d
  

  viirya

  authored and

  cloud-fan

  committed
  84bcf9d

  Copy full SHA for 84bcf9d

• [SPARK-23174][BUILD][PYTHON][FOLLOWUP] Add pycodestyle*.py to .gitignore file.

  Show description for a23187f
  

  ueshin

  authored and

  HyukjinKwon

  committed
  a23187f

  Copy full SHA for a23187f

• [SPARK-23267][SQL] Increase spark.sql.codegen.hugeMethodLimit to 65535

  Show description for 31c00ad
  gatorsmile

  committed
  31c00ad

  Copy full SHA for 31c00ad

• [SPARK-23275][SQL] hive/tests have been failing when run locally on the laptop (Mac) with OOM

  Show description for 58fcb5a
  

  dilipbiswal

  authored and

  gatorsmile

  committed
  58fcb5a

  Copy full SHA for 58fcb5a

• [MINOR] Fix typos in dev/* scripts.

  Show description for 9623a98
  

  ashashwat

  authored and

  HyukjinKwon

  committed
  9623a98

  Copy full SHA for 9623a98

Commits on Jan 31, 2018

• [SPARK-23276][SQL][TEST] Enable UDT tests in (Hive)OrcHadoopFsRelationSuite

  Show description for 7786616
  

  dongjoon-hyun

  authored and

  gatorsmile

  committed
  7786616

  Copy full SHA for 7786616

• [SPARK-23274][SQL] Fix ReplaceExceptWithFilter when the right's Filter contains the references that are not in the left output

  Show description for ca04c3f
  gatorsmile

  committed
  ca04c3f

  Copy full SHA for ca04c3f

• [SPARK-23279][SS] Avoid triggering distributed job for Console sink

  Show description for 8c6a9c9
  jerryshao

  committed
  8c6a9c9

  Copy full SHA for 8c6a9c9

• [SPARK-23272][SQL] add calendar interval type support to ColumnVector

  Show description for 695f714
  cloud-fan

  committed
  695f714

  Copy full SHA for 695f714

• [SPARK-23112][DOC] Update ML migration guide with breaking and behavior changes.

  Show description for 161a3f2
  

  Nick Pentreath

  committed
  161a3f2

  Copy full SHA for 161a3f2

• [SPARK-23228][PYSPARK] Add Python Created jsparkSession to JVM's defaultSession

  Show description for 3d0911b
  

  jerryshao

  authored and

  HyukjinKwon

  committed
  3d0911b

  Copy full SHA for 3d0911b

• revert [SPARK-22785][SQL] remove ColumnVector.anyNullsSet

  Show description for 48dd6a4
  cloud-fan

  committed
  48dd6a4

  Copy full SHA for 48dd6a4

• [SPARK-23249][SQL] Improved block merging logic for partitions

  Show description for 8c21170
  

  Glen Takahashi

  authored and

  cloud-fan

  committed
  8c21170

  Copy full SHA for 8c21170

• [SPARK-21525][STREAMING] Check error code from supervisor RPC.

  Show description for dd242ba
  

  Marcelo Vanzin

  committed
  dd242ba

  Copy full SHA for dd242ba

• [SPARK-23281][SQL] Query produces results in incorrect order when a composite order by clause refers to both original columns and aliases

  Show description for 9ff1d96
  

  dilipbiswal

  authored and

  gatorsmile

  committed
  9ff1d96

  Copy full SHA for 9ff1d96

Commits on Feb 1, 2018

• [SPARK-23157][SQL][FOLLOW-UP] DataFrame -> SparkDataFrame in R comment

  Show description for f470df2
  

  Henry Robinson

  authored and

  HyukjinKwon

  committed
  f470df2

  Copy full SHA for f470df2

• [SPARK-23280][SQL] add map type support to ColumnVector

  Show description for 52e00f7
  cloud-fan

  committed
  52e00f7

  Copy full SHA for 52e00f7

• [SPARK-23247][SQL] combines Unsafe operations and statistics operations in Scan Data Source

  Show description for 2ac895b
  

  heary-cao

  authored and

  cloud-fan

  committed
  2ac895b

  Copy full SHA for 2ac895b

• [SPARK-23268][SQL] Reorganize packages in data source V2

  Show description for 56ae326
  

  gengliangwang

  authored and

  gatorsmile

  committed
  56ae326

  Copy full SHA for 56ae326

• [SPARK-21396][SQL] Fixes MatchError when UDTs are passed through Hive Thriftserver

  Show description for b2e7677
  

  atallahhezbor

  authored and

  gatorsmile

  committed
  b2e7677

  Copy full SHA for b2e7677

• [SPARK-23188][SQL] Make vectorized columar reader batch size configurable

  Show description for cc41245
  

  jiangxb1987

  authored and

  cloud-fan

  committed
  cc41245

  Copy full SHA for cc41245

• [SQL][MINOR] Inline SpecifiedWindowFrame.defaultWindowFrame().

  Show description for b6b50ef
  

  jiangxb1987

  authored and

  gatorsmile

  committed
  b6b50ef

  Copy full SHA for b6b50ef

• Revert "[SPARK-23200] Reset Kubernetes-specific config on Checkpoint restore"

  Show description for 4b7cd47
  jerryshao

  committed
  4b7cd47

  Copy full SHA for 4b7cd47

• [SPARK-22274][PYTHON][SQL][FOLLOWUP] Use assertRaisesRegexp instead of assertRaisesRegex.

  Show description for 07cee33
  

  ueshin

  authored and

  gatorsmile

  committed
  07cee33

  Copy full SHA for 07cee33

• [SPARK-23107][ML] ML 2.3 QA: New Scala APIs, docs.

  Show description for e15da5b
  

  yanboliang

  authored and

  Nick Pentreath

  committed
  e15da5b

  Copy full SHA for e15da5b

• [SPARK-23280][SQL][FOLLOWUP] Fix Java style check issues.

  Show description for 8bb70b0
  ueshin

  committed
  8bb70b0

  Copy full SHA for 8bb70b0

• [SPARK-23280][SQL][FOLLOWUP] Enable MutableColumnarRow.getMap().

  Show description for 89e8d55
  ueshin

  committed
  89e8d55

  Copy full SHA for 89e8d55

• [SPARK-23202][SQL] Add new API in DataSourceWriter: onDataWriterCommit

  Show description for ffbca84
  

  gengliangwang

  authored and

  cloud-fan

  committed
  ffbca84

  Copy full SHA for ffbca84

• [SPARK-23289][CORE] OneForOneBlockFetcher.DownloadCallback.onData should write the buffer fully

  Show description for ec63e2d
  

  zsxwing

  authored and

  cloud-fan

  committed
  ec63e2d

  Copy full SHA for ec63e2d

• [SPARK-13983][SQL] Fix HiveThriftServer2 can not get "--hiveconf" and ''--hivevar" variables since 2.0

  Show description for f051f83
  

  wangyum

  authored and

  gatorsmile

  committed
  f051f83

  Copy full SHA for f051f83

• [SPARK-23293][SQL] fix data source v2 self join

  Show description for 73da3b6
  

  cloud-fan

  authored and

  gatorsmile

  committed
  73da3b6

  Copy full SHA for 73da3b6

• [INFRA] Close stale PRs.

  Show description for 4bcfdef
  

  Marcelo Vanzin

  committed
  4bcfdef

  Copy full SHA for 4bcfdef

• [SPARK-23296][YARN] Include stacktrace in YARN-app diagnostic

  Show description for 032c11b
  

  gerashegalov

  authored and

  Marcelo Vanzin

  committed
  032c11b

  Copy full SHA for 032c11b

Commits on Feb 2, 2018

• [SPARK-23284][SQL] Document the behavior of several ColumnVector's get APIs when accessing null slot

  Show description for 90848d5
  

  viirya

  authored and

  cloud-fan

  committed
  90848d5

  Copy full SHA for 90848d5

• [SPARK-23020][CORE] Fix another race in the in-process launcher test.

  Show description for 969eda4
  

  Marcelo Vanzin

  authored and

  cloud-fan

  committed
  969eda4

  Copy full SHA for 969eda4

• [SPARK-23306] Fix the oom caused by contention

  Show description for b3a0428
  

  Zhan Zhang

  authored and

  cloud-fan

  committed
  b3a0428

  Copy full SHA for b3a0428

• [SPARK-23301][SQL] data source column pruning should work for arbitrary expressions

  Show description for 19c7c7e
  

  cloud-fan

  authored and

  gatorsmile

  committed
  19c7c7e

  Copy full SHA for 19c7c7e

• [SPARK-23312][SQL] add a config to turn off vectorized cache reader

  Show description for b9503fc
  cloud-fan

  committed
  b9503fc

  Copy full SHA for b9503fc

• [SPARK-23253][CORE][SHUFFLE] Only write shuffle temporary index file when there is not an existing one

  Show description for dd52681
  

  yaooqinn

  authored and

  squito

  committed
  dd52681

  Copy full SHA for dd52681

• [SPARK-23295][BUILD][MINOR] Exclude Waring message when generating versions in make-distribution.sh

  Show description for eefec93
  

  yaooqinn

  authored and

  srowen

  committed
  eefec93

  Copy full SHA for eefec93

Commits on Feb 3, 2018

• [SPARK-23064][SS][DOCS] Stream-stream joins Documentation - follow up

  Show description for eaf35de
  tdas

  committed
  eaf35de

  Copy full SHA for eaf35de

• [SQL] Minor doc update: Add an example in DataFrameReader.schema

  Show description for 3ff83ad
  

  rxin

  authored and

  gatorsmile

  committed
  3ff83ad

  Copy full SHA for 3ff83ad

• [SPARK-23317][SQL] rename ContinuousReader.setOffset to setStartOffset

  Show description for fe73cb4
  

  cloud-fan

  authored and

  gatorsmile

  committed
  fe73cb4

  Copy full SHA for fe73cb4

• [SPARK-23311][SQL][TEST] add FilterFunction test case for test CombineTypedFilters

  Show description for 63b49fa
  

  heary-cao

  authored and

  gatorsmile

  committed
  63b49fa

  Copy full SHA for 63b49fa

• [SPARK-23305][SQL][TEST] Test spark.sql.files.ignoreMissingFiles for all file-based data sources

  Show description for 522e0b1
  

  dongjoon-hyun

  authored and

  gatorsmile

  committed
  522e0b1

  Copy full SHA for 522e0b1

• [MINOR][DOC] Use raw triple double quotes around docstrings where there are occurrences of backslashes.

  Show description for 4aaa7d4
  

  ashashwat

  authored and

  gatorsmile

  committed
  4aaa7d4

  Copy full SHA for 4aaa7d4

• [SPARK-21658][SQL][PYSPARK] Revert "[] Add default None for value in na.replace in PySpark"

  Show description for 551dff2
  

  HyukjinKwon

  authored and

  gatorsmile

  committed
  551dff2

  Copy full SHA for 551dff2

Commits on Feb 4, 2018

• [SPARK-23256][ML][PYTHON] Add columnSchema method to PySpark image reader

  Show description for 715047b
  HyukjinKwon

  committed
  715047b

  Copy full SHA for 715047b

• [SPARK-22036][SQL][FOLLOWUP] Fix decimalArithmeticOperations.sql

  Show description for 6fb3fd1
  

  wangyum

  authored and

  gatorsmile

  committed
  6fb3fd1

  Copy full SHA for 6fb3fd1

Commits on Feb 5, 2018

• [SPARK-23307][WEBUI] Sort jobs/stages/tasks/queries with the completed timestamp before cleaning up them

  Show description for a6bf3db
  

  zsxwing

  authored and

  cloud-fan

  committed
  a6bf3db

  Copy full SHA for a6bf3db

• [SPARK-23310][CORE] Turn off read ahead input stream for unshafe shuffle reader

  Show description for 03b7e12
  

  Sital Kedia

  authored and

  sameeragarwal

  committed
  03b7e12

  Copy full SHA for 03b7e12

• [SPARK-23330][WEBUI] Spark UI SQL executions page throws NPE

  Show description for c2766b0
  

  jiangxb1987

  authored and

  Marcelo Vanzin

  committed
  c2766b0

  Copy full SHA for c2766b0

Commits on Feb 6, 2018

• [SPARK-23326][WEBUI] schedulerDelay should return 0 when the task is running

  Show description for f3f1e14
  

  zsxwing

  authored and

  cloud-fan

  committed
  f3f1e14

  Copy full SHA for f3f1e14

• [SPARK-23290][SQL][PYTHON] Use datetime.date for date type when converting Spark DataFrame to Pandas DataFrame.

  Show description for a24c031
  

  ueshin

  authored and

  cloud-fan

  committed
  a24c031

  Copy full SHA for a24c031

• [SPARK-23300][TESTS] Prints out if Pandas and PyArrow are installed or not in PySpark SQL tests

  Show description for 8141c3e
  HyukjinKwon

  committed
  8141c3e

  Copy full SHA for 8141c3e

• [SPARK-23334][SQL][PYTHON] Fix pandas_udf with return type StringType() to handle str type properly in Python 2.

  Show description for 63c5bf1
  

  ueshin

  authored and

  HyukjinKwon

  committed
  63c5bf1

  Copy full SHA for 63c5bf1

• Merge remote-tracking branch 'apache/master' into ds/sync-upstream

  dansanduleac

  committed
  a56fceb

  Copy full SHA for a56fceb

• checkstyle

  

  Robert Kruszewski

  committed
  c05f7c4

  Copy full SHA for c05f7c4