Code

apache spark latest#15

Code

Merged

rekhajoshm merged 54 commits intorekhajoshm:masterrekhajoshm/spark:masterfrom apache:masterapache/spark:masterCopy head branch name to clipboardJun 30, 2018

+7,182-1,431

Commits

Commits on Jun 21, 2018

• [SPARK-24571][SQL] Support Char literals

  Show description for 54fcaaf
  

  MaxGekk

  authored and

  gatorsmile

  committed
  54fcaaf

  Copy full SHA for 54fcaaf

• [SPARK-24574][SQL] array_contains, array_position, array_remove and element_at functions deal with Column type

  Show description for 7236e75
  

  chongguang

  authored and

  HyukjinKwon

  committed
  7236e75

  Copy full SHA for 7236e75

• [SPARK-24614][PYSPARK] Fix for SyntaxWarning on tests.py

  Show description for c0cad59
  

  rekhajoshm

  authored and

  HyukjinKwon

  committed
  c0cad59

  Copy full SHA for c0cad59

• [SPARK-16630][YARN] Blacklist a node if executors won't launch on it

  Show description for b56e9c6
  

  attilapiros

  authored and

  squito

  committed
  b56e9c6

  Copy full SHA for b56e9c6

• [SPARK-24589][CORE] Correctly identify tasks in output commit coordinator.

  Show description for c8e909c
  

  Marcelo Vanzin

  authored and

  tgravescs

  committed
  c8e909c

  Copy full SHA for c8e909c

• [SPARK-24613][SQL] Cache with UDF could not be matched with subsequent dependent caches

  Show description for b9a6f74
  

  maryannxue

  authored and

  gatorsmile

  committed
  b9a6f74

  Copy full SHA for b9a6f74

• [SPARK-24588][SS] streaming join should require HashClusteredPartitioning from children

  Show description for dc8a6be
  

  cloud-fan

  authored and

  gatorsmile

  committed
  dc8a6be

  Copy full SHA for dc8a6be

Commits on Jun 22, 2018

• [SPARK-23934][SQL] Adding map_from_entries function

  Show description for 92c2f00
  

  mn-mikke

  authored and

  ueshin

  committed
  92c2f00

  Copy full SHA for 92c2f00

• [SPARK-24519] Make the threshold for highly compressed map status configurable

  Show description for 39dfaf2
  

  Hieu Huynh

  authored and

  tgravescs

  committed
  39dfaf2

  Copy full SHA for 39dfaf2

• [SPARK-24518][CORE] Using Hadoop credential provider API to store password

  Show description for 33e77fa
  

  jerryshao

  authored and

  Marcelo Vanzin

  committed
  33e77fa

  Copy full SHA for 33e77fa

• [SPARK-24372][BUILD] Add scripts to help with preparing releases.

  Show description for 4e7d867
  

  Marcelo Vanzin

  authored and

  tgravescs

  committed
  4e7d867

  Copy full SHA for 4e7d867

Commits on Jun 24, 2018

• [SPARK-24190][SQL] Allow saving of JSON files in UTF-16 and UTF-32

  Show description for c7e2742
  

  MaxGekk

  authored and

  gatorsmile

  committed
  c7e2742

  Copy full SHA for c7e2742

• [SPARK-24206][SQL] Improve FilterPushdownBenchmark benchmark code

  Show description for 98f363b
  

  maropu

  authored and

  gatorsmile

  committed
  98f363b

  Copy full SHA for 98f363b

• [SPARK-24324][PYTHON] Pandas Grouped Map UDF should assign result columns by name

  Show description for a5849ad
  

  BryanCutler

  authored and

  HyukjinKwon

  committed
  a5849ad

  Copy full SHA for a5849ad

Commits on Jun 25, 2018

• [SPARK-24327][SQL] Verify and normalize a partition column name based on the JDBC resolved schema

  Show description for f596ebe
  

  maropu

  authored and

  gatorsmile

  committed
  f596ebe

  Copy full SHA for f596ebe

• [SPARK-23931][SQL][FOLLOW-UP] Make arrays_zip in function.scala @scala.annotation.varargs.

  Show description for 6e0596e
  

  ueshin

  authored and

  gatorsmile

  committed
  6e0596e

  Copy full SHA for 6e0596e

• Fix minor typo in docs/cloud-integration.md

  Show description for 8ab8ef7
  

  Jim Kleckner

  authored and

  HyukjinKwon

  committed
  8ab8ef7

  Copy full SHA for 8ab8ef7

• [SPARK-24596][SQL] Non-cascading Cache Invalidation

  Show description for bac50aa
  

  maryannxue

  authored and

  gatorsmile

  committed
  bac50aa

  Copy full SHA for bac50aa

• [SPARK-24633][SQL] Fix codegen when split is required for arrays_zip

  Show description for 594ac4f
  

  mgaido91

  authored and

  cloud-fan

  committed
  594ac4f

  Copy full SHA for 594ac4f

• [SPARK-24648][SQL] SqlMetrics should be threadsafe

  Show description for 5264164
  

  dbkerkela

  authored and

  hvanhovell

  committed
  5264164

  Copy full SHA for 5264164

• [INFRA] Close stale PR.

  Show description for baa01c8
  

  Marcelo Vanzin

  committed
  baa01c8

  Copy full SHA for baa01c8

• [SPARK-24552][CORE][SQL] Use task ID instead of attempt number for writes.

  Show description for 6d16b98
  

  Marcelo Vanzin

  committed
  6d16b98

  Copy full SHA for 6d16b98

Commits on Jun 26, 2018

• [SPARK-24324][PYTHON][FOLLOWUP] Grouped Map positional conf should have deprecation note

  Show description for d48803b
  BryanCutler

  committed
  d48803b

  Copy full SHA for d48803b

• [SPARK-23776][DOC] Update instructions for running PySpark after building with SBT

  Show description for 4c059eb
  

  bersprockets

  authored and

  HyukjinKwon

  committed
  4c059eb

  Copy full SHA for 4c059eb

• [SPARK-24418][BUILD] Upgrade Scala to 2.11.12 and 2.12.6

  Show description for c7967c6
  

  dbtsai

  authored and

  jerryshao

  committed
  c7967c6

  Copy full SHA for c7967c6

• [SPARK-24636][SQL] Type coercion of arrays for array_join function

  Show description for e07aee2
  

  mn-mikke

  authored and

  HyukjinKwon

  committed
  e07aee2

  Copy full SHA for e07aee2

• [SPARK-24658][SQL] Remove workaround for ANTLR bug

  Show description for dcaa49f
  

  wangyum

  authored and

  gatorsmile

  committed
  dcaa49f

  Copy full SHA for dcaa49f

• [SPARK-24423][SQL] Add a new option for JDBC sources

  Show description for 02f8781
  

  dilipbiswal

  authored and

  gatorsmile

  committed
  02f8781

  Copy full SHA for 02f8781

• [SPARK-6237][NETWORK] Network-layer changes to allow stream upload.

  Show description for 16f2c3e
  

  squito

  authored and

  Marcelo Vanzin

  committed
  16f2c3e

  Copy full SHA for 16f2c3e

Commits on Jun 27, 2018

• [SPARK-24659][SQL] GenericArrayData.equals should respect element type differences

  Show description for 1b9368f
  

  rednaxelafx

  authored and

  cloud-fan

  committed
  1b9368f

  Copy full SHA for 1b9368f

• [SPARK-24605][SQL] size(null) returns null instead of -1

  Show description for d08f53d
  

  MaxGekk

  authored and

  cloud-fan

  committed
  d08f53d

  Copy full SHA for d08f53d

• [SPARK-23927][SQL] Add "sequence" expression

  Show description for 2669b4d
  

  wajda

  authored and

  ueshin

  committed
  2669b4d

  Copy full SHA for 2669b4d

• [SPARK-23927][SQL][FOLLOW-UP] Fix a build failure.

  Show description for 9a76f23
  

  ueshin

  authored and

  cloud-fan

  committed
  9a76f23

  Copy full SHA for 9a76f23

• [SPARK-21335][DOC] doc changes for disallowed un-aliased subquery use case

  Show description for a1a64e3
  

  cnZach

  authored and

  cloud-fan

  committed
  a1a64e3

  Copy full SHA for a1a64e3

• [SPARK-24215][PYSPARK][FOLLOW UP] Implement eager evaluation for DataFrame APIs in PySpark

  Show description for 6a0b77a
  

  xuanyuanking

  authored and

  gatorsmile

  committed
  6a0b77a

  Copy full SHA for 6a0b77a

• [SPARK-24446][YARN] Properly quote library path for YARN.

  Show description for 78ecb6d
  

  Marcelo Vanzin

  committed
  78ecb6d

  Copy full SHA for 78ecb6d

• [SPARK-21687][SQL] Spark SQL should set createTime for Hive partition

  Show description for c04cb2d
  

  debugger87

  authored and

  gatorsmile

  committed
  c04cb2d

  Copy full SHA for c04cb2d

• [SPARK-24660][SHS] Show correct error pages when downloading logs

  Show description for 776befb
  

  mgaido91

  authored and

  Marcelo Vanzin

  committed
  776befb

  Copy full SHA for 776befb

• [SPARK-24533] Typesafe rebranded to lightbend. Changing the build downloads path

  Show description for 221d03a
  

  Sanket Chintapalli

  authored and

  Marcelo Vanzin

  committed
  221d03a

  Copy full SHA for 221d03a

• [SPARK-24204][SQL] Verify a schema in Json/Orc/ParquetFileFormat

  Show description for 893ea22
  

  maropu

  authored and

  gatorsmile

  committed
  893ea22

  Copy full SHA for 893ea22

• [SPARK-24553][WEB-UI] http 302 fixes for href redirect

  Show description for c5aa54d
  

  SJKallman

  authored and

  gatorsmile

  committed
  c5aa54d

  Copy full SHA for c5aa54d

Commits on Jun 28, 2018

• [SPARK-24645][SQL] Skip parsing when csvColumnPruning enabled and partitions scanned only

  Show description for bd32b50
  

  maropu

  authored and

  HyukjinKwon

  committed
  bd32b50

  Copy full SHA for bd32b50

• [SPARK-24206][SQL][FOLLOW-UP] Update DataSourceReadBenchmark benchmark results

  Show description for 1c9acc2
  

  maropu

  authored and

  HyukjinKwon

  committed
  1c9acc2

  Copy full SHA for 1c9acc2

• [SPARK-24603][SQL] Fix findTightestCommonType reference in comments

  Show description for 6a97e8e
  

  Fokko Driesprong

  authored and

  HyukjinKwon

  committed
  6a97e8e

  Copy full SHA for 6a97e8e

• [SPARK-24564][TEST] Add test suite for RecordBinaryComparator

  Show description for 5b05966
  

  jiangxb1987

  authored and

  cloud-fan

  committed
  5b05966

  Copy full SHA for 5b05966

• [SPARK-14712][ML] LogisticRegressionModel.toString should summarize model

  Show description for 524827f
  

  jiayue-zhang

  authored and

  holdenk

  committed
  524827f

  Copy full SHA for 524827f

• [SPARK-23120][PYSPARK][ML] Add basic PMML export support to PySpark

  Show description for a95a4af
  holdenk

  committed
  a95a4af

  Copy full SHA for a95a4af

• [SPARK-24408][SQL][DOC] Move abs function to math_funcs group

  Show description for e1d3f80
  

  jaceklaskowski

  authored and

  holdenk

  committed
  e1d3f80

  Copy full SHA for e1d3f80

• [SPARK-24439][ML][PYTHON] Add distanceMeasure to BisectingKMeans in PySpark

  Show description for 2224861
  

  huaxingao

  authored and

  BryanCutler

  committed
  2224861

  Copy full SHA for 2224861

• [SPARK-24386][SS] coalesce(1) aggregates in continuous processing

  Show description for f6e6899
  

  jose-torres

  authored and

  tdas

  committed
  f6e6899

  Copy full SHA for f6e6899

Commits on Jun 29, 2018

• [SPARK-24566][CORE] Fix spark.storage.blockManagerSlaveTimeoutMs default config

  Show description for f71e8da
  

  xueyumusic

  authored and

  zsxwing

  committed
  f71e8da

  Copy full SHA for f71e8da

Commits on Jun 30, 2018

• [SPARK-24638][SQL] StringStartsWith support push down

  Show description for 03545ce
  

  wangyum

  authored and

  cloud-fan

  committed
  03545ce

  Copy full SHA for 03545ce

• [SPARK-24696][SQL] ColumnPruning rule fails to remove extra Project

  Show description for 797971e
  

  maryannxue

  authored and

  gatorsmile

  committed
  797971e

  Copy full SHA for 797971e

• simplify rand in dsl/package.scala

  gatorsmile

  committed
  d54d8b8

  Copy full SHA for d54d8b8