[SQL] More aggressive defaults

[marmbrus]

• Turns on compression for in-memory cached data by default
• Changes the default parquet compression format back to gzip (we have seen more OOMs with production workloads due to the way Snappy allocates memory)
• Ups the batch size to 10,000 rows
• Increases the broadcast threshold to 10mb.
• Uses our parquet implementation instead of the hive one by default.
• Cache parquet metadata by default.

[marmbrus]

/cc @liancheng @mateiz

[aarondav]

nit: comment seems slightly out of date now

[SparkQA]

Test build #22781 has started for PR 3064 at commit 97ee9f8.

• This patch merges cleanly.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22779/
Test FAILed.

[AmplabJenkins]

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22780/
Test FAILed.

[SparkQA]

Test build #506 has started for PR 3064 at commit 97ee9f8.

• This patch merges cleanly.

[SparkQA]

Test build #506 has finished for PR 3064 at commit 97ee9f8.

• This patch fails Spark unit tests.
• This patch merges cleanly.
• This patch adds no public classes.

[SparkQA]

Test build #22781 has finished for PR 3064 at commit 97ee9f8.

• This patch passes all tests.
• This patch merges cleanly.
• This patch adds the following public classes (experimental):
  • abstract class GenericStrategy[PhysicalPlan <: TreeNode[PhysicalPlan]] extends Logging
  • trait RunnableCommand extends logical.Command
  • case class ExecutedCommand(cmd: RunnableCommand) extends SparkPlan
  • protected case class Keyword(str: String)
  • sys.error(s"Failed to load class for data source: $provider")
  • case class EqualTo(attribute: String, value: Any) extends Filter
  • case class GreaterThan(attribute: String, value: Any) extends Filter
  • case class GreaterThanOrEqual(attribute: String, value: Any) extends Filter
  • case class LessThan(attribute: String, value: Any) extends Filter
  • case class LessThanOrEqual(attribute: String, value: Any) extends Filter
  • trait RelationProvider
  • abstract class BaseRelation
  • abstract class TableScan extends BaseRelation
  • abstract class PrunedScan extends BaseRelation
  • abstract class PrunedFilteredScan extends BaseRelation

[AmplabJenkins]

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/22781/
Test PASSed.

[liancheng]

This LGTM.

(Why does MiMA complain about interfaces introduced by the foreign data source API here?)

[marmbrus]

Thanks for looking at this!

I don't think that's MIMA, as that would actually fail the build (and is
turned off for Spark SQL). The PR notification about new classes is just
based on some rough git magic / string matching as far as I know.
On Nov 2, 2014 7:19 PM, "Cheng Lian" [email protected] wrote:

This LGTM.

(Why does MiMA complain about interfaces introduced by the foreign data
source API here?)

—
Reply to this email directly or view it on GitHub
#3064 (comment).