Druid 0.11.0 release notes

[jon-wei]

DRAFT

Druid 0.11.0 contains over a hundred performance improvements, stability improvements, and bug fixes from almost 40 contributors. This release adds two major security features, TLS support and extension points for authentication and authorization.

Other major new features include:

• Double columns support
• cachingCost Balancer Strategy
• jq expression support in JSON parser
• Redis cache extension
• GroupBy performance improvements
• Various improvements to Druid SQL

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.11.0

Documentation for this release is at: http://druid.io/docs/0.11.0/

Highlights

TLS support

Druid now supports TLS, enabling encrypted client and inter-node communications. Please see http://druid.io/docs/0.11.0-rc2/operations/tls-support.html for details on configuration and related extensions.

Added by @pjain1 in #4270.

Authentication/authorization extension points

Extension points for authenticating and authorizing requests have been added to Druid. Please see http://druid.io/docs/0.11.0-rc2/configuration/auth.html for information on configuration and extension implementation.

The existing Kerberos authentication extension has been updated to implement the new Authenticator interface, please see the "Kerberos configuration changes" section under "Updating from 0.10.1 and earlier" for more information if you are using the Kerberos extension.

Added by @jon-wei in #4271

Double columns support

Druid now supports Double type aggregator columns. Please see http://druid.io/docs/0.11.0-rc1/querying/aggregations.html for documentation on the new Double aggregators.

Added by @b-slim in #4491.

cachingCost Balancer Strategy

Users upgrading to 0.11.0 are encouraged to try the new cachingCost segment balancing strategy on their coordinators. This strategy offers large performance improvements over the existing cost balancer strategy, and it is planned to become the default strategy in the release following 0.11.0.

This strategy can be selected by setting the following property on coordinators:

druid.coordinator.balancer.strategy=cachingCost

Added by @dgolitsyn in #4731

jq expression support in JSON parser

Druid's JSON input parser now supports jq expressions using jackson-jq, enabling more input transforms before ingestion. Please see http://druid.io/docs/0.11.0-rc2/ingestion/flatten-json.html for more details.

Added by @knoguchi in #4171.

Redis cache extension

A new cache implementation using Redis has been added in an extension, added by @QiuMM in #4615. Please refer to the preceding pull request for more details.

GroupBy performance improvements

Several new performance optimizations have been added to the GroupBy query by @jihoonson in the following PRs:

#4660 Parallel sort for ConcurrentGrouper
#4576 Array-based aggregation for groupBy query
#4668 Add IntGrouper to avoid unnecessary boxing/unboxing in array-based aggregation

PR #4660 offers a general improvement by parallelizing partial result sorting, while PR #4576 and #4668 offer significant improvements when grouping on a single String column.

SQL improvements

Various improvements and features have been added to Druid SQL, by @gianm in the following PRs:

#4750 - TRIM support
#4720 - Rounding for count distinct
#4561 - Metrics for SQL queries
#4360 - SQL expressions support

And much more!

The full list of changes is here: https://github.com/druid-io/druid/pulls?utf8=%E2%9C%93&q=is%3Apr%20is%3Aclosed%20milestone%3A0.11.0

Updating from 0.10.1 and earlier

Please see below for changes between 0.10.1 and 0.11.0 that you should be aware of before upgrading. If you're updating from an earlier version than 0.10.1, please see release notes of the relevant intermediate versions for additional notes.

Upgrading coordinators and overlords

The following patch changes the way coordinator->overlord redirects are handled:
#5037

The overlord leader election algorithm has changed in 0.11.0: #4699.

As a result of the two patches above, special care is needed when upgrading Coordinator or Overlord to 0.11.0. All coordinators and overlords must be shut down and upgraded together.

For example, to upgrade Coordinators, you would shutdown all coordinators, upgrade them to 0.11.0 and then start them. Overlords should be upgraded in a similar way.

During the upgrade process, there must not be any time period where a non-0.11.0 coordinator or overlord is running simultaneously with an 0.11.0 coordinator or overlord.

Note that at least one overlord should be brought up as quickly as possible after shutting them all down so that peons, tranquility etc continue to work after some retries.

Also note that the druid.zk.paths.indexer.leaderLatchPath property is no longer used now.

Service name changes

In earlier versions of Druid, / characters in service names defined by druid.service would be replaced by : characters because these service names were used in Zookeeper paths. Druid 0.11.0 no longer performs these character replacements.

Example:1 - if the old configuration had a broker with service name test/broker:
druid.service=test/broker

and a Router was configured assuming that / will be replaced with : in the broker service name,
druid.router.tierToBrokerMap={"hot":"test:broker","_default_tier":"test:broker"}

the Router configuration should be updated to remove that assumption:
druid.router.tierToBrokerMap={"hot":"test/broker","_default_tier":"test/broker"}

Example:2 - If the old configuration had overlord with service Name test/overlord then value of druid.coordinator.asOverlord.overlordService or druid.selectors.indexing.serviceName should be test/overlord and not test:overlord

Example:3 - If the old configuration had overlord with service Name test:overlord then value of druid.coordinator.asOverlord.overlordService or druid.selectors.indexing.serviceName should be test:overlord and not test/overlord

Following service name-related configurations are also affected and should be updated to exactly match the value of druid.service property on other node being discovered.

druid.coordinator.asOverlord.overlordService
druid.selectors.coordinator.serviceName
druid.selectors.indexing.serviceName
druid.router.defaultBrokerServiceName
druid.router.coordinatorServiceName
druid.router.tierToBrokerMap

Please see #4992 for more details.

Kerberos configuration changes

The Kerberos authentication configuration format has changed as a result of the new interfaces introduced by #4271. Please refer to http://druid.io/docs/0.11.0-rc2/development/extensions-core/druid-kerberos.html for the new configuration properties.

Users can point the Kerberos authenticator's authorizerName to an instance of an "allowAll" authorizer to replicate the pre-0.11.0 behavior of a cluster using Kerberos authentication with no authorization.

Lookups API path changes

The paths for the lookups configuration API have changed due to #5058.

Configuration paths that had the form /druid/coordinator/v1/lookups now have the form /druid/coordinator/v1/lookups/config.

Please see http://druid.io/docs/0.11.0-rc2/querying/lookups.html for the current API.

Migrating to Double columns

Prior to 0.11.0, the Double* aggregators would store column values on disk as Float while performing aggregations using Double representations.

PR #4491 allows the Double aggregators to store column values on disk as Doubles. Due to concerns related to rolling updates and version downgrades, this behavior is disabled by default and Druid will continue to store Double aggregators on disk as floats.

To enable Double column storage, set the following property in the common runtime properties:

druid.indexing.doubleStorage=double

Users should not set this property during an initial rolling upgrade to 0.11.0, as any nodes running pre-0.11.0 Druid will not be able to handle Double columns created during the upgrade period. Users will also need to reindex any segments with Double columns if downgrading from 0.11.0 to an older version. Please see #4944 and #4605 for more information.

Scan query changes

The Scan query has been moved from extensions-contrib to core Druid. As part of this migration: #4751, the scan query's handling of the time column has changed.

The time column is now is returned as "__time" rather than "timestamp", it is no longer included if you do not specifically ask for it in your "columns", and it is returned as a long rather than a string.

Users can revert the Scan query's time handling to the legacy extension behavior by setting "legacy" : true in their queries, or setting the property druid.query.scan.legacy = true. This is meant to provide a migration path for users that were formerly using the contrib extension.

Extension Interface Changes

Aggregator double column support

The Aggregator interface has gained a getDouble() method, which defaults to casting the result of getFloat(). The getDouble() method should be re-implemented for any custom aggregators that can support doubles.

See #4595 for more details.

QueryRunner interface change

The QueryRunner interface has changed and the old run() method has been removed, replaced by a new method that accepts a QueryPlus object.

Custom query extensions will need to implement the new interface.

Please see #4184 and #4482 for more details.

Filter interface change

The Filter.getBitmapResult() method no longer has a default implementation: #4481

Custom filter extensions will need to provide an implementation for getBitmapResult() now.

Other Notes

jvm/gc/time metric

The jvm/gc/time metric is no longer emitted, replaced by a new metric named jvm/gc/cpu for the reasons described here: #4480

Default worker select strategy

Please note that the default worker select strategy has changed from fillCapacity to equalDistribution. This change was introduced in 0.10.1, the previous release, but was not mentioned in the 0.10.1 release notes, so it is called out again here.

V8 segment creation removed

Druid will now always build V9 segments, creating V8 segments is no longer supported and the buildV9Directly property for ingestion tasks has been removed.

Please see #4420 for more details.

LogLevelAdjuster removed

Please note that the LogLevelAdjuster has been removed: #4236

Any user using mbeans to configure log levels should configure log4j2 using jmx instead.

Credits

Thanks to everyone who contributed to this release!

@a2l007
@akashdw
@Andy256
@asifmansoora
@b-slim
@benvogan
@blugowski
@chrisgavin
@dclim
@dgolitsyn
@drcrallen
@egor-ryashin
@erikdubbelboer
@Fokko
@fuji-151a
@gaodayue
@gianm
@ginoledesma
@himanshug
@hzy001
@jihoonson
@jon-wei
@kevinconaway
@knoguchi
@leiwangx
@leventov
@michalmisiewicz
@niketh
@pjain1
@praveev
@QiuMM
@scan-the-automator
@solimant
@SpotXPeterCunningham
@tkyaw
@wywlds
@xanec
@yuusaku-t
@zhangxinyu1

[gianm]

Some suggestions:

• Double check for any PRs that landed in the 0.11.0 branch but are not marked as 0.11.0. Since the recent decision to move to time-based releases, we have not been necessarily marking all PRs with the milestone (just "important/blocker" ones). So the best way to find them is trawling the git commit history for the 0.11.0 branch.
• Put TLS & Authentication/authorization extension points next to each other as the first two points, and insert some language in the preamble ("Major new features include…") about how this release is a big deal from a security perspective.
• The "Updating from 0.10.1" section should have something about the setting introduced in Make float default representation for DoubleSum/Min/Max aggregators #4944.
• The "Updating from 0.10.1" section is scarily long, but many of the points are actually not that big of a deal. I'd suggest keeping the ones that are a big deal in their own subsections (like they are now) but grouping together all the minor ones into an "other" subsection. Also group together all the ones that only apply to extension authors.
• There's a "Placeholder" in the notes for some reason.

[leventov]

Suggested to mention the new balancer strategy as druid.coordinator.balancer.strategy=cachingCost, because otherwise it's not very easy to find in docs how to configure it

[gianm]

Btw, for "Authentication/authorization extension points" I think it'd be good to add language making it clear that this version does include one implementation (Kerberos) with a link to instructions on how to use that one.

[Igosuki]

Hey guys, for us one of the biggest niceties from this version is the upgraded core extensions such as this one #4832 I think it's worth mentioning since many people actually have nested schemas.

[SpotXPeterCunningham]

I second @Igosuki. The nested Avro support is excellent for us and performs really well, we would be stuck without it. Will it make it make it into 11.0?

[gianm]

I think #4832 didn't make it to 0.11.0, but will be in 0.11.1 (or whatever the version after 0.11.0 is).

[l15k4]

If anyone is interested in Prometheus integration, it will be possible with Graphite Emitter #4265 plain text protocol support.

Then changing port from graphite 2004 to 9109 of https://github.com/prometheus/graphite_exporter which makes it available for prometheus...

[himanshug]

@jon-wei @leventov

I see mention of removal of druid.segmentCache.numLoadingThreads config . It was removed because it wasn't ever utilized and batch load/drops were not supported.
However this config is brought back in 0.11.1 with #4966 that introduces batch support for http based segment management.

I think we can omit any information related to this config in 0.11.0 release notes as users aren't required to take any action. We can put some notes about this config in next release and its intended behavior as documented in #4997 .

[jon-wei]

@himanshug thanks, I removed that section