Docusaurus3 mdx syntax #29
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* remove legacy SQL incompatible modes changes: * druid.generic.useDefaultValueForNull is no longer wired up to anything and will log.warn if set to true * druid.generic.useThreeValueLogicForNativeFilters is no longer wired up to anything and will log.warn if set to false * clean up and simplify all code and tests related to null handling configs * fixes * fix * fix IT
This PR adds the ability to attach restrictions (e.x. row filters) to a query, thus restrict row-level data access for given users. Note that this feature is not directly available, and requires additional work in Authorizer. This PR has been tested in a local druid cluster with additional Authorizer changes. Description A query follows these steps: initialize -> authorize -> execute. In the authorize step, the permissions are checked for all the required resources in the query. Before this PR, the authorize step only returns allow or deny access on a table. Granting access to a table means a user can see all data in this table. After this PR, the authorize step can return allow access along with restrictions (i.e. a row filter that must be applied to the table ), which restrict users' data access at row level. For example, customers can only see rows relevant to their company. The authorizeAllResourceActions now returns a AuthorizationResult instead of Access. The main difference between AuthorizationResult and Access is that the former contains a map of table with restriction (i.e. Policy) In the authorize step of QueryLifecycle, it would add the Policy to tables in the datasource tree, transform TableDataSource to RestrictedDataSource. In the execute step, Policy is enforced through RestrictedSegment and RestrictedCursorFactory, the filter inside Policy would be attached to the CursorHolder.
* new Explore layout * fix feedback comments
Instead of running the unit tests first and then running ITs, and other JDK tests, we will rather run all in parallel for faster build time. This is done only when the PR is approved so we are still not wasting gha runner bandwidth.
Despite actually being minor problems, these are all at "error" (high sev) level, so fixing them will make it easier to raise future patches without tripping into CodeQL failures.
* fix supervisor pagiantion * fix time chart bug * update snapshot * fix supervisor hisotry tabs
* update docs after legacy configs have been removed * better runtime.properties validation with new StartupInjectorBuilder.PropertiesValidator to fail fast if removed properties are set
* move druid-ranger-security from core to contrib * fix package to be contrib
1. adds DruidRelFieldTrimmer 2. Unnest now uses unnestFieldType instead of blindly taking the rowType 3. removed the possibly problematic column reuse in UnnestCleanupRule (the trimmer should take care of that) 4. lots of iq file changes because earlier no proper trimming was done below LogicalCorrelate and Unnest -s
Noticed a stale reference to Java 8 while doing some other stuff
* make k8s ingestion core * add redirects * missing licenses * Add disclaimer for druid 28 * Fix redirect
Code coverage additions for Processing Directory: HttpPostEmitterMonitorTest and JavaScriptConfigTest
Code coverage additions for Cloud: AWSCredentialsConfigTest
… when inverted (apache#17655) Fixes a bug with the OR filters vectorized value matcher that causes vector engines processing filter an OR filter under a NOT filter (ex. of the form NOT(x OR y)) to produce incorrect results for null values matched. This bug is due to incorrectly hard coding the includeUnknown parameter as false for OR filter child vector matchers after the initial filter clause instead of passing it through the function parameter to the underlying matchers.
ZK-based segment loading was completely disabled in apache#16360 and apache#15705. This PR removes all related code which is now unused. Changes: - Remove CuratorLoadQueuePeon and CuratorLoadQueuePeonTest - Refactor DruidCoordinatorTest to use a TestLoadQueuePeon instead
* docs: add examples for SQL functions (apache#16745) * updating first batch of numeric functions * First batch of functions * addressing first few comments * alphabetize list * draft with suggestions applied * minor discrepency expr -> <NUMERIC> * changed raises to calculates * Update docs/querying/sql-functions.md * switch to underscore * changed to exp(1) to match slack message * adding html text for trademark symbol to .spelling * fixed discrepancy between description and example --------- Co-authored-by: Benedict Jin <[email protected]> (cherry picked from commit 721a650) * [docs] batch02 of updating functions (apache#16761) * applying changes * ensuring batch is updated * Update docs/querying/sql-functions.md * raise -> raises * addressing review * Apply suggestions from code review Co-authored-by: Charles Smith <[email protected]> --------- Co-authored-by: Benedict Jin <[email protected]> Co-authored-by: Charles Smith <[email protected]> (cherry picked from commit ca78788) * [Docs] batch 03 - trig functions (apache#16795) * batch 03 - trig functions * Apply suggestions from code review Co-authored-by: Charles Smith <[email protected]> * applying suggestions and corrections --------- Co-authored-by: Charles Smith <[email protected]> (cherry picked from commit 028ee23) * [Docs]Batch04 - Bitwise numeric functions (apache#16805) * Batch04 - Bitwise numeric functions * Batch04 - Bitwise numeric functions * minor fixes * rewording bitwise_shift functions * rewording bitwise_shift functions * Update docs/querying/sql-functions.md * applying suggestions --------- Co-authored-by: Benedict Jin <[email protected]> (cherry picked from commit 85a8a1d) * [docs] batch 5 updating functions (apache#16812) * batch 5 * Update docs/querying/sql-functions.md * applying suggestions --------- Co-authored-by: Benedict Jin <[email protected]> (cherry picked from commit 3bb6d40) * [Docs] Batch06: starting string functions (apache#16838) * batch06, starting string functions * addind space after Syntax * quick change * correcting spelling * Update docs/querying/sql-functions.md * Update sql-functions.md * applying suggestions * Update docs/querying/sql-functions.md * Update docs/querying/sql-functions.md --------- Co-authored-by: Benedict Jin <[email protected]> Co-authored-by: Charles Smith <[email protected]> (cherry picked from commit ebea34a) * [Docs] Batch08: adding examples to string functions (apache#16871) * batch08 completed * reviewing batch08 * apply corrections suggestions by @FrankChen021 (cherry picked from commit 5b94839) * [Docs] Batch07: adding examples to string functions (apache#16862) * Lower,Upper,Lpad,Rpad,Parse_long * up to REGEXP_EXTRACT * batch 07 ready for review * updated definitions in scalar * Apply suggestions from code review Co-authored-by: Charles Smith <[email protected]> * rpad and lpad * addressing comments * minor fixes * improving examples based on suggestions * matched -> matches * correcting typo * Apply suggestions from code review Co-authored-by: Charles Smith <[email protected]> --------- Co-authored-by: Charles Smith <[email protected]> (cherry picked from commit 7256953) * [Docs] Batch09: only `lookup` (apache#16878) * [Docs] Batch09: only `lookup` * slight changes * Apply suggestions from code review Co-authored-by: Katya Macedo <[email protected]> * applying suggestiontions * Apply suggestions from code review Co-authored-by: Victoria Lim <[email protected]> * otherwise null -> otherwise returns null * updating definition in sql-scalar.md * Apply suggestions from code review Co-authored-by: Charles Smith <[email protected]> * hoping to re-run web checks * change replaceMissingValueWith -> defaultValue * Update docs/querying/sql-scalar.md Co-authored-by: Katya Macedo <[email protected]> * acronym_to_name -> airportcode_to_name * shortens `airportcode_to_name` to `code_to_name` --------- Co-authored-by: Katya Macedo <[email protected]> Co-authored-by: Victoria Lim <[email protected]> Co-authored-by: Charles Smith <[email protected]> (cherry picked from commit fda2d19) * [docs] Batch10 date and time functions (apache#16900) * just starting * TIME_PARSE and TIME_FORMAT remaining * fixing typo * adding last two functions * review sql-functions.md * Apply suggestions from code review Suggestions that were accepted as is Co-authored-by: Katya Macedo <[email protected]> * Update docs/querying/sql-functions.md Co-authored-by: Katya Macedo <[email protected]> * Update docs/querying/sql-functions.md needed to confirm that it did indeed return as a number Co-authored-by: Katya Macedo <[email protected]> * reviewing remaining suggestions * addressing review for time_format * Apply suggestions from code review Accepted as is Co-authored-by: Katya Macedo <[email protected]> * addressing final suggestion * time_zone -> timezone * timezone fix --------- Co-authored-by: Katya Macedo <[email protected]> (cherry picked from commit c4981e3) * [docs] batch 12: reduction functions (apache#16930) * [docs] batch 12: reduction functions * Update docs/querying/sql-functions.md * Update docs/querying/sql-functions.md * Update docs/querying/sql-functions.md * applying suggestions * Apply suggestions from code review Co-authored-by: Katya Macedo <[email protected]> --------- Co-authored-by: Benedict Jin <[email protected]> Co-authored-by: Katya Macedo <[email protected]> (cherry picked from commit c49dc83) * [docs] Batch13 IP functions (apache#16947) * new datasource * reviewing before pr * Update docs/querying/sql-functions.md * Apply suggestions from code review Co-authored-by: Katya Macedo <[email protected]> * Apply suggestions from code review Co-authored-by: Katya Macedo <[email protected]> * Apply suggestions from code review Co-authored-by: Charles Smith <[email protected]> * Applying suggestions to IPV4_PARSE --------- Co-authored-by: Benedict Jin <[email protected]> Co-authored-by: Katya Macedo <[email protected]> Co-authored-by: Charles Smith <[email protected]> (cherry picked from commit ed81126) * [docs] Batch11 date and time functions (apache#16926) * first draft of functions * minor improvments * Update docs/querying/sql-functions.md * Update docs/querying/sql-scalar.md * Apply suggestions from code review Accepted as is Co-authored-by: Katya Macedo <[email protected]> * applying next round of suggestions * fixing missing column name * addressing floor and ceil functions * Apply suggestions from code review Co-authored-by: Katya Macedo <[email protected]> * Apply suggestions from code review Co-authored-by: Katya Macedo <[email protected]> * re-wording TIMESTAMPADD --------- Co-authored-by: Benedict Jin <[email protected]> Co-authored-by: Katya Macedo <[email protected]> (cherry picked from commit 2d9e92c) * Update docs/querying/sql-functions.md * Update docs/querying/sql-functions.md Co-authored-by: Benedict Jin <[email protected]> * [docs] Batches 14-16, 18: HLL, Theta, Quantiles, other (apache#93) Co-authored-by: Katya Macedo <[email protected]> Co-authored-by: edgar2020 <[email protected]> Co-authored-by: Katya Macedo <[email protected]> Co-authored-by: Katya Macedo <[email protected]> Co-authored-by: Charles Smith <[email protected]> * batches 20 21 24 25 * fix unnest list * Add LISTAGG to spelling * cherry pick batch 21 * cherry pick batch 21 --------- Co-authored-by: Edgar Melendrez <[email protected]> Co-authored-by: Edgar Melendrez <[email protected]> Co-authored-by: Benedict Jin <[email protected]> Co-authored-by: Katya Macedo <[email protected]> Co-authored-by: Katya Macedo <[email protected]> Co-authored-by: Charles Smith <[email protected]>
…s instead of using capabilities (apache#17662) * fix search query to check for the specific type of indexes it requires instead of using capabilities * cleanup * add comment
…pache#17664) * fix missing required columns of CursorBuildSpec from RowFilterPolicy changes: * RowFilterPolicy.visit now uses new method CursorBuildSpecBuilder.andFilter to re-use the CursorBuildSpec transform of FilteredCursorFactory of adding a filter and its required columns to a CursorBuildSpec * Added javadoc to CursorFactory, CursorHolder, and CursorBuildSpec to clarify usage * Added tests for interesting CursorBuildSpec and CursorBuildSpecBuilder methods
* Query with timezone * Update web-console/src/views/explore-view/explore-view.tsx Co-authored-by: John Gozde <[email protected]> --------- Co-authored-by: John Gozde <[email protected]>
* remove json column v4 format writers * fixes * fix test * log.warn on invalid nested column format version instead of failure
* [docs] Batch26 JSON functions * Updated * Updated * Updated * Fixed typo * Small wording update * Updated after review --------- Co-authored-by: Victoria Lim <[email protected]>
…h23) (apache#17670) Co-authored-by: Victoria Lim <[email protected]>
changes: * introduces FilteredInputBinding which adds better conditional expression processing support using a VectorMatch internally to selectively evaluate input vectors instead of precomputing all inputs, with nvl updated to take advantage of this * refactor some stuff to streamline expression vector processor implementation for simple functions like most math and logical operations with some new factory classes * update vector identifier expression processor to delegate evaluating results directly to the input binding selectors with ExprEvalBindingVector * add maxVectorSize() to ExprVectorProcessor to avoid having to pass max vector size around everywhere
This property allows interval-based limiting of the number of segments killed, as well as the already-existing segment-count-based limiting. This is useful because the kill task retrieves payloads of all active segments that fall within the kill interval, which can be a large number. The new default is 30 days, which seems reasonable for common situations.
…che#17613) Speeds up analysis of functions with large numbers of arguments, such as CASE statements with many branches. The prior code would call "with" for each argument on the accumulated analysis so far, which needlessly re-created the sets of variables over and over.
…ache#17686) PR apache#16849 changed the behavior such that maxIngestedEventTime is not updated for non-real-time data. This patch restores the old behavior for non-real-time data by using a TimeBoundaryInspector when MaxIngestedEventTimeInspector is not present.
This PR adds 34 sets of QA window function tests to quidem. These tests cover window functions in a comprehensive way. Each test has a mixture of positive tests and negative tests. There are a total of 23072 queries in these tests. They use the existing kttm dataset.
* Various fixes for large columns. This patch fixes a class of bugs where various primitive column readers were not providing a SmooshedFileMapper to GenericIndexed, even though the corresponding writer could potentially write multi-file columns. For example, apache#7943 is an instance of this bug. This patch also includes a fix for an issue on the writer for compressed multi-value string columns, V3CompressedVSizeColumnarMultiIntsSerializer, where it would use the same base filename for both the offset and values sections. This bug would only be triggered for segments in excess of 500 million rows. When a segment has fewer rows than that, it could potentially have a values section that needs to be split over multiple files, but the offset is never more than 4 bytes per row. This bug was triggered by the new tests, which use a smaller fileSizeLimit. * Use a Random seed. * Remove erroneous test code. * Fix two compilation problems. * Add javadocs. * Another javadoc.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #XXXX.
Description
Fixed the bug ...
Renamed the class ...
Added a forbidden-apis entry ...
Release note
Key changed/added classes in this PR
MyFooOurBarTheirBazThis PR has: