diff --git a/.github/workflows/notify-docs-update.yaml b/.github/workflows/notify-docs-update.yaml index 261c2d02d1..ab4e53085f 100644 --- a/.github/workflows/notify-docs-update.yaml +++ b/.github/workflows/notify-docs-update.yaml @@ -13,4 +13,4 @@ jobs: steps: - name: Send dispatch event run: | - curl -XPOST -u "${{ secrets.PAT_USERNAME }}:${{ secrets.PAT_TOKEN }}" -H "Accept: application/vnd.github.v3+json" -H "Content-Type: application/json" https://api.github.com/repos/brimdata/zed-docs-site/dispatches --data '{"event_type":"zed-docs-update"}' + curl -XPOST -u "${{ secrets.PAT_USERNAME }}:${{ secrets.PAT_TOKEN }}" -H "Accept: application/vnd.github.v3+json" -H "Content-Type: application/json" https://api.github.com/repos/brimdata/superdb-website/dispatches --data '{"event_type":"super-docs-update", "client_payload": {"commit_sha": "${{ github.event.pull_request.merge_commit_sha }}"}}' diff --git a/CHANGELOG.md b/CHANGELOG.md index b375ff021b..30599d74db 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -3,10 +3,10 @@ * Improve the performance of the [`load` operator](docs/language/operators/load.md) by removing an unnecessary/inefficient merge (#5200) * Improve the [`sort` operator](docs/language/operators/sort.md) to allow different ordering for each key (#5203, #5262) * Update the [Zeek reference shaper docs](docs/integrations/zeek/shaping-zeek-json.md#reference-shaper-contents) to incorporate changes for [Zeek v7.0.0](https://github.com/zeek/zeek/releases/tag/v7.0.0) logs (#5212) -* Update the [`summarize` operator docs](docs/language/operators/summarize.md) to show the use of `by` without an [aggregate function](docs/language/aggregates/README.md) (#5216) +* Update the [`summarize` operator docs](docs/language/operators/summarize.md) to show the use of `by` without an [aggregate function](docs/language/aggregates/_index.md) (#5216) * Update the [`grok` function docs](docs/language/functions/grok.md) with additional examples and guidance (#5243) * Update the [Lateral Subquery docs](docs/language/lateral-subqueries.md) with an emphasis on when primitive values or arrays are returned by [Lateral Expressions](docs/language/lateral-subqueries.md#lateral-expressions) (#5264) -* The terms "pipeline" and "branch" are now used throughout the [Zed docs](docs/README.md) instead of "dataflow" and "leg" (#5272) +* The terms "pipeline" and "branch" are now used throughout the [Zed docs](docs/_index.md) instead of "dataflow" and "leg" (#5272) * Add docs for [`lake` output format](docs/commands/super.md#superdb-data-lake-metadata-output) and [`zed ls`](docs/commands/super-db.md#ls) (#5187) * Add docs for the [`top` operator](docs/language/operators/top.md) (#5276) * Add [`fluentd` integration docs](docs/integrations/fluentd.md) (#5190, #5195) @@ -22,13 +22,13 @@ * Add the `-pool` flag to [`zed manage`](docs/commands/super-db.md#manage) (#5164) * Fix an issue where the lake API was not providing query descriptions for Zed programs that contain scopes (#5152) * Fix an issue where attempts to use the [`load` operator](docs/language/operators/load.md) in `zq` caused a panic (#5162) -* Fix a parser issue with collisions between the names of [user-defined operators](docs/language/statements.md#operator-statements) and [functions](docs/language/statements.md#func-statements) and some built-in [operators](docs/language/operators/README.md) (#5161) +* Fix a parser issue with collisions between the names of [user-defined operators](docs/language/statements.md#operator-statements) and [functions](docs/language/statements.md#func-statements) and some built-in [operators](docs/language/operators/_index.md) (#5161) * Fix an issue where using `null` values in math caused a panic (#5163) ## v1.16.0 * Improve ZNG scanning performance (#5101, #5103) * Improve the error message shown when `zq` is invoked with a single argument that's not a valid query and doesn't contain a source (#5119) -* Update [Zeek integration docs](docs/integrations/zeek/README.md), including [reference shaper](docs/integrations/zeek/shaping-zeek-json.md) changes for [Zeek v6.2.0](https://github.com/zeek/zeek/releases/tag/v6.2.0) data (#5106) +* Update [Zeek integration docs](docs/integrations/zeek/_index.md), including [reference shaper](docs/integrations/zeek/shaping-zeek-json.md) changes for [Zeek v6.2.0](https://github.com/zeek/zeek/releases/tag/v6.2.0) data (#5106) * [String literals](docs/language/expressions.md#formatted-string-literals) now use the "f-string" format `f"{ }"` instead of the previous `${ }` (#5123) * Prototype SQL support has been dropped from the Zed language (full SQL support is expected at a later date) (#5109) * Empty objects and arrays in JSON output are now consistently printed on a single line (#5127) @@ -150,7 +150,7 @@ ## v1.7.0 * Add [`regexp_replace()`](docs/language/functions/regexp_replace.md) function for replacing regular expression matches in a string (#4435, #4449) -* Add [documentation](docs/integrations/zed-lake-auth.md) showing how to configure Auth0 for authenticated access to a Zed lake service (#4439) +* Add [documentation](docs/integrations/zed-lake-auth/index.md) showing how to configure Auth0 for authenticated access to a Zed lake service (#4439) * Fix an issue where elements of map could not be accessed if the key was of a union type (#4447) * Allow [`head`](docs/language/operators/head.md) operator to accept an expression (#4451) * Allow [`tail`](docs/language/operators/tail.md) operator to accept an expression (#4464) @@ -241,8 +241,8 @@ ## v1.0.0 -* Comprehensive [documentation](docs/README.md) -* Substantial improvements to the [Zed language](docs/language/README.md) +* Comprehensive [documentation](docs/_index.md) +* Substantial improvements to the [Zed language](docs/language/_index.md) * Revamped [`zed` command](docs/commands/super-db.md) * New Zed lake format (see #3634 for a migration script) * New version of the [ZNG format](docs/formats/bsup.md) (with read-only support for the previous version) @@ -264,7 +264,7 @@ * `zapi`, `zed lake serve`: Add authentication with Auth0 (#3266) * Fix an issue preventing casting from `ip` to `ip` (#3259) * `zed lake serve`: Respect the Accept request header for `GET /events` (#3246) -* Add [function documentation](docs/language/functions/README.md) (#3215) +* Add [function documentation](docs/language/functions/_index.md) (#3215) * `zed lake serve`: Change the default response content encoding to ZSON (#3242) * `zapi load`, `zed lake load`: Add the `-meta` flag to embed custom metadata in commits (#3237) @@ -280,7 +280,7 @@ * Fix a panic when compiling `SELECT ... GROUP BY ...` (#3193) * Fix a bug in which data loaded through the Zed lake service was stored uncompressed (#3198) * Add all lake index commands to Zed lake service (#3181) -* Reorganize [language documentation](docs/language/README.md) (#3187) +* Reorganize [language documentation](docs/language/_index.md) (#3187) * Make `fuse()` output deterministic (#3190) * Use lake indexes to speed up queries (#3158) * Fix bug where constants blocked `from` operator wiring logic (#3185) @@ -329,7 +329,7 @@ As you can see below, there's been many changes since the last Zed GA release! * Enhancements to the Zed language to unify search and expression syntax, introduce new operators and functions for data exploration and shaping, and more! Review the - [Zed language docs](docs/language/README.md) + [Zed language docs](docs/language/_index.md) for details. The exhaustive set of changes is listed below. Come talk to us on @@ -454,7 +454,7 @@ questions. * Fix an issue where `len()` of a `null` array was evaluating to something greater than zero (#2761) * Fix an issue where `sort` with no fields was ignoring alias types and nested fields when picking a sort field (#2762) * Fix an issue where unexpected `cut: no record found` warnings were returned by `zed lake query` but not when the same data was queried via `zq` (#2764) -* Move and extend the [Zeek interoperability docs](docs/integrations/zeek/README.md) (#2770, #2782, #2830) +* Move and extend the [Zeek interoperability docs](docs/integrations/zeek/_index.md) (#2770, #2782, #2830) * Create endpoints in the Zed lake service API that correspond to underlying Zed lake operations, and expose them via `zapi` commands (#2741, #2774, #2786, #2775, #2794, #2795, #2796, #2920, #2925, #2928) * Fix an issue where `zq` would surface a syntax error when reading ZSON it had sent as output (#2792) * Add an `/events` endpoint to the API, which can be used by clients such as the Brim app to be notified of pool updates (#2791) @@ -469,8 +469,8 @@ questions. * Have the [Python client](python) use the `/query` endpoint for the Zed lake (#2869) * Minimize the amount of surrounding context shown when reporting parse errors (#2864) * Field assignments in [`join`](docs/language/operators/join.md) now behave like [`cut`](docs/language/operators/cut.md) instead of `pick` (#2868) -* Add more background/context to Zed top-level language [README](docs/language/README.md) (#2866 #2878, #2901) -* Unify `from`, `split`, and `switch` syntax to the forms shown [here](docs/language/README.md) (#2871, #2896) +* Add more background/context to Zed top-level language [README](docs/language/_index.md) (#2866 #2878, #2901) +* Unify `from`, `split`, and `switch` syntax to the forms shown [here](docs/language/_index.md) (#2871, #2896) * Shapers can now cast values of the `null` type to any type (e.g., arrays or records) (#2882) * Fix an issue where [`join`](docs/language/operators/join.md) was failing to match on values of comparable types (e.g., `string` and `bstring`) (#2880, #2884) * Shapers can now cast a value to a `union` type (#2881) @@ -628,7 +628,7 @@ questions. ## v0.23.0 * zql: Add `week` as a unit for [time grouping with `every`](docs/language/functions/every.md) (#1374) -* zq: Fix an issue where a `null` value in a [JSON type definition](docs/integrations/zeek/README.md) caused a failure without an error message (#1377) +* zq: Fix an issue where a `null` value in a [JSON type definition](docs/integrations/zeek/_index.md) caused a failure without an error message (#1377) * zq: Add [`zst` format](docs/formats/csup.md) to `-i` and `-f` command-line help (#1384) * zq: ZNG spec and `zq` updates to introduce the beta ZNG storage format (#1375, #1415, #1394, #1457, #1512, #1523, #1529), also addressing the following: * New data type `bytes` for storing sequences of bytes encoded as base64 (#1315) @@ -644,11 +644,11 @@ questions. * zqd: Check and convert alpha ZNG filestores to beta ZNG (#1574, #1576) * zq: Fix an issue where spill-to-disk file names could collide (#1391) * zq: Allow the [`fuse` operator](docs/language/operators/fuse.md) to spill-to-disk to avoid memory limitations (#1355, #1402) -* zq: No longer require `_path` as a first column in a [JSON type definition](docs/integrations/zeek/README.md) (#1370) +* zq: No longer require `_path` as a first column in a [JSON type definition](docs/integrations/zeek/_index.md) (#1370) * zql: Improve ZQL docs for [aggregate functions](docs/language/operators/summarize.md) and grouping (#1385) * zql: Point links for developer docs at [pkg.go.dev](https://pkg.go.dev/) instead of [godoc.org](https://godoc.org/) (#1401) * zq: Add support for timestamps with signed timezone offsets (#1389) -* zq: Add a [JSON type definition](docs/integrations/zeek/README.md) for alert events in [Suricata EVE logs](https://suricata.readthedocs.io/en/suricata-5.0.2/output/eve/eve-json-output.html) (#1400) +* zq: Add a [JSON type definition](docs/integrations/zeek/_index.md) for alert events in [Suricata EVE logs](https://suricata.readthedocs.io/en/suricata-5.0.2/output/eve/eve-json-output.html) (#1400) * zq: Update the [ZNG over JSON (ZJSON)](docs/formats/zjson.md) spec and implementation (#1299) * zar: Use buffered streaming for archive import (#1397) * zq: Add an `ast` command that prints parsed ZQL as its underlying JSON object (#1416) @@ -789,7 +789,7 @@ questions. * zql: Add a new function `Time.trunc()` (#842) * zql: Support grouping by computed keys (#860) * zq: Change implementation of `every X` to use a computed groupby key (#893) -* zql: Clean up the [ZQL docs](docs/language/README.md) (#884) +* zql: Clean up the [ZQL docs](docs/language/_index.md) (#884) * zql: Change `cut` operator to emit any matching fields (#899) * zq: Allow output to an S3 bucket (#889) diff --git a/README.md b/README.md index 9598e589d5..be885c162f 100644 --- a/README.md +++ b/README.md @@ -4,12 +4,12 @@ SuperDB is a new analytics database that supports relational tables and JSON on an equal footing. It shines when it comes to data wrangling where you need to explore or process large eclectic data sets. It's also pretty decent at analytics and -[search use cases](https://zed.brimdata.io/docs/language/search-expressions). +[search use cases](https://superdb.org/docs/language/search-expressions). Unlike other relational systems that do performance-fragile "schema inference" of JSON, SuperDB won't fall over if you throw a bunch of eclectic JSON at it. You can easily do -[schema inference if you want](https://zed.brimdata.io/docs/language/operators/fuse), +[schema inference if you want](https://superdb.org/docs/language/operators/fuse), but data is ingested by default in its natural form no matter how much heterogeneity it might have. And unlike systems based on the document data model, every value in SuperDB is strongly and dynamically typed thus providing the @@ -21,7 +21,7 @@ way to do things" and a different "JSON way to do things". Instead of having a relational type system for structured data and completely separate JSON type system for semi-structured data, all data handled by SuperDB (e.g., JSON, CSV, Parquet files, Arrow streams, relational tables, etc) is automatically massaged into -[super-structured data](https://zed.brimdata.io/docs/formats/#2-zed-a-super-structured-pattern) +[super-structured data](https://superdb.org/docs/formats/#2-a-super-structured-pattern) form. This super-structured data is then processed by a runtime that simultaneously supports the statically-typed relational model and the dynamically-typed JSON data model in a unified compute engine. @@ -30,7 +30,7 @@ JSON data model in a unified compute engine. SuperDB uses SQL as its query language, but it's a SQL that has been extended with [pipe syntax](https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syntax-in-sql/) -and [lots of fun shortcuts](https://zed.brimdata.io/docs/language/pipeline-model#implied-operators). +and [lots of fun shortcuts](https://superdb.org/docs/language/pipeline-model/#implied-operators). This extended SQL is called SuperSQL. Here's a SuperSQL query that fetches some data from GitHub Archive, @@ -57,11 +57,11 @@ FROM 'https://data.gharchive.org/2015-01-01-15.json.gz' Super-structured data is strongly typed and "polymorphic": any value can take on any type and sequences of data need not all conform to a predefined schema. To this end, SuperDB extends the JSON format to support super-structured data in a format called -[Super JSON](https://zed.brimdata.io/docs/next/formats/jsup) where all JSON values +[Super JSON](https://superdb.org/docs/formats/jsup) where all JSON values are also Super JSON values. Similarly, -the [Super Binary](https://zed.brimdata.io/docs/formats/zng) format is an efficient +the [Super Binary](https://superdb.org/docs/formats/bsup) format is an efficient binary representation of Super JSON (a bit like Avro) and the -[Super Columnar](https://zed.brimdata.io/docs/formats/vng) format is a columnar +[Super Columnar](https://superdb.org/docs/formats/csup) format is a columnar representation of Super JSON (a bit like Parquet). Even though SuperDB is based on these super-structured data formats, it can read and write @@ -69,17 +69,17 @@ most common data formats. ## Try It -Trying out SuperDB is super easy: just [install](https://zed.brimdata.io/docs/#getting-started) -the command-line tool [`super`](https://zed.brimdata.io/docs/commands/zq/). +Trying out SuperDB is super easy: just [install](https://superdb.org/docs/install) +the command-line tool [`super`](https://superdb.org/docs/commands/super). Detailed documentation for the entire SuperDB system and its piped SQL syntax -is available on the [SuperDB docs site](https://zed.brimdata.io/docs). +is available on the [SuperDB docs site](https://superdb.org/docs). The SuperDB query engine can run locally without a storage engine by accessing files, HTTP endpoints, or S3 paths using the `super` command. While -[earlier in its development](https://zed.brimdata.io/docs/commands/zed/#status), +[earlier in its development](https://superdb.org/docs/commands/super-db/#status), SuperDB can also run on a -[super-structured data lake](https://zed.brimdata.io/docs/commands/zed/#the-lake-model) +[super-structured data lake](https://superdb.org/docs/commands/super-db/#the-lake-model) using the `super db` sub-commands. ## Project Status diff --git a/docs/README.md b/docs/_index.md similarity index 93% rename from docs/README.md rename to docs/_index.md index 6a3dba787a..ff82fadb17 100644 --- a/docs/README.md +++ b/docs/_index.md @@ -1,12 +1,12 @@ --- -sidebar_position: 1 -sidebar_label: Introduction +weight: 1 +title: Introduction +heading: SuperDB +breadcrumb: Docs --- -# SuperDB - SuperDB offers a new approach that makes it easier to manipulate and manage -your data. With its [super-structured data model](formats/README.md#2-a-super-structured-pattern), +your data. With its [super-structured data model](formats/_index.md#2-a-super-structured-pattern), messy JSON data can easily be given the fully-typed precision of relational tables without giving up JSON's uncanny ability to represent eclectic data. @@ -27,7 +27,7 @@ while for a technical user, SuperDB exposes its technical underpinnings in a gradual slope, providing as much detail as desired, packaged up in the easy-to-understand [Super JSON data format](formats/jsup.md) and -[SuperPipe language](language/README.md). +[SuperPipe language](language/_index.md). While `super` and its accompanying data formats are production quality, the project's [SuperDB data lake](commands/super-db.md) is a bit earlier in development. @@ -38,12 +38,12 @@ While `super` and its accompanying data formats are production quality, the proj a number of different elements of the system: * The [super data model](formats/zed.md) is the abstract definition of the data types and semantics that underlie the super-structured data formats. -* The [super data formats](formats/README.md) are a family of +* The [super data formats](formats/_index.md) are a family of [human-readable (Super JSON, JSUP)](formats/jsup.md), [sequential (Super Binary, BSUP)](formats/bsup.md), and [columnar (Super Columnar, CSUP)](formats/csup.md) formats that all adhere to the same abstract super data model. -* The [SuperPipe language](language/README.md) is the system's pipeline language for performing +* The [SuperPipe language](language/_index.md) is the system's pipeline language for performing queries, searches, analytics, transformations, or any of the above combined together. * A [SuperPipe query](language/overview.md) is a script that performs search and/or analytics. @@ -58,7 +58,7 @@ accessed via a [Git](https://git-scm.com/)-like API. ## Digging Deeper -The [SuperPipe language documentation](language/README.md) +The [SuperPipe language documentation](language/_index.md) is the best way to learn about `super` in depth. All of its examples use `super` commands run on the command line. Run `super -h` for a list of command options and online help. diff --git a/docs/commands/_category_.yaml b/docs/commands/_category_.yaml deleted file mode 100644 index e5c770bade..0000000000 --- a/docs/commands/_category_.yaml +++ /dev/null @@ -1,2 +0,0 @@ -position: 3 -label: Commands diff --git a/docs/commands/README.md b/docs/commands/_index.md similarity index 82% rename from docs/commands/README.md rename to docs/commands/_index.md index c4eaa41682..f618316376 100644 --- a/docs/commands/README.md +++ b/docs/commands/_index.md @@ -1,4 +1,7 @@ -# Command Tooling +--- +title: Commands +weight: 2 +--- The [`super` command](super.md) is used to execute command-line queries on inputs from files, HTTP URLs, or [S3](../integrations/amazon-s3.md). @@ -8,7 +11,7 @@ into, querying, and orchestrating SuperDB data lakes. These sub-commands are organized into further subcommands like the familiar command patterns of `docker` or `kubectl`. -All operations with these commands utilize the [super data model](../formats/README.md) -and provide querying via [SuperSQL](../language/README.md). +All operations with these commands utilize the [super data model](../formats/_index.md) +and provide querying via [SuperSQL](../language/_index.md). Built-in help for `super` and all sub-commands is always accessible with the `-h` flag. diff --git a/docs/commands/super-db.md b/docs/commands/super-db.md index a2237260ab..a338a50389 100644 --- a/docs/commands/super-db.md +++ b/docs/commands/super-db.md @@ -1,13 +1,11 @@ --- -sidebar_position: 2 -sidebar_label: super db +weight: 2 +title: super db --- -# `super db` - > **TL;DR** `super db` is a sub-command of `super` to manage and query SuperDB data lakes. > You can import data from a variety of formats and it will automatically -> be committed in [super-structured](../formats/README.md) +> be committed in [super-structured](../formats/_index.md) > format, providing full fidelity of the original format and the ability > to reconstruct the original data without loss of information. > @@ -17,8 +15,9 @@ sidebar_label: super db

-:::tip Status -While [`super`](super.md) and its accompanying [formats](../formats/README.md) +{{% tip "Status" %}} + +While [`super`](super.md) and its accompanying [formats](../formats/_index.md) are production quality, the SuperDB data lake is still fairly early in development and alpha quality. That said, SuperDB data lakes can be utilized quite effectively at small scale, @@ -27,7 +26,8 @@ is deployed to manage the lake's data layout via the [lake API](../lake/api.md). Enhanced scalability with self-tuning configuration is under development. -::: + +{{% /tip %}} ## The Lake Model @@ -155,7 +155,8 @@ running any `super db` lake command all pointing at the same storage endpoint and the lake's data footprint will always remain consistent as the endpoints all adhere to the consistency semantics of the lake. -:::tip caveat +{{% tip "Caveat" %}} + Data consistency is not fully implemented yet for the S3 endpoint so only single-node access to S3 is available right now, though support for multi-node access is forthcoming. @@ -166,7 +167,8 @@ access to a local file system has been thoroughly tested and should be deemed reliable, i.e., you can run a direct-access instance of `super db` alongside a server instance of `super db` on the same file system and data consistency will be maintained. -::: + +{{% /tip %}} ### Locating the Lake @@ -208,11 +210,13 @@ Each commit object is assigned a global ID. Similar to Git, commit objects are arranged into a tree and represent the entire commit history of the lake. -:::tip note +{{% tip "Note" %}} + Technically speaking, Git can merge from multiple parents and thus Git commits form a directed acyclic graph instead of a tree; SuperDB does not currently support multiple parents in the commit object history. -::: + +{{% /tip %}} A branch is simply a named pointer to a commit object in the lake and like a pool, a branch name can be any valid UTF-8 string. @@ -274,10 +278,12 @@ key. For example, on a pool with pool key `ts`, the query `ts == 100` will be optimized to scan only the data objects where the value `100` could be present. -:::tip note +{{% tip "Note" %}} + The pool key will also serve as the primary key for the forthcoming CRUD semantics. -::: + +{{% /tip %}} A pool also has a configured sort order, either ascending or descending and data is organized in the pool in accordance with this order. @@ -327,9 +333,11 @@ using that pool's "branches log" in a similar fashion, then its corresponding commit object can be used to construct the data of that branch at that past point in time. -:::tip note +{{% tip "Note" %}} + Time travel using timestamps is a forthcoming feature. -::: + +{{% /tip %}} ## `super db` Commands @@ -358,7 +366,7 @@ format. However, the `-f` option can be used to specify any supported super db auth login|logout|method|verify ``` Access to a lake can be secured with [Auth0 authentication](https://auth0.com/). -A [guide](../integrations/zed-lake-auth.md) is available with example configurations. +A [guide](../integrations/zed-lake-auth/index.md) is available with example configurations. Please reach out to us on our [community Slack](https://www.brimdata.io/join-slack/) if you have feedback on your experience or need additional help. @@ -409,11 +417,13 @@ the [special value `this`](../language/pipeline-model.md#the-special-value-this) A newly created pool is initialized with a branch called `main`. -:::tip note +{{% tip "Note" %}} + Lakes can be used without thinking about branches. When referencing a pool without a branch, the tooling presumes the "main" branch as the default, and everything can be done on main without having to think about branching. -::: + +{{% /tip %}} ### Delete ``` @@ -584,9 +594,11 @@ that is stored in the commit journal for reference. These values may be specified as options to the [`load`](#load) command, and are also available in the [lake API](../lake/api.md) for automation. -:::tip note +{{% tip "Note" %}} + The branchlog meta-query source is not yet implemented. -::: + +{{% /tip %}} ### Ls ``` @@ -659,7 +671,7 @@ according to configured policies and logic. ``` super db query [options] ``` -The `query` command runs a [SuperSQL](../language/README.md) query with data from a lake as input. +The `query` command runs a [SuperSQL](../language/_index.md) query with data from a lake as input. A query typically begins with a [`from` operator](../language/operators/from.md) indicating the pool and branch to use as input. diff --git a/docs/commands/super.md b/docs/commands/super.md index f13ba84cfd..7736f0f0b0 100644 --- a/docs/commands/super.md +++ b/docs/commands/super.md @@ -1,11 +1,9 @@ --- -sidebar_position: 1 -sidebar_label: super +weight: 1 +title: super --- -# `super` - -> **TL;DR** `super` is a command-line tool that uses [SuperSQL](../language/README.md) +> **TL;DR** `super` is a command-line tool that uses [SuperSQL](../language/_index.md) > to query a variety of data formats in files, over HTTP, or in [S3](../integrations/amazon-s3.md) > storage. Best performance is achieved when operating on data in binary formats such as > [Super Binary](../formats/bsup.md), [Super Columnar](../formats/csup.md), @@ -20,7 +18,7 @@ super [ options ] [ -c query ] input [ input ... ] `super` is a command-line tool for processing data in diverse input formats, providing data wrangling, search, analytics, and extensive transformations -using the [SuperSQL](../language/README.md) dialect of SQL. Any SQL query expression +using the [SuperSQL](../language/_index.md) dialect of SQL. Any SQL query expression may be extended with [pipe syntax](https://research.google/pubs/sql-has-problems-we-can-fix-them-pipe-syntax-in-sql/) to filter, transform, and/or analyze input data. Super's SQL pipes dialect is extensive, so much so that it can resemble @@ -30,10 +28,10 @@ The `super` command works with data from ephemeral sources like files and URLs. If you want to persist your data into a data lake for persistent storage, check out the [`super db`](super-db.md) set of commands. -By invoking the `-c` option, a query expressed in the [SuperSQL language](../language/README.md) +By invoking the `-c` option, a query expressed in the [SuperSQL language](../language/_index.md) may be specified and applied to the input stream. -The [super data model](../formats/zed.md) is based on [super-structured data](../formats/README.md#2-a-super-structured-pattern), meaning that all data +The [super data model](../formats/zed.md) is based on [super-structured data](../formats/_index.md#2-a-super-structured-pattern), meaning that all data is both strongly _and_ dynamically typed and need not conform to a homogeneous schema. The type structure is self-describing so it's easy to daisy-chain queries and inspect data at any point in a complex query or data pipeline. @@ -189,13 +187,15 @@ not desirable because (1) the Super JSON parser is not particularly performant a (2) all JSON numbers are floating point but the Super JSON parser will parse as JSON any number that appears without a decimal point as an integer type. -:::tip note +{{% tip "Note" %}} + The reason `super` is not particularly performant for Super JSON is that the [Super Binary](../formats/bsup.md) or [Super Columnar](../formats/csup.md) formats are semantically equivalent to Super JSON but much more efficient and the design intent is that these efficient binary formats should be used in use cases where performance matters. Super JSON is typically used only when data needs to be human-readable in interactive settings or in automated tests. -::: + +{{% /tip %}} To this end, `super` uses a heuristic to select between Super JSON and plain JSON when the `-i` option is not specified. Specifically, plain JSON is selected when the first values @@ -601,15 +601,43 @@ error("divide by zero") ## Examples -As you may have noticed, many examples of the [SuperSQL language](../language/README.md) -are illustrated using this pattern +As you may have noticed, many examples shown above were illustrated using this +pattern: ``` echo | super -c - ``` -which is used throughout the [language documentation](../language/README.md) -and [operator reference](../language/operators/README.md). -The language documentation and [tutorials directory](../tutorials/README.md) +While this is suitable for showing command line operations, in documentation +focused on showing the [SuperSQL language](../language/_index.md) (such as the +[operator](../language/operators/_index.md) and [function](../language/functions/_index.md) +references), interactive examples are often used that are pre-populated with +the query, input, and a "live" result that's generated using a +[browser-based packaging of the SuperDB runtime](https://github.com/brimdata/superdb-wasm). +This allows you to modify the query and/or inputs and immediately see how it +changes the result, all without having to drop to the shell. If you make +changes to an example and want to return it to its original state, just +refresh the page in your browser. + +For example, here's an interactive rendering of the example from the +[Error Handling](#error-handling) section above. Try adding an additional +input value and notice the immediate change in the result. + +```mdtest-spq +# spq +10.0/this +# input +1 +2 +0 +3 +# expected output +10. +5. +error("divide by zero") +3.3333333333333335 +``` + +The language documentation and [tutorials directory](../tutorials/_index.md) have many examples, but here are a few more simple `super` use cases. _Hello, world_ @@ -622,55 +650,75 @@ produces this Super JSON output ``` _Some values of available [data types](../language/data-types.md)_ -```mdtest-command -echo '1 1.5 [1,"foo"] |["apple","banana"]|' | super -z - -``` -produces -```mdtest-output +```mdtest-spq +# spq +SELECT VALUE this +# input +1 +1.5 +[1,"foo"] +|["apple","banana"]| +# expected output 1 1.5 [1,"foo"] |["apple","banana"]| ``` + _The types of various data_ -```mdtest-command -echo '1 1.5 [1,"foo"] |["apple","banana"]|' | super -z -c 'SELECT VALUE typeof(this)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +SELECT VALUE typeof(this) +# input +1 +1.5 +[1,"foo"] +|["apple","banana"]| +# expected output <[(int64,string)]> <|[string]|> ``` -_A simple [aggregation](../language/aggregates/README.md)_ -```mdtest-command -echo '{key:"foo",val:1}{key:"bar",val:2}{key:"foo",val:3}' | - super -z -c 'sum(val) by key | sort key' - -``` -produces -```mdtest-output + +_A simple [aggregation](../language/aggregates/_index.md)_ +```mdtest-spq +# spq +sum(val) by key | sort key +# input +{key:"foo",val:1} +{key:"bar",val:2} +{key:"foo",val:3} +# expected output {key:"bar",sum:2} {key:"foo",sum:4} ``` + _Read CSV input and [cast](../language/functions/cast.md) a to an integer from default float_ -```mdtest-command -printf "a,b\n1,foo\n2,bar\n" | super -z -c 'a:=int64(a)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +a:=int64(a) +# input +a,b +1,foo +2,bar +# expected output {a:1,b:"foo"} {a:2,b:"bar"} ``` + _Read JSON input and cast to an integer from default float_ -```mdtest-command -echo '{"a":1,"b":"foo"}{"a":2,"b":"bar"}' | super -z -c 'a:=int64(a)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +a:=int64(a) +# input +{"a":1,"b":"foo"} +{"a":2,"b":"bar"} +# expected output {a:1,b:"foo"} {a:2,b:"bar"} ``` + _Make a schema-rigid Parquet file using fuse, then output the Parquet file as Super JSON_ ```mdtest-command echo '{a:1}{a:2}{b:3}' | super -f parquet -o tmp.parquet -c fuse - diff --git a/docs/formats/README.md b/docs/formats/_index.md similarity index 99% rename from docs/formats/README.md rename to docs/formats/_index.md index 98395229ee..e7fa09c7bf 100644 --- a/docs/formats/README.md +++ b/docs/formats/_index.md @@ -1,4 +1,7 @@ -# Formats +--- +title: Formats +weight: 5 +--- > **TL;DR** The super data model defines a new and easy way to manage, store, > and process data utilizing an emerging concept called @@ -228,11 +231,13 @@ anywhere that a value can appear. In particular, types can be aggregation keys. This is very powerful for data discovery and introspection. For example, to count the different shapes of data, you might have a SuperSQL query, operating on each input value as `this`, that has the form: -``` + +```sql SELECT count(), typeof(this) AS shape GROUP BY shape, count ``` Likewise, you could select a sample value of each shape like this: -``` + +```sql SELECT shape FROM ( SELECT any(this) AS sample, typeof(this) AS shape GROUP BY shape,sample ) diff --git a/docs/formats/bsup.md b/docs/formats/bsup.md index 4982f6f295..66dec48288 100644 --- a/docs/formats/bsup.md +++ b/docs/formats/bsup.md @@ -1,10 +1,9 @@ --- -sidebar_position: 2 -sidebar_label: Super Binary +weight: 2 +title: Super Binary +heading: Super Binary Specification --- -# Super Binary Specification - ## 1. Introduction Super Binary is an efficient, sequence-oriented serialization format for any data @@ -131,7 +130,8 @@ size decompression buffers in advance of decoding. Values for the `format` byte are defined in the [Super Binary compression format specification](./compression.md). -:::tip note +{{% tip "Note" %}} + This arrangement of frames separating types and values allows for efficient scanning and parallelization. In general, values depend on type definitions but as long as all of the types are known when @@ -144,7 +144,8 @@ heuristics, e.g., knowing a filtering predicate can't be true based on a quick scan of the data perhaps using the Boyer-Moore algorithm to determine that a comparison with a string constant would not work for any value in the buffer. -::: + +{{% /tip %}} Whether the payload was originally uncompressed or was decompressed, it is then interpreted according to the `T` bits of the frame code as a @@ -212,12 +213,14 @@ is further encoded as a "counted string", which is the `uvarint` encoding of the length of the string followed by that many bytes of UTF-8 encoded string data. -:::tip note +{{% tip "Note" %}} + As defined by [Super JSON](jsup.md), a field name can be any valid UTF-8 string much like JSON objects can be indexed with arbitrary string keys (via index operator) even if the field names available to the dot operator are restricted by language syntax for identifiers. -::: + +{{% /tip %}} The type ID follows the field name and is encoded as a `uvarint`. @@ -296,7 +299,7 @@ indicated by ``. #### 2.1.8 Named Type Typedef -A named type defines a new type ID that binds a name to a previously existing type ID. +A named type defines a new type ID that binds a name to a previously existing type ID. A named type is encoded as follows: ``` diff --git a/docs/formats/compression.md b/docs/formats/compression.md index 5934ad09b6..4b3ed3e40a 100644 --- a/docs/formats/compression.md +++ b/docs/formats/compression.md @@ -1,10 +1,9 @@ --- -sidebar_position: 6 -sidebar_label: Compression +weight: 6 +title: Compression +heading: ZNG Compression Types --- -# ZNG Compression Types - This document specifies values for the `` byte of a [Super Binary compressed value message block](bsup.md#2-the-super-binary-format) and the corresponding algorithms for the `` byte sequence. diff --git a/docs/formats/csup.md b/docs/formats/csup.md index 55377db18d..a64b394511 100644 --- a/docs/formats/csup.md +++ b/docs/formats/csup.md @@ -1,14 +1,13 @@ --- -sidebar_position: 4 -sidebar_label: Super Columnar +weight: 4 +title: Super Columnar +heading: Super Columnar Specification --- -# Super Columnar Specification - Super Columnar is a file format based on the [super data model](zed.md) where data is stacked to form columns. Its purpose is to provide for efficient analytics and search over -bounded-length sequences of [super-structured data](./README.md#2-a-super-structured-pattern) that is stored in columnar form. +bounded-length sequences of [super-structured data](./_index.md#2-a-super-structured-pattern) that is stored in columnar form. Like [Parquet](https://github.com/apache/parquet-format), Super Columnar provides an efficient representation for semi-structured data, @@ -65,12 +64,14 @@ then write the metadata into the reassembly section along with the trailer at the end. This allows a stream to be converted to a Super Columnar file in a single pass. -:::tip note +{{% tip "Note" %}} + That said, the layout is flexible enough that an implementation may optimize the data layout with additional passes or by writing the output to multiple files then merging them together (or even leaving the Super Columnar entity as separate files). -::: + +{{% /tip %}} ### The Data Section @@ -86,7 +87,8 @@ There is no information in the data section for how segments relate to one another or how they are reconstructed into columns. They are just blobs of Super Binary data. -:::tip note +{{% tip "Note" %}} + Unlike Parquet, there is no explicit arrangement of the column chunks into row groups but rather they are allowed to grow at different rates so a high-volume column might be comprised of many segments while a low-volume @@ -94,9 +96,11 @@ column must just be one or several. This allows scans of low-volume record type (the "mice") to perform well amongst high-volume record types (the "elephants"), i.e., there are not a bunch of seeks with tiny reads of mice data interspersed throughout the elephants. -::: -:::tip TBD +{{% /tip %}} + +{{% tip "TBD" %}} + The mice/elephants model creates an interesting and challenging layout problem. If you let the row indexes get too far apart (call this "skew"), then you have to buffer very large amounts of data to keep the column data aligned. @@ -110,7 +114,8 @@ if you use lots of buffering on ingest, you can write the mice in front of the elephants so the read path requires less buffering to align columns. Or you can do two passes where you store segments in separate files then merge them at close according to an optimization plan. -::: + +{{% /tip %}} ### The Reassembly Section @@ -118,7 +123,8 @@ The reassembly section provides the information needed to reconstruct column streams from segments, and in turn, to reconstruct the original values from column streams, i.e., to map columns back to composite values. -:::tip note +{{% tip "Note" %}} + Of course, the reassembly section also provides the ability to extract just subsets of columns to be read and searched efficiently without ever needing to reconstruct the original rows. How well this performs is up to any particular @@ -128,7 +134,8 @@ Also, the reassembly section is in general vastly smaller than the data section so the goal here isn't to express information in cute and obscure compact forms but rather to represent data in an easy-to-digest, programmer-friendly form that leverages Super Binary. -::: + +{{% /tip %}} The reassembly section is a Super Binary stream. Unlike Parquet, which uses an externally described schema @@ -148,9 +155,11 @@ A super type's integer position in this sequence defines its identifier encoded in the [super column](#the-super-column). This identifier is called the super ID. -:::tip note +{{% tip "Note" %}} + Change the first N values to type values instead of nulls? -::: + +{{% /tip %}} The next N+1 records contain reassembly information for each of the N super types where each record defines the column streams needed to reconstruct the original @@ -172,11 +181,13 @@ type signature: In the rest of this document, we will refer to this type as `` for shorthand and refer to the concept as a "segmap". -:::tip note +{{% tip "Note" %}} + We use the type name "segmap" to emphasize that this information represents a set of byte ranges where data is stored and must be read from *rather than* the data itself. -::: + +{{% /tip %}} #### The Super Column @@ -217,11 +228,13 @@ This simple top-down arrangement, along with the definition of the other column structures below, is all that is needed to reconstruct all of the original data. -:::tip note +{{% tip "Note" %}} + Each row reassembly record has its own layout of columnar values and there is no attempt made to store like-typed columns from different schemas in the same physical column. -::: + +{{% /tip %}} The notation `` refers to any instance of the five column types: * [``](#record-column), @@ -245,7 +258,7 @@ and has the form: :{column:,presence:}, ... :{column:,presence:} -} +} ``` where * `` through `` are the names of the top-level fields of the @@ -268,7 +281,7 @@ An `` has the form: {values:,lengths:} ``` where -* `values` represents a continuous sequence of values of the array elements +* `values` represents a continuous sequence of values of the array elements that are sliced into array values based on the length information, and * `lengths` encodes an `int32` sequence of values that represent the length of each array value. @@ -297,9 +310,11 @@ in the same column order implied by the union type, and * `tags` is a column of `int32` values where each subsequent value encodes the tag of the union type indicating which column the value falls within. -:::tip TBD +{{% tip "TBD" %}} + Change code to conform to columns array instead of record{c0,c1,...} -::: + +{{% /tip %}} The number of times each value of `tags` appears must equal the number of values in each respective column. @@ -351,14 +366,16 @@ data in the file, it will typically fit comfortably in memory and it can be very fast to scan the entire reassembly structure for any purpose. -:::tip Example +{{% tip "Example" %}} + For a given query, a "scan planner" could traverse all the reassembly records to figure out which segments will be needed, then construct an intelligent plan for reading the needed segments and attempt to read them in mostly sequential order, which could serve as an optimizing intermediary between any underlying storage API and the Super Columnar decoding logic. -::: + +{{% /tip %}} To decode the "next" row, its schema index is read from the root reassembly column stream. diff --git a/docs/formats/jsup.md b/docs/formats/jsup.md index 601ee9df48..44d24e8257 100644 --- a/docs/formats/jsup.md +++ b/docs/formats/jsup.md @@ -1,10 +1,9 @@ --- -sidebar_position: 3 -sidebar_label: Super JSON +weight: 3 +title: Super JSON +heading: Super JSON Specification --- -# Super JSON Specification - ## 1. Introduction Super JSON is the human-readable, text-based serialization format of diff --git a/docs/formats/zed.md b/docs/formats/zed.md index 6018704201..17e5e6b12e 100644 --- a/docs/formats/zed.md +++ b/docs/formats/zed.md @@ -1,10 +1,8 @@ --- -sidebar_position: 1 -sidebar_label: Data Model +weight: 1 +title: Data Model --- -# Zed Data Model - Zed data is defined as an ordered sequence of one or more typed data values. Each value's type is either a "primitive type", a "complex type", the "type type", a "named type", or the "null type". @@ -154,7 +152,7 @@ have a common Zed type and the values have a common Zed type. Each key across an instance of a map value must be a unique value. -A map value may be empty. +A map value may be empty. A map type is uniquely defined by its key type and value type. @@ -196,7 +194,7 @@ the type order of the constituent types in left to right order. ### 2.7 Error -An error represents any value designated as an error. +An error represents any value designated as an error. The type order of an error is the type order of the type of its contained value. diff --git a/docs/formats/zjson.md b/docs/formats/zjson.md index 9d677be7c9..f4e8d506fa 100644 --- a/docs/formats/zjson.md +++ b/docs/formats/zjson.md @@ -1,10 +1,9 @@ --- -sidebar_position: 5 -sidebar_label: ZJSON +weight: 5 +title: ZJSON +heading: ZJSON Specification --- -# ZJSON Specification - ## 1. Introduction The [super data model](zed.md) @@ -61,7 +60,7 @@ Also, it is at the whim of a JSON implementation whether or not the order of object keys is preserved. While JSON is well suited for data exchange of generic information, it is not -sufficient for the [super-structured data model](./README.md#2-a-super-structured-pattern). +sufficient for the [super-structured data model](./_index.md#2-a-super-structured-pattern). That said, JSON can be used as an encoding format for super data with another layer of encoding on top of a JSON-based protocol. This allows clients like web apps or Electron apps to receive and understand Super JSON and, with the help of client diff --git a/docs/install.md b/docs/install.md index b643127e72..1b91d023bd 100644 --- a/docs/install.md +++ b/docs/install.md @@ -1,10 +1,7 @@ --- -sidebar_position: 2 -sidebar_label: Installation +title: Installation --- -# Installation - Several options for installing `super` are available: * [Homebrew](#homebrew) for Mac or Linux, * [Binary download](#binary-download), or @@ -42,11 +39,13 @@ This installs the `super` binary in your `$GOPATH/bin`. Once installed, run a [quick test](#quick-tests). -:::tip note +{{% tip "Note" %}} + If you don't have Go installed, download and install it from the [Go install page](https://golang.org/doc/install). Go 1.23 or later is required. -::: + +{{% /tip %}} ## Quick Tests diff --git a/docs/integrations/_category_.yaml b/docs/integrations/_category_.yaml deleted file mode 100644 index 034b94791f..0000000000 --- a/docs/integrations/_category_.yaml +++ /dev/null @@ -1,2 +0,0 @@ -position: 9 -label: Integrations diff --git a/docs/integrations/_index.md b/docs/integrations/_index.md new file mode 100644 index 0000000000..f5b53b81af --- /dev/null +++ b/docs/integrations/_index.md @@ -0,0 +1,4 @@ +--- +weight: 8 +title: Integrations +--- diff --git a/docs/integrations/amazon-s3.md b/docs/integrations/amazon-s3.md index 54cf0b57d8..671825a042 100644 --- a/docs/integrations/amazon-s3.md +++ b/docs/integrations/amazon-s3.md @@ -1,10 +1,8 @@ --- -sidebar_position: 1 -sidebar_label: Amazon S3 +weight: 1 +title: Amazon S3 --- -# Amazon S3 - Zed tools can access [Amazon S3](https://aws.amazon.com/s3/) and S3-compatible storage via `s3://` URIs. Details are described below. @@ -18,11 +16,13 @@ You must specify an AWS region via one of the following: You can create `~/.aws/config` by installing the [AWS CLI](https://aws.amazon.com/cli/) and running `aws configure`. -:::tip Note +{{% tip "Note" %}} + If using S3-compatible storage that does not recognize the concept of regions, a region must still be specified, e.g., by providing a dummy value for `AWS_REGION`. -::: + +{{% /tip %}} ## Credentials diff --git a/docs/integrations/fluentd.md b/docs/integrations/fluentd.md index e4146f9388..648da0b29d 100644 --- a/docs/integrations/fluentd.md +++ b/docs/integrations/fluentd.md @@ -1,10 +1,8 @@ --- -sidebar_position: 3 -sidebar_label: Fluentd +weight: 3 +title: Fluentd --- -# Fluentd - The [Fluentd](https://www.fluentd.org/) open source data collector can be used to push log data to a [SuperDB data lake](../commands/super-db.md) in a continuous manner. This allows for querying near-"live" event data to enable use cases such as @@ -14,7 +12,7 @@ record for archiving and analytics. This guide walks through two simple configurations of Fluentd with a Zed lake that can be used as reference for starting your own production configuration. As it's a data source important to many in the Zed community, log data from -[Zeek](./zeek/README.md) is used in this guide. The approach shown can be +[Zeek](./zeek/_index.md) is used in this guide. The approach shown can be easily adapted to any log data source. ## Software @@ -83,13 +81,15 @@ The default settings when running `zed create` set the field and sort the stored data in descending order by that key. This configuration is ideal for Zeek log data. -:::tip Note +{{% tip "Note" %}} + The [Zui](https://zui.brimdata.io/) desktop application automatically starts a Zed lake service when it launches. Therefore if you are using Zui you can skip the first set of commands shown above. The pool can be created from Zui by clicking **+**, selecting **New Pool**, then entering `ts` for the [pool key](../commands/super-db.md#pool-key). -::: + +{{% /tip %}} ### Fluentd @@ -307,7 +307,7 @@ Example output: Notice quotes are no longer present around the values that contain IP addresses and times, since they are no longer stored as strings. With the data in this -shaped form, we could now invoke [Zed language](../language/README.md) +shaped form, we could now invoke [Zed language](../language/_index.md) functionality that leverages the richer data typing such as filtering `ip` values by CIDR block, e.g., @@ -368,7 +368,8 @@ leverage, you can reduce the lake's storage footprint by periodically running storage that contain the granular commits that have already been rolled into larger objects by compaction. -:::tip Note +{{% tip "Note" %}} + As described in issue [super/4934](https://github.com/brimdata/super/issues/4934), even after running `zed vacuum`, some files related to commit history are currently still left behind below the lake storage path. The issue describes @@ -376,7 +377,8 @@ manual steps that can be taken to remove these files safely, if desired. However, if you find yourself needing to take these steps in your environment, please [contact us](#contact-us) as it will allow us to boost the priority of addressing the issue. -::: + +{{% /tip %}} ## Ideas For Enhancement diff --git a/docs/integrations/grafana.md b/docs/integrations/grafana.md index 230bebdae4..b1bad65e6a 100644 --- a/docs/integrations/grafana.md +++ b/docs/integrations/grafana.md @@ -1,10 +1,9 @@ --- -sidebar_position: 4 -sidebar_label: Grafana +weight: 4 +title: Grafana +heading: Grafana Data Source Plugin --- -# Grafana Data Source Plugin - A [data source plugin](https://grafana.com/grafana/plugins/?type=datasource) for [Grafana](https://grafana.com/) is available that enables plotting of time-series data that's stored in [SuperDB data lakes](../commands/super-db.md). See the diff --git a/docs/integrations/allowed-callback-urls.png b/docs/integrations/zed-lake-auth/allowed-callback-urls.png similarity index 100% rename from docs/integrations/allowed-callback-urls.png rename to docs/integrations/zed-lake-auth/allowed-callback-urls.png diff --git a/docs/integrations/api-allow-offline-access.png b/docs/integrations/zed-lake-auth/api-allow-offline-access.png similarity index 100% rename from docs/integrations/api-allow-offline-access.png rename to docs/integrations/zed-lake-auth/api-allow-offline-access.png diff --git a/docs/integrations/api-name-identifier.png b/docs/integrations/zed-lake-auth/api-name-identifier.png similarity index 100% rename from docs/integrations/api-name-identifier.png rename to docs/integrations/zed-lake-auth/api-name-identifier.png diff --git a/docs/integrations/application-name.png b/docs/integrations/zed-lake-auth/application-name.png similarity index 100% rename from docs/integrations/application-name.png rename to docs/integrations/zed-lake-auth/application-name.png diff --git a/docs/integrations/click-api-settings.png b/docs/integrations/zed-lake-auth/click-api-settings.png similarity index 100% rename from docs/integrations/click-api-settings.png rename to docs/integrations/zed-lake-auth/click-api-settings.png diff --git a/docs/integrations/click-application-settings.png b/docs/integrations/zed-lake-auth/click-application-settings.png similarity index 100% rename from docs/integrations/click-application-settings.png rename to docs/integrations/zed-lake-auth/click-application-settings.png diff --git a/docs/integrations/click-grant-types.png b/docs/integrations/zed-lake-auth/click-grant-types.png similarity index 100% rename from docs/integrations/click-grant-types.png rename to docs/integrations/zed-lake-auth/click-grant-types.png diff --git a/docs/integrations/create-api.png b/docs/integrations/zed-lake-auth/create-api.png similarity index 100% rename from docs/integrations/create-api.png rename to docs/integrations/zed-lake-auth/create-api.png diff --git a/docs/integrations/create-application.png b/docs/integrations/zed-lake-auth/create-application.png similarity index 100% rename from docs/integrations/create-application.png rename to docs/integrations/zed-lake-auth/create-application.png diff --git a/docs/integrations/device-code-grant-type.png b/docs/integrations/zed-lake-auth/device-code-grant-type.png similarity index 100% rename from docs/integrations/device-code-grant-type.png rename to docs/integrations/zed-lake-auth/device-code-grant-type.png diff --git a/docs/integrations/domain-client-id.png b/docs/integrations/zed-lake-auth/domain-client-id.png similarity index 100% rename from docs/integrations/domain-client-id.png rename to docs/integrations/zed-lake-auth/domain-client-id.png diff --git a/docs/integrations/expand-advanced-settings.png b/docs/integrations/zed-lake-auth/expand-advanced-settings.png similarity index 100% rename from docs/integrations/expand-advanced-settings.png rename to docs/integrations/zed-lake-auth/expand-advanced-settings.png diff --git a/docs/integrations/zed-lake-auth.md b/docs/integrations/zed-lake-auth/index.md similarity index 92% rename from docs/integrations/zed-lake-auth.md rename to docs/integrations/zed-lake-auth/index.md index e1c432e499..6ddd770d71 100644 --- a/docs/integrations/zed-lake-auth.md +++ b/docs/integrations/zed-lake-auth/index.md @@ -1,15 +1,14 @@ --- -sidebar_position: 2 -sidebar_label: Authentication Configuration +weight: 2 +title: Authentication Configuration +heading: Configuring Authentication for a Zed Lake Service --- -# Configuring Authentication for a Zed Lake Service - -A [SuperDB data lake service](../commands/super-db.md#serve) may be configured to require +A [SuperDB data lake service](../../commands/super-db.md#serve) may be configured to require user authentication to be accessed from clients such as the [Zui](https://zui.brimdata.io/) application, the -[`super db`](../commands/super.md) CLI commands, or the -[SuperDB Python client](../libraries/python.md). This document describes a simple +[`super db`](../../commands/super.md) CLI commands, or the +[SuperDB Python client](../../libraries/python.md). This document describes a simple [Auth0](https://auth0.com) configuration with accompanying `super db serve` flags that can be used as a starting point for creating similar configurations in your own environment. @@ -31,10 +30,12 @@ and then clicking the **Create API** button. 2. Enter any **Name** and URL **Identifier** for the API, then click the **Create** button. -:::tip +{{% tip "Tip" %}} + Note the value you enter for the **Identifier** as you'll need it later for the Zed lake service configuration. -::: + +{{% /tip %}} ![api-name-identifier](api-name-identifier.png) @@ -51,11 +52,13 @@ need it later for the Zed lake service configuration. 1. Begin creating a new application by clicking **Applications** in the left navigation menu and then clicking the **Create Application** button. -:::tip Note +{{% tip "Note" %}} + Neither the "Zed lake (Test Application)" that was created for us automatically when we created our API nor the Default App that came with the trial are used in this configuration. -::: + +{{% /tip %}} ![create-application](create-application.png) @@ -97,7 +100,7 @@ checkbox to enable the **Device Code** grant type. ## Zed Lake Service Configuration -1. Login to our Linux VM and [install](../install.md#building-from-source) +1. Login to our Linux VM and [install](../../install.md#building-from-source) the most recent Zed tools from source. ``` @@ -136,7 +139,7 @@ authentication configuration along with a directory name for lake storage. -auth.jwkspath=jwks.json \ -auth.audience=$auth0_api_identifier \ -lake=lake - + {"level":"info","ts":1678909988.9797907,"logger":"core","msg":"Started"} {"level":"info","ts":1678909988.9804773,"logger":"httpd","msg":"Listening","addr":"[::]:9867"} ... diff --git a/docs/integrations/login-flow-thumbnail.png b/docs/integrations/zed-lake-auth/login-flow-thumbnail.png similarity index 100% rename from docs/integrations/login-flow-thumbnail.png rename to docs/integrations/zed-lake-auth/login-flow-thumbnail.png diff --git a/docs/integrations/save-application-settings.png b/docs/integrations/zed-lake-auth/save-application-settings.png similarity index 100% rename from docs/integrations/save-application-settings.png rename to docs/integrations/zed-lake-auth/save-application-settings.png diff --git a/docs/integrations/zeek/_category_.yaml b/docs/integrations/zeek/_category_.yaml deleted file mode 100644 index 71bc2e94e2..0000000000 --- a/docs/integrations/zeek/_category_.yaml +++ /dev/null @@ -1,2 +0,0 @@ -position: 4 -label: Zeek diff --git a/docs/integrations/zeek/README.md b/docs/integrations/zeek/_index.md similarity index 86% rename from docs/integrations/zeek/README.md rename to docs/integrations/zeek/_index.md index 3a7c17b28e..1298a50099 100644 --- a/docs/integrations/zeek/README.md +++ b/docs/integrations/zeek/_index.md @@ -1,4 +1,7 @@ -# Zed Interoperability with Zeek Logs +--- +title: Zeek +heading: Zed Interoperability with Zeek Logs +--- Zed includes functionality and reference configurations specific to working with logs from the [Zeek](https://zeek.org/) open source network security diff --git a/docs/integrations/zeek/data-type-compatibility.md b/docs/integrations/zeek/data-type-compatibility.md index 1d0b109828..3f8f9a9619 100644 --- a/docs/integrations/zeek/data-type-compatibility.md +++ b/docs/integrations/zeek/data-type-compatibility.md @@ -1,10 +1,8 @@ --- -sidebar_position: 2 -sidebar_label: Zed/Zeek Data Type Compatibility +weight: 2 +title: Zed/Zeek Data Type Compatibility --- -# Zed/Zeek Data Type Compatibility - As the [super data model](../../formats/zed.md) was in many ways inspired by the [Zeek TSV log format](https://docs.zeek.org/en/master/log-formats.html#zeek-tsv-format-logs), SuperDB's rich storage formats ([Super JSON](../../formats/jsup.md), @@ -21,7 +19,7 @@ the types that may appear in Zeek logs. Zed tools maintain an internal Zed-typed representation of any Zeek data that is read or imported. Therefore, knowing the equivalent types will prove useful when performing operations in the -[Zed language](../../language/README.md) such as +[Zed language](../../language/_index.md) such as [type casting](../../language/shaping.md#cast) or looking at the data when output as Super JSON. @@ -51,7 +49,8 @@ applicable to handling certain types. | [`vector`](https://docs.zeek.org/en/current/script-reference/types.html#type-vector) | [`array`](../../formats/zed.md#22-array | | | [`record`](https://docs.zeek.org/en/current/script-reference/types.html#type-record) | [`record`](../../formats/zed.md#21-record | See [`record` details](#record) | -:::tip Note +{{% tip "Note" %}} + The [Zeek data types](https://docs.zeek.org/en/current/script-reference/types.html) page describes the types in the context of the [Zeek scripting language](https://docs.zeek.org/en/master/scripting/index.html). @@ -59,7 +58,8 @@ The Zeek types available in scripting are a superset of the data types that may appear in Zeek log files. The encodings of the types also differ in some ways between the two contexts. However, we link to this reference because there is no authoritative specification of the Zeek TSV log format. -::: + +{{% /tip %}} ## Example diff --git a/docs/integrations/zeek/reading-zeek-log-formats.md b/docs/integrations/zeek/reading-zeek-log-formats.md index fdc191e62b..8d78a9304f 100644 --- a/docs/integrations/zeek/reading-zeek-log-formats.md +++ b/docs/integrations/zeek/reading-zeek-log-formats.md @@ -1,13 +1,11 @@ --- -sidebar_position: 1 -sidebar_label: Reading Zeek Log Formats +weight: 1 +title: Reading Zeek Log Formats --- -# Reading Zeek Log Formats - Zed is capable of reading both common Zeek log formats. This document provides guidance for what to expect when reading logs of these formats using -the Zed [command line tools](../../commands/README.md). +the Zed [command line tools](../../commands/_index.md). ## Zeek TSV diff --git a/docs/integrations/zeek/shaping-zeek-json.md b/docs/integrations/zeek/shaping-zeek-json.md index 9ff9c13d0f..6ae365d0ee 100644 --- a/docs/integrations/zeek/shaping-zeek-json.md +++ b/docs/integrations/zeek/shaping-zeek-json.md @@ -1,10 +1,8 @@ --- -sidebar_position: 3 -sidebar_label: Shaping Zeek JSON +weight: 3 +title: Shaping Zeek JSON --- -# Shaping Zeek JSON - When [reading Zeek JSON format logs](reading-zeek-log-formats.md#zeek-json), much of the rich data typing that was originally present inside Zeek is at risk of being lost. This detail can be restored using a Zed @@ -195,11 +193,13 @@ specification. ... ``` -:::tip note +{{% tip "Note" %}} + See [the role of `_path`](reading-zeek-log-formats.md#the-role-of-_path) for important details if you're using Zeek's built-in [ASCII logger](https://docs.zeek.org/en/current/scripts/base/frameworks/logging/writers/ascii.zeek.html) rather than the [JSON Streaming Logs](https://github.com/corelight/json-streaming-logs) package. -::: + +{{% /tip %}} ### Zed Pipeline diff --git a/docs/lake/_category_.yaml b/docs/lake/_category_.yaml deleted file mode 100644 index 5beb193f7a..0000000000 --- a/docs/lake/_category_.yaml +++ /dev/null @@ -1,2 +0,0 @@ -position: 7 -label: Lake diff --git a/docs/lake/_index.md b/docs/lake/_index.md new file mode 100644 index 0000000000..5e175901aa --- /dev/null +++ b/docs/lake/_index.md @@ -0,0 +1,4 @@ +--- +title: Lake +weight: 6 +--- diff --git a/docs/lake/api.md b/docs/lake/api.md index 19ca7b0ed9..05b279869f 100644 --- a/docs/lake/api.md +++ b/docs/lake/api.md @@ -1,10 +1,9 @@ --- -sidebar_position: 1 -sidebar_label: API +weight: 1 +title: API +heading: Zed lake API --- -# Zed lake API - ## _Status_ > This is a brief sketch of the functionality exposed in the diff --git a/docs/lake/format.md b/docs/lake/format.md index dcc7a93921..f223ac2700 100644 --- a/docs/lake/format.md +++ b/docs/lake/format.md @@ -1,10 +1,9 @@ --- -sidebar_position: 2 -sidebar_label: Format +weight: 2 +title: Format +heading: Zed Lake Format --- -# Zed Lake Format - ## _Status_ >This document is a rough draft and work in progress. We plan to @@ -32,7 +31,7 @@ to provide a universal data representation for all of these different approaches Also, while we are not currently focused on building a SQL engine for the Zed lake, it is most certainly possible to do so, as a Zed record type -[is analagous to](../formats/README.md#2-a-super-structured-pattern) +[is analagous to](../formats/_index.md#2-a-super-structured-pattern) a SQL table definition. SQL tables can essentially be dynamically projected via a table virtualization layer built on top of the Zed lake model. @@ -69,7 +68,7 @@ such an assumption). #### Data Objects A data object is created by a single writer using a globally unique name -with an embedded KSUID. +with an embedded KSUID. New objects are written in their entirety. No updates, appends, or modifications may be made once an object exists. Given these semantics, any such object may be diff --git a/docs/language/README.md b/docs/language/_index.md similarity index 71% rename from docs/language/README.md rename to docs/language/_index.md index 2bcd9ab64d..303b399ad1 100644 --- a/docs/language/README.md +++ b/docs/language/_index.md @@ -1,4 +1,8 @@ -# The Zed Language +--- +title: Language +weight: 4 +heading: The Zed Language +--- The language documents: * provide an [overview](overview.md) of the Zed language, @@ -8,5 +12,5 @@ The language documents: * describe the syntax of [expressions](expressions.md) and [search expressions](search-expressions.md), * explain [lateral subqueries](lateral-subqueries.md), * describe [shaping and type fusion](shaping.md), and -* enumerate the [operators](operators/README.md), [functions](functions/README.md), -and [aggregate functions](aggregates/README.md) in reference format. +* enumerate the [operators](operators/_index.md), [functions](functions/_index.md), +and [aggregate functions](aggregates/_index.md) in reference format. diff --git a/docs/language/aggregates/README.md b/docs/language/aggregates/_index.md similarity index 95% rename from docs/language/aggregates/README.md rename to docs/language/aggregates/_index.md index 86a5f8488c..1513f23bad 100644 --- a/docs/language/aggregates/README.md +++ b/docs/language/aggregates/_index.md @@ -1,5 +1,6 @@ -# Aggregate Functions - +--- +weight: 3 +title: Aggregate Functions --- Aggregate functions appear in either [summarization](../operators/summarize.md) diff --git a/docs/language/aggregates/and.md b/docs/language/aggregates/and.md index 1de11e15b6..2f6666fda3 100644 --- a/docs/language/aggregates/and.md +++ b/docs/language/aggregates/and.md @@ -14,31 +14,42 @@ The _and_ aggregate function computes the logical AND over all of its input. ### Examples Anded value of simple sequence: -```mdtest-command -echo 'true false true' | super -z -c 'and(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +and(this) +# input +true +false +true +# expected output false ``` Continuous AND of simple sequence: -```mdtest-command -echo 'true false true' | super -z -c 'yield and(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield and(this) +# input +true +false +true +# expected output true false false ``` Unrecognized types are ignored and not coerced for truthiness: -```mdtest-command -echo 'true "foo" 0 false true' | super -z -c 'yield and(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield and(this) +# input +true +"foo" +0 +false +true +# expected output true true true @@ -47,12 +58,15 @@ false ``` AND of values grouped by key: -```mdtest-command -echo '{a:true,k:1} {a:true,k:1} {a:true,k:2} {a:false,k:2}' | - super -z -c 'and(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +and(a) by k | sort +# input +{a:true,k:1} +{a:true,k:1} +{a:true,k:2} +{a:false,k:2} +# expected output {k:1,and:true} {k:2,and:false} ``` diff --git a/docs/language/aggregates/any.md b/docs/language/aggregates/any.md index 5c866ac46c..90be37f87f 100644 --- a/docs/language/aggregates/any.md +++ b/docs/language/aggregates/any.md @@ -15,20 +15,25 @@ The semantics of how the item is selected is not defined. ### Examples Any picks the first one in this scenario but this behavior is undefined: -```mdtest-command -echo '1 2 3 4' | super -z -c 'any(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +any(this) +# input +1 +2 +3 +4 +# expected output 1 ``` Continuous any over a simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'yield any(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield any(this) +# input +1 2 3 4 +# expected output 1 1 1 @@ -36,21 +41,28 @@ echo '1 2 3 4' | super -z -c 'yield any(this)' - ``` Any is not sensitive to mixed types as it just picks one: -```mdtest-command -echo '"foo" 1 2 3 ' | super -z -c 'any(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +any(this) +# input +"foo" +1 +2 +3 +# expected output "foo" ``` Pick from groups bucketed by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2} {a:4,k:2}' | - super -z -c 'any(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +any(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +{a:4,k:2} +# expected output {k:1,any:1} {k:2,any:3} ``` diff --git a/docs/language/aggregates/avg.md b/docs/language/aggregates/avg.md index 368cccbd10..1d010eb2fe 100644 --- a/docs/language/aggregates/avg.md +++ b/docs/language/aggregates/avg.md @@ -14,20 +14,28 @@ The _avg_ aggregate function computes the mathematical average value of its inpu ### Examples Average value of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'avg(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +avg(this) +# input +1 +2 +3 +4 +# expected output 2.5 ``` Continuous average of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'yield avg(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield avg(this) +# input +1 +2 +3 +4 +# expected output 1. 1.5 2. @@ -35,21 +43,29 @@ echo '1 2 3 4' | super -z -c 'yield avg(this)' - ``` Unrecognized types are ignored: -```mdtest-command -echo '1 2 3 4 "foo"' | super -z -c 'avg(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +avg(this) +# input +1 +2 +3 +4 +"foo" +# expected output 2.5 ``` Average of values bucketed by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2} {a:4,k:2}' | - super -z -c 'avg(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +avg(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +{a:4,k:2} +# expected output {k:1,avg:1.5} {k:2,avg:3.5} ``` diff --git a/docs/language/aggregates/collect.md b/docs/language/aggregates/collect.md index 74c4cd3e6d..9dff9741d7 100644 --- a/docs/language/aggregates/collect.md +++ b/docs/language/aggregates/collect.md @@ -16,20 +16,28 @@ of union of the types encountered. ### Examples Simple sequence collected into an array: -```mdtest-command -echo '1 2 3 4' | super -z -c 'collect(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +collect(this) +# input +1 +2 +3 +4 +# expected output [1,2,3,4] ``` Continuous collection over a simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'yield collect(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield collect(this) +# input +1 +2 +3 +4 +# expected output [1] [1,2] [1,2,3] @@ -37,21 +45,29 @@ echo '1 2 3 4' | super -z -c 'yield collect(this)' - ``` Mixed types create a union type for the array elements: -```mdtest-command -echo '1 2 3 4 "foo"' | super -z -c 'collect(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +collect(this) +# input +1 +2 +3 +4 +"foo" +# expected output [1,2,3,4,"foo"] ``` Create arrays of values bucketed by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2} {a:4,k:2}' | - super -z -c 'collect(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +collect(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +{a:4,k:2} +# expected output {k:1,collect:[1,2]} {k:2,collect:[3,4]} ``` diff --git a/docs/language/aggregates/collect_map.md b/docs/language/aggregates/collect_map.md index 3032ac13b0..cb1be6ac60 100644 --- a/docs/language/aggregates/collect_map.md +++ b/docs/language/aggregates/collect_map.md @@ -17,37 +17,40 @@ of union of those types. ### Examples Combine a sequence of records into a map: -```mdtest-command -echo '{stock:"APPL",price:145.03} {stock:"GOOG",price:87.07}' | - super -z -c 'collect_map(|{stock:price}|)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +collect_map(|{stock:price}|) +# input +{stock:"APPL",price:145.03} +{stock:"GOOG",price:87.07} +# expected output |{"APPL":145.03,"GOOG":87.07}| ``` Continuous collection over a simple sequence: -```mdtest-command -echo '|{"APPL":145.03}| |{"GOOG":87.07}| |{"APPL":150.13}|' | - super -z -c 'yield collect_map(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield collect_map(this) +# input +|{"APPL":145.03}| +|{"GOOG":87.07}| +|{"APPL":150.13}| +# expected output |{"APPL":145.03}| |{"APPL":145.03,"GOOG":87.07}| |{"APPL":150.13,"GOOG":87.07}| ``` Create maps by key: -```mdtest-command -echo '{stock:"APPL",price:145.03,day:0} - {stock:"GOOG",price:87.07,day:0} - {stock:"APPL",price:150.13,day:1} - {stock:"GOOG",price:89.15,day:1}' | - super -z -c 'collect_map(|{stock:price}|) by day |> sort' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +collect_map(|{stock:price}|) by day | sort +# input +{stock:"APPL",price:145.03,day:0} +{stock:"GOOG",price:87.07,day:0} +{stock:"APPL",price:150.13,day:1} +{stock:"GOOG",price:89.15,day:1} +# expected output {day:0,collect_map:|{"APPL":145.03,"GOOG":87.07}|} {day:1,collect_map:|{"APPL":150.13,"GOOG":89.15}|} ``` diff --git a/docs/language/aggregates/count.md b/docs/language/aggregates/count.md index 22a25b27ee..1c808fdd07 100644 --- a/docs/language/aggregates/count.md +++ b/docs/language/aggregates/count.md @@ -14,68 +14,87 @@ The _count_ aggregate function computes the number of values in its input. ### Examples Count of values in a simple sequence: -```mdtest-command -echo '1 2 3' | super -z -c 'count()' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +count() +# input +1 +2 +3 +# expected output 3(uint64) ``` Continuous count of simple sequence: -```mdtest-command -echo '1 2 3' | super -z -c 'yield count()' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield count() +# input +1 +2 +3 +# expected output 1(uint64) 2(uint64) 3(uint64) ``` Mixed types are handled: -```mdtest-command -echo '1 "foo" 10.0.0.1' | super -z -c 'yield count()' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield count() +# input +1 +"foo" +10.0.0.1 +# expected output 1(uint64) 2(uint64) 3(uint64) ``` Count of values in buckets grouped by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2}' | super -z -c 'count() by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +count() by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +# expected output {k:1,count:2(uint64)} {k:2,count:1(uint64)} ``` A simple count with no input values returns no output: -```mdtest-command -echo '1 "foo" 10.0.0.1' | super -z -c 'where grep("bar") |> count()' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +where grep("bar") | count() +# input +1 +"foo" +10.0.0.1 +# expected output ``` Count can return an explicit zero when using a `where` clause in the aggregation: -```mdtest-command -echo '1 "foo" 10.0.0.1' | super -z -c 'count() where grep("bar")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +count() where grep("bar") +# input +1 +"foo" +10.0.0.1 +# expected output 0(uint64) ``` Note that the number of input values are counted, unlike the [`len` function](../functions/len.md) which counts the number of elements in a given value: -```mdtest-command -echo '[1,2,3]' | super -z -c 'count()' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +count() +# input +[1,2,3] +# expected output 1(uint64) ``` diff --git a/docs/language/aggregates/dcount.md b/docs/language/aggregates/dcount.md index 52357046bd..8ea0aa048e 100644 --- a/docs/language/aggregates/dcount.md +++ b/docs/language/aggregates/dcount.md @@ -4,7 +4,7 @@ ### Synopsis ``` -dcount() -> uint64 +dcount(any) -> uint64 ``` ### Description @@ -15,20 +15,28 @@ of the input in a memory efficient manner. ### Examples Count of values in a simple sequence: -```mdtest-command -echo '1 2 2 3' | super -z -c 'dcount(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +dcount(this) +# input +1 +2 +2 +3 +# expected output 3(uint64) ``` Continuous count of simple sequence: -```mdtest-command -echo '1 2 2 3' | super -z -c 'yield dcount(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield dcount(this) +# input +1 +2 +2 +3 +# expected output 1(uint64) 2(uint64) 2(uint64) @@ -36,11 +44,14 @@ echo '1 2 2 3' | super -z -c 'yield dcount(this)' - ``` Mixed types are handled: -```mdtest-command -echo '1 "foo" 10.0.0.1' | super -z -c 'yield dcount(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield dcount(this) +# input +1 +"foo" +10.0.0.1 +# expected output 1(uint64) 2(uint64) 3(uint64) @@ -56,11 +67,14 @@ seq 10000 | super -z -c 'dcount(this)' - ``` Count of values in buckets grouped by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2}' | super -z -c 'dcount(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +dcount(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +# expected output {k:1,dcount:2(uint64)} {k:2,dcount:1(uint64)} ``` diff --git a/docs/language/aggregates/fuse.md b/docs/language/aggregates/fuse.md index 498c856bcf..f07818e835 100644 --- a/docs/language/aggregates/fuse.md +++ b/docs/language/aggregates/fuse.md @@ -12,27 +12,32 @@ fuse(any) -> type The _fuse_ aggregate function applies [type fusion](../shaping.md#type-fusion) to its input and returns the fused type. -This aggregation is useful with group-by for data exploration and discovery +This aggregation is useful with group-by for data exploration and discovery when searching for shaping rules to cluster a large number of varied input types to a smaller number of fused types each from a set of interrelated types. ### Examples Fuse two records: -```mdtest-command -echo '{a:1,b:2}{a:2,b:"foo"}' | super -z -c 'fuse(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +fuse(this) +# input +{a:1,b:2} +{a:2,b:"foo"} +# expected output <{a:int64,b:(int64,string)}> ``` + Fuse records with a group-by key: -```mdtest-command -echo '{a:1,b:"bar"}{a:2.1,b:"foo"}{a:3,b:"bar"}' | - super -z -c 'fuse(this) by b |> sort' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +fuse(this) by b | sort +# input +{a:1,b:"bar"} +{a:2.1,b:"foo"} +{a:3,b:"bar"} +# expected output {b:"bar",fuse:<{a:int64,b:string}>} {b:"foo",fuse:<{a:float64,b:string}>} ``` diff --git a/docs/language/aggregates/max.md b/docs/language/aggregates/max.md index 4b672e3ffe..24463475f9 100644 --- a/docs/language/aggregates/max.md +++ b/docs/language/aggregates/max.md @@ -14,20 +14,25 @@ The _max_ aggregate function computes the maximum value of its input. ### Examples Maximum value of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'max(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +max(this) +# input +1 2 3 4 +# expected output 4 ``` Continuous maximum of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'yield max(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield max(this) +# input +1 +2 +3 +4 +# expected output 1 2 3 @@ -35,21 +40,29 @@ echo '1 2 3 4' | super -z -c 'yield max(this)' - ``` Unrecognized types are ignored: -```mdtest-command -echo '1 2 3 4 "foo"' | super -z -c 'max(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +max(this) +# input +1 +2 +3 +4 +"foo" +# expected output 4 ``` Maximum value within buckets grouped by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2} {a:4,k:2}' | - super -z -c 'max(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +max(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +{a:4,k:2} +# expected output {k:1,max:2} {k:2,max:4} ``` diff --git a/docs/language/aggregates/min.md b/docs/language/aggregates/min.md index 06a8786494..8c480e378d 100644 --- a/docs/language/aggregates/min.md +++ b/docs/language/aggregates/min.md @@ -4,7 +4,7 @@ ### Synopsis ``` -min(...number) -> number +min(number) -> number ``` ### Description @@ -14,20 +14,28 @@ The _min_ aggregate function computes the minimum value of its input. ### Examples Minimum value of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'min(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +min(this) +# input +1 +2 +3 +4 +# expected output 1 ``` Continuous minimum of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'yield min(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield min(this) +# input +1 +2 +3 +4 +# expected output 1 1 1 @@ -35,21 +43,29 @@ echo '1 2 3 4' | super -z -c 'yield min(this)' - ``` Unrecognized types are ignored: -```mdtest-command -echo '1 2 3 4 "foo"' | super -z -c 'min(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +min(this) +# input +1 +2 +3 +4 +"foo" +# expected output 1 ``` Minimum value within buckets grouped by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2} {a:4,k:2}' | - super -z -c 'min(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +min(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +{a:4,k:2} +# expected output {k:1,min:1} {k:2,min:3} ``` diff --git a/docs/language/aggregates/or.md b/docs/language/aggregates/or.md index 884cca4e41..2afe11f54d 100644 --- a/docs/language/aggregates/or.md +++ b/docs/language/aggregates/or.md @@ -14,31 +14,42 @@ The _or_ aggregate function computes the logical OR over all of its input. ### Examples Ored value of simple sequence: -```mdtest-command -echo 'false true false' | super -z -c 'or(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +or(this) +# input +false +true +false +# expected output true ``` Continuous OR of simple sequence: -```mdtest-command -echo 'false true false' | super -z -c 'yield or(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield or(this) +# input +false +true +false +# expected output false true true ``` Unrecognized types are ignored and not coerced for truthiness: -```mdtest-command -echo 'false "foo" 1 true false' | super -z -c 'yield or(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield or(this) +# input +false +"foo" +1 +true +false +# expected output false false false @@ -47,12 +58,15 @@ true ``` OR of values grouped by key: -```mdtest-command -echo '{a:true,k:1} {a:false,k:1} {a:false,k:2} {a:false,k:2}' | - super -z -c 'or(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +or(a) by k | sort +# input +{a:true,k:1} +{a:false,k:1} +{a:false,k:2} +{a:false,k:2} +# expected output {k:1,or:true} {k:2,or:false} ``` diff --git a/docs/language/aggregates/sum.md b/docs/language/aggregates/sum.md index 00dfd41e0f..fc757f96c6 100644 --- a/docs/language/aggregates/sum.md +++ b/docs/language/aggregates/sum.md @@ -14,20 +14,28 @@ The _sum_ aggregate function computes the mathematical sum of its input. ### Examples Sum of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'sum(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +sum(this) +# input +1 +2 +3 +4 +# expected output 10 ``` Continuous sum of simple sequence: -```mdtest-command -echo '1 2 3 4' | super -z -c 'yield sum(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield sum(this) +# input +1 +2 +3 +4 +# expected output 1 3 6 @@ -35,21 +43,29 @@ echo '1 2 3 4' | super -z -c 'yield sum(this)' - ``` Unrecognized types are ignored: -```mdtest-command -echo '1 2 3 4 "foo"' | super -z -c 'sum(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +sum(this) +# input +1 +2 +3 +4 +"foo" +# expected output 10 ``` Sum of values bucketed by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2} {a:4,k:2}' | - super -z -c 'sum(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +sum(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +{a:4,k:2} +# expected output {k:1,sum:3} {k:2,sum:7} ``` diff --git a/docs/language/aggregates/union.md b/docs/language/aggregates/union.md index 4265fa1da9..fde1795145 100644 --- a/docs/language/aggregates/union.md +++ b/docs/language/aggregates/union.md @@ -17,20 +17,28 @@ types encountered. ### Examples Create a set of values from a simple sequence: -```mdtest-command -echo '1 2 3 3' | super -z -c 'union(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +union(this) +# input +1 +2 +3 +3 +# expected output |[1,2,3]| ``` Create sets continuously from values in a simple sequence: -```mdtest-command -echo '1 2 3 3' | super -z -c 'yield union(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield union(this) +# input +1 +2 +3 +3 +# expected output |[1]| |[1,2]| |[1,2,3]| @@ -38,22 +46,29 @@ echo '1 2 3 3' | super -z -c 'yield union(this)' - ``` Mixed types create a union type for the set elements: -```mdtest-command -echo '1 2 3 "foo"' | super -z -c 'set:=union(this) |> yield this,typeof(set)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +set:=union(this) | yield this,typeof(set) +# input +1 +2 +3 +"foo" +# expected output {set:|[1,2,3,"foo"]|} <|[(int64,string)]|> ``` Create sets of values bucketed by key: -```mdtest-command -echo '{a:1,k:1} {a:2,k:1} {a:3,k:2} {a:4,k:2}' | - super -z -c 'union(a) by k |> sort' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +union(a) by k | sort +# input +{a:1,k:1} +{a:2,k:1} +{a:3,k:2} +{a:4,k:2} +# expected output {k:1,union:|[1,2]|} {k:2,union:|[3,4]|} ``` diff --git a/docs/language/conventions.md b/docs/language/conventions.md index 39d5743eca..5e9823a520 100644 --- a/docs/language/conventions.md +++ b/docs/language/conventions.md @@ -1,11 +1,10 @@ --- -sidebar_position: 9 -sidebar_label: Conventions +weight: 9 +title: Conventions +heading: Type Conventions --- -# Type Conventions - -[Function](functions/README.md) arguments and [operator](operators/README.md) input values are all dynamically typed, +[Function](functions/_index.md) arguments and [operator](operators/_index.md) input values are all dynamically typed, yet certain functions expect certain specific [data types](data-types.md) or classes of data types. To this end, the function and operator prototypes in the Zed documentation include several type classes as follows: diff --git a/docs/language/data-types.md b/docs/language/data-types.md index 44d4969562..5f3e1b87b6 100644 --- a/docs/language/data-types.md +++ b/docs/language/data-types.md @@ -1,10 +1,8 @@ --- -sidebar_position: 3 -sidebar_label: Data Types +weight: 3 +title: Data Types --- -# Data Types - The SuperPipe language includes most data types of a typical programming language as defined in the [super data model](../formats/zed.md). @@ -54,20 +52,25 @@ serve as group-by keys or be used in ["data shaping"](shaping.md) logic. A common workflow for data introspection is to first perform a search of exploratory data and then count the shapes of each type of data as follows: ``` -search ... |> count() by typeof(this) +search ... | count() by typeof(this) ``` For example, -```mdtest-command -echo '1 2 "foo" 10.0.0.1 ' | - super -z -c 'count() by typeof(this) |> sort this' - -``` -produces -```mdtest-output +``` mdtest-spq +# spq +count() by typeof(this) | sort this +# input +1 +2 +"foo" +10.0.0.1 + +# expected output {typeof:,count:2(uint64)} {typeof:,count:1(uint64)} {typeof:,count:1(uint64)} {typeof:,count:1(uint64)} ``` + When running such a query over complex, semi-structured data, the results can be quite illuminating and can inform the design of "data shaping" queries to transform raw, messy data into clean data for downstream tooling. @@ -109,51 +112,64 @@ Named types may also be defined by the input data itself, as super data is comprehensively self describing. When named types are defined in the input data, there is no need to declare their type in a query. -In this case, a SuperPipe expression may refer to the type by the name that simply -appears to the runtime as a side effect of operating upon the data. If the type -name referred to in this way does not exist, then the type value reference -results in `error("missing")`. For example, -```mdtest-command -echo '1(=foo) 2(=bar) 3(=foo)' | super -z -c 'typeof(this)==' - -``` -results in -```mdtest-output +In this case, a SuperSQL expression may refer to the type by the name that simply +appears to the runtime as a side effect of operating upon the data, e.g., + +```mdtest-spq +# spq +typeof(this)== +# input +1(=foo) +2(=bar) +3(=foo) +# expected output 1(=foo) 3(=foo) ``` + and -```mdtest-command -echo '1(=foo)' | super -z -c 'yield ' - -``` -results in -```mdtest-output + +```mdtest-spq +# spq +yield +# input +1(=foo) +# expected output ``` -but -```mdtest-command -super -z -c 'yield ' -``` -gives -```mdtest-output + +If the type name referred to in this way does not exist, then the type value +reference results in `error("missing")`. For example, +```mdtest-spq +# spq +yield +# input +1 +# expected output error("missing") ``` + Each instance of a named type definition overrides any earlier definition. In this way, types are local in scope. Each value that references a named type retains its local definition of the named type retaining the proper type binding while accommodating changes in a particular named type. For example, -```mdtest-command -echo '1(=foo) 2(=bar) "hello"(=foo) 3(=foo)' | - super -z -c 'count() by typeof(this) |> sort this' - -``` -results in -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +count() by typeof(this) | sort this +# input +1(=foo) +2(=bar) +"hello"(=foo) +3(=foo) +# expected output {typeof:,count:1(uint64)} {typeof:,count:2(uint64)} {typeof:,count:1(uint64)} ``` -Here, the two versions of type "foo" are retained in the group-by results. + +Here, the two versions of type "foo" were retained in the group-by results. In general, it is bad practice to define multiple versions of a single named type, though the SuperDB system and super data model accommodate such dynamic bindings. @@ -163,7 +179,7 @@ the scope of the Zed data model and language. That said, Zed provides flexible building blocks so systems can define their own schema versioning and schema management policies on top of these Zed primitives. -The [super-structured data model](../formats/README.md#2-a-super-structured-pattern) +The [super-structured data model](../formats/_index.md#2-a-super-structured-pattern) is a superset of relational tables and SuperPipe's type system can easily make this connection. As an example, consider this type definition for "employee": @@ -216,34 +232,42 @@ logs to find out what happened. For example, input values can be transformed to errors as follows: -```mdtest-command -echo '0 "foo" 10.0.0.1' | super -z -c 'error(this)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +error(this) +# input +0 +"foo" +10.0.0.1 +# expected output error(0) error("foo") error(10.0.0.1) ``` + More practically, errors from the runtime show up as error values. For example, -```mdtest-command -echo 0 | super -z -c '1/this' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +1/this +# input +0 +# expected output error("divide by zero") ``` + And since errors are first-class and just values, they have a type. In particular, they are a complex type where the error value's type is the complex type `error` containing the type of the value. For example, -```mdtest-command -echo 0 | super -z -c 'typeof(1/this)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +typeof(1/this) +# input +0 +# expected output ``` + First-class errors are particularly useful for creating structured errors. When a SuperPipe query encounters a problematic condition, instead of silently dropping the problematic error @@ -315,7 +339,7 @@ results from accessing a field that is not present. Thus, `x==NULL` and `x==MISSING` could disambiguate the two cases above. SuperPipe, instead, recognizes that the SQL value `MISSING` is a paradox: -I'm here but I'm not. +I'm here but I'm not. In reality, a `MISSING` value is not a value. It's an error condition that resulted from trying to reference something that didn't exist. @@ -326,34 +350,41 @@ approach because it lacks first-class errors. But SuperPipe has first-class errors so a reference to something that does not exist is an error of type `error(string)` whose value is `error("missing")`. For example, -```mdtest-command -echo "{x:1} {y:2}" | super -z -c 'yield x' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield x +# input +{x:1} +{y:2} +# expected output 1 error("missing") ``` + Sometimes you want missing errors to show up and sometimes you don't. The [`quiet` function](functions/quiet.md) transforms missing errors into "quiet errors". A quiet error is the value `error("quiet")` and is ignored by most operators, in particular `yield`. For example, -```mdtest-command -echo "{x:1} {y:2}" | super -z -c "yield quiet(x)" - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield quiet(x) +# input +{x:1} +{y:2} +# expected output 1 ``` -And what if you want a default value instead of a missing error? The +And what if you want a default value instead of a "missing" error? The [`coalesce` function](functions/coalesce.md) returns the first value that is not null, `error("missing")`, or `error("quiet")`. For example, -```mdtest-command -echo "{x:1} {y:2}" | super -z -c "yield coalesce(x, 0)" - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield coalesce(x, 0) +# input +{x:1} +{y:2} +# expected output 1 0 ``` diff --git a/docs/language/expressions.md b/docs/language/expressions.md index b2257b9eab..87c67c3c5e 100644 --- a/docs/language/expressions.md +++ b/docs/language/expressions.md @@ -1,10 +1,8 @@ --- -sidebar_position: 5 -sidebar_label: Expressions +weight: 5 +title: Expressions --- -# Expressions - Zed expressions follow the typical patterns in programming languages. Expressions are typically used within pipeline operators to perform computations on input values and are typically evaluated once per each @@ -20,11 +18,12 @@ and semantics and are left-associative with multiplication and division having precedence over addition and subtraction. `%` is the modulo operator. For example, -```mdtest-command -super -z -c 'yield 2*3+1, 11%5, 1/0, "foo"+"bar"' -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield 2*3+1, 11%5, 1/0, "foo"+"bar" +# input +null +# expected output 7 1 error("divide by zero") @@ -45,12 +44,12 @@ If either operand to a comparison is `error("missing")`, then the result is `error("missing")`. For example, -```mdtest-command -super -z -c 'yield 1 > 2, 1 < 2, "b" > "a", 1 > "a", 1 > x' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield 1 > 2, 1 < 2, "b" > "a", 1 > "a", 1 > x +# input +null +# expected output false true true @@ -70,21 +69,25 @@ The right-hand side value can be any Zed value and complex values are recursively traversed to determine if the item is present anywhere within them. For example, -```mdtest-command -echo '{a:[1,2]}{b:{c:3}}{d:{e:1}}' | super -z -c '1 in this' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +1 in this +# input +{a:[1,2]} +{b:{c:3}} +{d:{e:1}} +# expected output {a:[1,2]} {d:{e:1}} ``` + You can also use this operator with a static array: -```mdtest-command -echo '{accounts:[{id:1},{id:2},{id:3}]}' | - super -z -c 'over accounts |> where id in [1,2]' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +over accounts | where id in [1,2] +# input +{accounts:[{id:1},{id:2},{id:3}]} +# expected output {id:1} {id:2} ``` @@ -186,11 +189,13 @@ the result. Otherwise, the second `` expression is evaluated and becomes the result. For example, -```mdtest-command -echo '{s:"foo",v:1}{s:"bar",v:2}' | super -z -c 'yield (s=="foo") ? v : -v' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield (s=="foo") ? v : -v +# input +{s:"foo",v:1} +{s:"bar",v:2} +# expected output 1 -2 ``` @@ -199,12 +204,14 @@ Conditional expressions can be chained, providing behavior equivalent to "else if" as appears in other languages. For example, -```mdtest-command -echo '{s:"foo",v:1}{s:"bar",v:2}{s:"baz",v:3}' | - super -z -c 'yield (s=="foo") ? v : (s=="bar") ? -v : v*v' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield (s=="foo") ? v : (s=="bar") ? -v : v*v +# input +{s:"foo",v:1} +{s:"bar",v:2} +{s:"baz",v:3} +# expected output 1 -2 9 @@ -215,12 +222,14 @@ as with [aggregate function calls](expressions.md#aggregate-function-calls), onl will be evaluated. For example, -```mdtest-command -echo '"foo" "bar" "foo"' | - super -z -c 'yield this=="foo" ? {foocount:count()} : {barcount:count()}' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield this=="foo" ? {foocount:count()} : {barcount:count()} +# input +"foo" +"bar" +"foo" +# expected output {foocount:1(uint64)} {barcount:1(uint64)} {foocount:2(uint64)} @@ -232,24 +241,25 @@ Functions perform stateless transformations of their input value to their return value and utilize call-by value semantics with positional and unnamed arguments. For example, -```mdtest-command -super -z -c 'yield pow(2,3), lower("ABC")+upper("def"), typeof(1)' -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield pow(2,3), lower("ABC")+upper("def"), typeof(1) +# input +null +# expected output 8. "abcDEF" ``` -Zed includes many [built-in functions](functions/README.md), some of which take -a variable number of arguments. +Zed includes many [built-in functions](functions/_index.md), some of which take +a variable number of arguments. Zed also allows you to create [user-defined functions](statements.md#func-statements). ## Aggregate Function Calls -[Aggregate functions](aggregates/README.md) may be called within an expression. +[Aggregate functions](aggregates/_index.md) may be called within an expression. Unlike the aggregation context provided by a [summarizing group-by](operators/summarize.md), such calls in expression context yield an output value for each input value. @@ -259,21 +269,29 @@ optimizer from parallelizing a query. That said, aggregate function calls can be quite useful in a number of contexts. For example, a unique ID can be assigned to the input quite easily: -```mdtest-command -echo '"foo" "bar" "baz"' | super -z -c 'yield {id:count(),value:this}' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield {id:count(),value:this} +# input +"foo" +"bar" +"baz" +# expected output {id:1(uint64),value:"foo"} {id:2(uint64),value:"bar"} {id:3(uint64),value:"baz"} ``` + In contrast, calling aggregate functions within the [`summarize` operator](operators/summarize.md) -```mdtest-command -echo '"foo" "bar" "baz"' | super -z -c 'summarize count(),union(this)' - -``` -produces just one output value -```mdtest-output +produces just one output value. +```mdtest-spq {data-layout="stacked"} +# spq +summarize count(),union(this) +# input +"foo" +"bar" +"baz" +# expected output {count:3(uint64),union:|["bar","baz","foo"]|} ``` @@ -299,12 +317,12 @@ with the result of evaluating the expression ``. If this result is not a string, it is implicitly cast to a string. For example, -```mdtest-command -echo '{numerator:22.0, denominator:7.0}' | - super -z -c 'yield f"pi is approximately {numerator / denominator}"' - -``` -produces -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield f"pi is approximately {numerator / denominator}" +# input +{numerator:22.0, denominator:7.0} +# expected output "pi is approximately 3.142857142857143" ``` @@ -318,12 +336,12 @@ first error encountered in left-to-right order. F-strings may be nested, where a child `` may contain f-strings. For example, -```mdtest-command -echo '{foo:"hello", bar:"world", HELLOWORLD:"hi!"}' | - super -z -c 'yield f"oh {this[upper(f"{foo + bar}")]}"' - -``` -produces -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield f"oh {this[upper(f"{foo + bar}")]}" +# input +{foo:"hello", bar:"world", HELLOWORLD:"hi!"} +# expected output "oh hi!" ``` @@ -331,11 +349,12 @@ To represent a literal `{` character inside an f-string, it must be escaped, i.e., `\{`. For example, -```mdtest-command -echo '"brackets"' | super -z -c 'yield f"{this} look like: \{ }"' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield f"{this} look like: \{ }" +# input +"brackets" +# expected output "brackets look like: { }" ``` @@ -365,12 +384,12 @@ field names collide the rightmost instance of the name determines that field's value. For example, -```mdtest-command -echo '{x:1,y:2,r:{a:1,b:2}}' | - super -z -c 'yield {a:0},{x}, {...r}, {a:0,...r,b:3}' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield {a:0},{x}, {...r}, {a:0,...r,b:3} +# input +{x:1,y:2,r:{a:1,b:2}} +# expected output {a:0} {x:1} {a:1,b:2} @@ -398,21 +417,23 @@ When the expressions result in values of non-uniform type, then the implied type of the array is an array of type `union` of the types that appear. For example, -```mdtest-command -super -z -c 'yield [1,2,3],["hello","world"]' -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield [1,2,3],["hello","world"] +# input +null +# expected output [1,2,3] ["hello","world"] ``` Arrays can be concatenated using the spread operator, -```mdtest-command -echo '{a:[1,2],b:[3,4]}' | super -z -c 'yield [...a,...b,5]' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield [...a,...b,5] +# input +{a:[1,2],b:[3,4]} +# expected output [1,2,3,4,5] ``` @@ -440,21 +461,23 @@ Set values are always organized in their "natural order" independent of the orde they appear in the set literal. For example, -```mdtest-command -super -z -c 'yield |[3,1,2]|,|["hello","world","hello"]|' -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield |[3,1,2]|,|["hello","world","hello"]| +# input +null +# expected output |[1,2,3]| |["hello","world"]| ``` Arrays and sets can be concatenated using the spread operator, -```mdtest-command -echo '{a:[1,2],b:|[2,3]|}' | super -z -c 'yield |[...a,...b,4]|' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield |[...a,...b,4]| +# input +{a:[1,2],b:|[2,3]|} +# expected output |[1,2,3,4]| ``` @@ -471,11 +494,12 @@ then the implied type of the map has a key type and/or value type that is a union of the types that appear in each respective category. For example, -```mdtest-command -super -z -c 'yield |{"foo":1,"bar"+"baz":2+3}|' -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield |{"foo":1,"bar"+"baz":2+3}| +# input +null +# expected output |{"foo":1,"barbaz":5}| ``` @@ -486,33 +510,39 @@ and `string` is expressed as `(int64,string)` and any value that has a type that appears in the union type may be cast to that union type. Since 1 is an `int64` and "foo" is a `string`, they both can be values of type `(int64,string)`, e.g., -```mdtest-command -echo '1 "foo"' | super -z -c 'yield cast(this,<(int64,string)>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield cast(this,<(int64,string)>) +# input +1 +"foo" +# expected output 1((int64,string)) "foo"((int64,string)) ``` + The value underlying a union-tagged value is accessed with the [`under` function](functions/under.md): -```mdtest-command -echo '1((int64,string))' | super -z -c 'yield under(this)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield under(this) +# input +1((int64,string)) +# expected output 1 ``` + Union values are powerful because they provide a mechanism to precisely describe the type of any nested, semi-structured value composed of elements of different types. For example, the type of the value `[1,"foo"]` in JavaScript is simply a generic JavaScript "object". But in Zed, the type of this value is an array of union of string and integer, e.g., -```mdtest-command -echo '[1,"foo"]' | super -z -c 'typeof(this)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +typeof(this) +# input +[1,"foo"] +# expected output <[(int64,string)]> ``` @@ -531,11 +561,15 @@ If the result of `` cannot be converted to the indicated type, then the cast's result is an error value. For example, -```mdtest-command -echo '1 200 "123" "200"' | super -z -c 'yield int8(this)' - -``` -produces -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield int8(this) +# input +1 +200 +"123" +"200" +# expected output 1(int8) error({message:"cannot cast to int8",on:200}) 123(int8) @@ -546,11 +580,13 @@ Casting attempts to be fairly liberal in conversions. For example, values of type `time` can be created from a diverse set of date/time input strings based on the [Go Date Parser library](https://github.com/araddon/dateparse). -```mdtest-command -echo '"May 8, 2009 5:57:51 PM" "oct 7, 1970"' | super -z -c 'yield time(this)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +yield time(this) +# input +"May 8, 2009 5:57:51 PM" +"oct 7, 1970" +# expected output 2009-05-08T17:57:51Z 1970-10-07T00:00:00Z ``` @@ -562,11 +598,15 @@ either in functional form or with `cast`: cast(, ) ``` For example -```mdtest-command -echo '80 8080' | super -z -c 'type port = uint16 yield (this)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +type port = uint16 + +yield (this) +# input +80 +8080 +# expected output 80(port=uint16) 8080(port=uint16) ``` @@ -574,20 +614,23 @@ produces Casts may be used with complex types as well. As long as the target type can accommodate the value, the case will be recursively applied to the components of a nested value. For example, -```mdtest-command -echo '["10.0.0.1","10.0.0.2"]' | super -z -c 'cast(this,<[ip]>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +cast(this,<[ip]>) +# input +["10.0.0.1","10.0.0.2"] +# expected output [10.0.0.1,10.0.0.2] ``` + and -```mdtest-command -echo '{ts:"1/1/2022",r:{x:"1",y:"2"}} {ts:"1/2/2022",r:{x:3,y:4}}' | - super -z -c 'cast(this,<{ts:time,r:{x:float64,y:float64}}>)' - -``` -produces -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +cast(this,<{ts:time,r:{x:float64,y:float64}}>) +# input +{ts:"1/1/2022",r:{x:"1",y:"2"}} +{ts:"1/2/2022",r:{x:3,y:4}} +# expected output {ts:2022-01-01T00:00:00Z,r:{x:1.,y:2.}} {ts:2022-01-02T00:00:00Z,r:{x:3.,y:4.}} ``` diff --git a/docs/language/functions/README.md b/docs/language/functions/_index.md similarity index 99% rename from docs/language/functions/README.md rename to docs/language/functions/_index.md index 65280e0848..99ad7fd0c4 100644 --- a/docs/language/functions/README.md +++ b/docs/language/functions/_index.md @@ -1,5 +1,6 @@ -# Functions - +--- +weight: 2 +title: Functions --- Functions appear in [expression](../expressions.md) context and diff --git a/docs/language/functions/abs.md b/docs/language/functions/abs.md index 7bef9ce3bb..e03f0cdac1 100644 --- a/docs/language/functions/abs.md +++ b/docs/language/functions/abs.md @@ -15,12 +15,19 @@ must be a numeric type. ### Examples -Absolute value of a various numbers: -```mdtest-command -echo '1 -1 0 -1.0 -1(int8) 1(uint8) "foo"' | super -z -c 'yield abs(this)' - -``` -=> -```mdtest-output +Absolute value of various numbers: +```mdtest-spq {data-layout="stacked"} +# spq +yield abs(this) +# input +1 +-1 +0 +-1.0 +-1(int8) +1(uint8) +"foo" +# expected output 1 1 0 diff --git a/docs/language/functions/base64.md b/docs/language/functions/base64.md index 01443ee3ac..37ef15d28d 100644 --- a/docs/language/functions/base64.md +++ b/docs/language/functions/base64.md @@ -18,34 +18,41 @@ or decodes a Base64 string `s` into a bytes value. ### Examples Encode byte sequence `0x010203` into its Base64 string: -```mdtest-command -echo '0x010203' | super -z -c 'yield base64(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield base64(this) +# input +0x010203 +# expected output "AQID" ``` + Decode "AQID" into byte sequence `0x010203`: -```mdtest-command -echo '"AQID"' | super -z -c 'yield base64(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield base64(this) +# input +"AQID" +# expected output 0x010203 ``` + Encode ASCII string into Base64-encoded string: -```mdtest-command -echo '"hello, world"' | super -z -c 'yield base64(bytes(this))' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield base64(bytes(this)) +# input +"hello, world" +# expected output "aGVsbG8sIHdvcmxk" ``` + Decode a Base64 string and cast the decoded bytes to a string: -```mdtest-command -echo '"aGVsbG8gd29ybGQ="' | super -z -c 'yield string(base64(this))' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield string(base64(this)) +# input +"aGVsbG8gd29ybGQ=" +# expected output "hello world" ``` diff --git a/docs/language/functions/bucket.md b/docs/language/functions/bucket.md index ad3dd90fa7..c9799c4f99 100644 --- a/docs/language/functions/bucket.md +++ b/docs/language/functions/bucket.md @@ -19,12 +19,13 @@ aligns with 0. ### Examples Bucket a couple times to hour intervals: -```mdtest-command -echo '2020-05-26T15:27:47Z "5/26/2020 3:27pm"' | - super -z -c 'yield bucket(time(this), 1h)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield bucket(time(this), 1h) +# input +2020-05-26T15:27:47Z +"5/26/2020 3:27pm" +# expected output 2020-05-26T15:00:00Z 2020-05-26T15:00:00Z ``` diff --git a/docs/language/functions/cast.md b/docs/language/functions/cast.md index 63819d1b66..191282ff76 100644 --- a/docs/language/functions/cast.md +++ b/docs/language/functions/cast.md @@ -41,57 +41,69 @@ to match the output type's order but rather just modifies the leaf values. If a cast fails, an error is returned when casting to primitive types and the input value is returned when casting to complex types. -:::tip +{{% tip "Note" %}} + Many users seeking to `cast` record values prefer to use the [`shape` function](./shape.md) which applies the `cast`, [`fill`](./fill.md), and [`order`](./order.md) functions simultaneously. -::: + +{{% /tip %}} ### Examples _Cast primitives to type `ip`_ -```mdtest-command -echo '"10.0.0.1" 1 "foo"' | super -z -c 'cast(this, )' - -``` -produces -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +cast(this, ) +# input +"10.0.0.1" +1 +"foo" +# expected output 10.0.0.1 error({message:"cannot cast to ip",on:1}) error({message:"cannot cast to ip",on:"foo"}) ``` _Cast a record to a different record type_ -```mdtest-command -echo '{a:1,b:2}{a:3}{b:4}' | super -z -c 'cast(this, <{b:string}>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +cast(this, <{b:string}>) +# input +{a:1,b:2} +{a:3} +{b:4} +# expected output {a:1,b:"2"} {a:3} {b:"4"} ``` -_Create a name a typed and cast value to the new type_ -```mdtest-command -echo '{a:1,b:2}{a:3,b:4}' | super -z -c 'cast(this, "foo")' - -``` -produces -```mdtest-output +_Create a named type and cast value to the new type_ +```mdtest-spq +# spq +cast(this, "foo") +# input +{a:1,b:2} +{a:3,b:4} +# expected output {a:1,b:2}(=foo) {a:3,b:4}(=foo) ``` _Derive type names from the properties of data_ -```mdtest-command -echo '{x:1,y:2}{r:3}{x:4,y:5}' | - super -z -c 'switch ( - case has(x) => cast(this, "point") - default => cast(this, "radius") - ) - |> sort this' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +switch ( + case has(x) => cast(this, "point") + default => cast(this, "radius") +) +| sort this +# input +{x:1,y:2} +{r:3} +{x:4,y:5} +# expected output {r:3}(=radius) {x:1,y:2}(=point) {x:4,y:5}(=point) diff --git a/docs/language/functions/ceil.md b/docs/language/functions/ceil.md index 5f0425a93a..bdfaea88b3 100644 --- a/docs/language/functions/ceil.md +++ b/docs/language/functions/ceil.md @@ -15,12 +15,16 @@ which must be a numeric type. The return type retains the type of the argument. ### Examples -The ceiling of a various numbers: -```mdtest-command -echo '1.5 -1.5 1(uint8) 1.5(float32)' | super -z -c 'yield ceil(this)' - -``` -=> -```mdtest-output +The ceiling of various numbers: +```mdtest-spq +# spq +yield ceil(this) +# input +1.5 +-1.5 +1(uint8) +1.5(float32) +# expected output 2. -1. 1(uint8) diff --git a/docs/language/functions/cidr_match.md b/docs/language/functions/cidr_match.md index 81c6f9fd4b..7e3ff03874 100644 --- a/docs/language/functions/cidr_match.md +++ b/docs/language/functions/cidr_match.md @@ -18,35 +18,41 @@ If `network` is not type `net`, then an error is returned. ### Examples Test whether values are IP addresses in a network: -```mdtest-command -echo '10.1.2.129 11.1.2.129 10 "foo"' | - super -z -c 'yield cidr_match(10.0.0.0/8, this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield cidr_match(10.0.0.0/8, this) +# input +10.1.2.129 +11.1.2.129 +10 +"foo" +# expected output true false false false ``` -It also works for IPs in complex values: -```mdtest-command -echo '[10.1.2.129,11.1.2.129] {a:10.0.0.1} {a:11.0.0.1}' | - super -z -c 'yield cidr_match(10.0.0.0/8, this)' - -``` -=> -```mdtest-output +It also works for IPs in complex values: +```mdtest-spq +# spq +yield cidr_match(10.0.0.0/8, this) +# input +[10.1.2.129,11.1.2.129] +{a:10.0.0.1} +{a:11.0.0.1} +# expected output true true false ``` The first argument must be a network: -```mdtest-command -echo '10.0.0.1' | super -z -c 'yield cidr_match([1,2,3], this)' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield cidr_match([1,2,3], this) +# input +10.0.0.1 +# expected output error({message:"cidr_match: not a net",on:[1,2,3]}) ``` diff --git a/docs/language/functions/coalesce.md b/docs/language/functions/coalesce.md index 4cf16c8950..cd60c55c18 100644 --- a/docs/language/functions/coalesce.md +++ b/docs/language/functions/coalesce.md @@ -16,18 +16,20 @@ are null, `error("missing")`, or `error("quiet")`. ### Examples -```mdtest-command -super -z -c 'yield coalesce(null, error("missing"), error("quiet"), 1)' -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield coalesce(null, error("missing"), error("quiet"), this) +# input +1 +# expected output 1 ``` -```mdtest-command -super -z -c 'yield coalesce(null, error("missing"), error("quiet"))' -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield coalesce(null, error("missing"), this) +# input +error("quiet") +# expected output null ``` diff --git a/docs/language/functions/compare.md b/docs/language/functions/compare.md index a0f157feb2..d14be0ac2a 100644 --- a/docs/language/functions/compare.md +++ b/docs/language/functions/compare.md @@ -20,10 +20,11 @@ is treated as the minimum or maximum value. ### Examples -```mdtest-command -echo '{a: 2, b: "1"}' | super -z -c 'yield compare(a, b)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield compare(a, b) +# input +{a:2,b:"1"} +# expected output -1 ``` diff --git a/docs/language/functions/crop.md b/docs/language/functions/crop.md index f736012b30..a7144d5372 100644 --- a/docs/language/functions/crop.md +++ b/docs/language/functions/crop.md @@ -25,29 +25,33 @@ If `val` is not a record, it is returned unmodified. ### Examples _Crop a record_ -```mdtest-command -echo '{a:1,b:2}' | super -z -c 'crop(this, <{a:int64}>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +crop(this, <{a:int64}>) +# input +{a:1,b:2} +# expected output {a:1} ``` _Crop an array of records_ -```mdtest-command -echo '[{a:1,b:2},{a:3,b:4}]' | super -z -c 'crop(this, <[{a:int64}]>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +crop(this, <[{a:int64}]>) +# input +[{a:1,b:2},{a:3,b:4}] +# expected output [{a:1},{a:3}] ``` _Cropped primitives are returned unmodified_ -```mdtest-command -echo '10.0.0.1 1 "foo"' | super -z -c 'crop(this, <{a:int64}>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +crop(this, <{a:int64}>) +# input +10.0.0.1 1 +"foo" +# expected output 10.0.0.1 1 "foo" diff --git a/docs/language/functions/error.md b/docs/language/functions/error.md index b67c34ce8c..8ea80b764f 100644 --- a/docs/language/functions/error.md +++ b/docs/language/functions/error.md @@ -17,52 +17,49 @@ a means to create structured and stacked errors. ### Examples Wrap a record as a structured error: -```mdtest-command -echo '{foo:"foo"}' | - super -z -c 'yield error({message:"bad value", value:this})' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield error({message:"bad value", value:this}) +# input +{foo:"foo"} +# expected output error({message:"bad value",value:{foo:"foo"}}) ``` Wrap any value as an error: -```mdtest-command -echo '1 "foo" [1,2,3]' | super -z -c 'yield error(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield error(this) +# input +1 +"foo" +[1,2,3] +# expected output error(1) error("foo") error([1,2,3]) ``` Test if a value is an error and show its type "kind": -```mdtest-command -echo 'error("exception") "exception"' | - super -Z -c 'yield {this,err:is_error(this),kind:kind(this)}' - -``` -=> -```mdtest-output -{ - this: error("exception"), - err: true, - kind: "error" -} -{ - this: "exception", - err: false, - kind: "primitive" -} +```mdtest-spq {data-layout="stacked"} +# spq +yield {this,err:is_error(this),kind:kind(this)} +# input +error("exception") +"exception" +# expected output +{this:error("exception"),err:true,kind:"error"} +{this:"exception",err:false,kind:"primitive"} ``` Comparison of a missing error results in a missing error even if they are the same missing errors so as to not allow field comparisons of two missing fields to succeed: -```mdtest-command -echo '{}' | super -z -c 'badfield:=x |> yield badfield==error("missing")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +badfield:=x | yield badfield==error("missing") +# input +{} +# expected output error("missing") ``` diff --git a/docs/language/functions/every.md b/docs/language/functions/every.md index cd5b0e053e..b1895fe6ee 100644 --- a/docs/language/functions/every.md +++ b/docs/language/functions/every.md @@ -17,27 +17,28 @@ when analyzing time-series data like logs that have a `ts` field. ### Examples Operate on a sequence of times: -```mdtest-command -echo '{ts:2021-02-01T12:00:01Z}' | - super -z -c 'yield {ts,val:0},{ts:ts+1s},{ts:ts+2h2s} - |> yield every(1h) - |> sort' - -``` --> -```mdtest-output +```mdtest-spq +# spq +yield {ts,val:0},{ts:ts+1s},{ts:ts+2h2s} +| yield every(1h) +| sort +# input +{ts:2021-02-01T12:00:01Z} +# expected output 2021-02-01T12:00:00Z 2021-02-01T12:00:00Z 2021-02-01T14:00:00Z ``` + Use as a group-by key: -```mdtest-command -echo '{ts:2021-02-01T12:00:01Z}' | - super -z -c 'yield {ts,val:1},{ts:ts+1s,val:2},{ts:ts+2h2s,val:5} - |> sum(val) by every(1h) - |> sort' - -``` --> -```mdtest-output +```mdtest-spq +# spq +yield {ts,val:1},{ts:ts+1s,val:2},{ts:ts+2h2s,val:5} +| sum(val) by every(1h) +| sort +# input +{ts:2021-02-01T12:00:01Z} +# expected output {ts:2021-02-01T12:00:00Z,sum:3} {ts:2021-02-01T14:00:00Z,sum:5} ``` diff --git a/docs/language/functions/fields.md b/docs/language/functions/fields.md index ebce2916c6..50de843ee1 100644 --- a/docs/language/functions/fields.md +++ b/docs/language/functions/fields.md @@ -20,30 +20,34 @@ can appear in a field name. ### Examples Extract the fields of a nested record: -```mdtest-command -echo '{a:1,b:2,c:{d:3,e:4}}' | super -z -c 'yield fields(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield fields(this) +# input +{a:1,b:2,c:{d:3,e:4}} +# expected output [["a"],["b"],["c","d"],["c","e"]] ``` + Easily convert to dotted names if you prefer: -```mdtest-command -echo '{a:1,b:2,c:{d:3,e:4}}' | - super -z -c 'over fields(this) |> yield join(this,".")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +over fields(this) | yield join(this,".") +# input +{a:1,b:2,c:{d:3,e:4}} +# expected output "a" "b" "c.d" "c.e" ``` + A record is expected: -```mdtest-command -echo 1 | super -z -c 'yield {f:fields(this)}' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield {f:fields(this)} +# input +1 +# expected output {f:error("missing")} ``` diff --git a/docs/language/functions/fill.md b/docs/language/functions/fill.md index a5dd47897e..067a9e9faf 100644 --- a/docs/language/functions/fill.md +++ b/docs/language/functions/fill.md @@ -20,38 +20,45 @@ you want to be sure that all fields in a schema are present in a record. If `val` is not a record, it is returned unmodified. -:::tip +{{% tip "Tip" %}} + Many users seeking the functionality of `fill` prefer to use the [`shape` function](./shape.md) which applies the `fill`, [`cast`](./cast.md), and [`order`](./order.md) functions simultaneously on a record. -::: + +{{% /tip %}} ### Examples _Fill a record_ -```mdtest-command -echo '{a:1}' | super -z -c 'fill(this, <{a:int64,b:string}>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +fill(this, <{a:int64,b:string}>) +# input +{a:1} +# expected output {a:1,b:null(string)} ``` _Fill an array of records_ -```mdtest-command -echo '[{a:1},{a:2}]' | super -z -c 'fill(this, <[{a:int64,b:int64}]>)' - -``` -produces -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +fill(this, <[{a:int64,b:int64}]>) +# input +[{a:1},{a:2}] +# expected output [{a:1,b:null(int64)},{a:2,b:null(int64)}] ``` _Non-records are returned unmodified_ -```mdtest-command -echo '10.0.0.1 1 "foo"' | super -z -c 'fill(this, <{a:int64,b:int64}>)' - -``` -produces -```mdtest-output +```mdtest-spq +# spq +fill(this, <{a:int64,b:int64}>) +# input +10.0.0.1 +1 +"foo" +# expected output 10.0.0.1 1 "foo" diff --git a/docs/language/functions/flatten.md b/docs/language/functions/flatten.md index b5d69da1d4..7ad48a94c2 100644 --- a/docs/language/functions/flatten.md +++ b/docs/language/functions/flatten.md @@ -17,10 +17,11 @@ inner type is a union of the record types present. ### Examples -```mdtest-command -echo '{a:1,b:{c:"foo"}}' | super -z -c 'yield flatten(this)' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield flatten(this) +# input +{a:1,b:{c:"foo"}} +# expected output [{key:["a"],value:1},{key:["b","c"],value:"foo"}] ``` diff --git a/docs/language/functions/floor.md b/docs/language/functions/floor.md index bcaff652a8..0610ddd0ab 100644 --- a/docs/language/functions/floor.md +++ b/docs/language/functions/floor.md @@ -16,11 +16,15 @@ which must be a numeric type. The return type retains the type of the argument. ### Examples The floor of a various numbers: -```mdtest-command -echo '1.5 -1.5 1(uint8) 1.5(float32)' | super -z -c 'yield floor(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield floor(this) +# input +1.5 +-1.5 +1(uint8) +1.5(float32) +# expected output 1. -2. 1(uint8) diff --git a/docs/language/functions/grep.md b/docs/language/functions/grep.md index 904bdde6d8..3076bec1d6 100644 --- a/docs/language/functions/grep.md +++ b/docs/language/functions/grep.md @@ -17,9 +17,13 @@ The _grep_ function searches all of the strings in its input value `e` [glob pattern](../search-expressions.md#globs), or string. If the pattern matches for any string, then the result is `true`. Otherwise, it is `false`. -> Note that string matches are case insensitive while regular expression -> and glob matches are case sensitive. In a forthcoming release, case sensitivity -> will be a expressible for all three pattern types. +{{% tip "Note" %}} + +String matches are case insensitive while regular expression +and glob matches are case sensitive. In a forthcoming release, case sensitivity +will be a expressible for all three pattern types. + +{{% /tip %}} The entire input value is traversed: * for records, each field name is traversed and each field value is traversed or descended @@ -30,44 +34,59 @@ if a complex type, ### Examples _Reach into nested records_ -```mdtest-command -echo '{foo:10}{bar:{s:"baz"}}' | super -z -c 'grep("baz")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +grep("baz") +# input +{foo:10} +{bar:{s:"baz"}} +# expected output {bar:{s:"baz"}} ``` + _It only matches string fields_ -```mdtest-command -echo '{foo:10}{bar:{s:"baz"}}' | super -z -c 'grep("10")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +grep("10") +# input +{foo:10} +{bar:{s:"baz"}} +# expected output ``` + _Match a field name_ -```mdtest-command -echo '{foo:10}{bar:{s:"baz"}}' | super -z -c 'grep("foo")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +grep("foo") +# input +{foo:10} +{bar:{s:"baz"}} +# expected output {foo:10} ``` + _Regular expression_ -```mdtest-command -echo '{foo:10}{bar:{s:"baz"}}' | super -z -c 'grep(/foo|baz/)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +grep(/foo|baz/) +# input +{foo:10} +{bar:{s:"baz"}} +# expected output {foo:10} {bar:{s:"baz"}} ``` -_Glob with a second argument_ -```mdtest-command -echo '{s:"bar"}{s:"foo"}{s:"baz"}{t:"baz"}' | super -z -c 'grep(b*, s)' - -``` -=> -```mdtest-output +_Glob with a second argument_ +```mdtest-spq +# spq +grep(b*, s) +# input +{s:"bar"} +{s:"foo"} +{s:"baz"} +{t:"baz"} +# expected output {s:"bar"} {s:"baz"} ``` diff --git a/docs/language/functions/grok.md b/docs/language/functions/grok.md index ab6ae22341..e13b873b39 100644 --- a/docs/language/functions/grok.md +++ b/docs/language/functions/grok.md @@ -42,12 +42,14 @@ been published by Elastic and others that provide helpful guidance on becoming proficient in Grok. To help you adapt what you learn from these resources to the use of the `grok` function, review the tips below. -:::tip Note +{{% tip "Note" %}} + As these represent areas of possible future SuperPipe enhancement, links to open issues are provided. If you find a functional gap significantly impacts your ability to use the `grok` function, please add a comment to the relevant issue describing your use case. -::: + +{{% /tip %}} 1. Logstash's Grok offers an optional data type conversion syntax, e.g., @@ -79,24 +81,15 @@ issue describing your use case. Instead use the the approach shown later in that section of the Logstash docs by including a custom pattern in the `definitions` argument, e.g., - ```mdtest-command - echo '"Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>"' | - super -Z -c 'yield grok("%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}", - this, - "POSTFIX_QUEUEID [0-9A-F]{10,11}")' - - ``` - - produces - - ```mdtest-output - { - timestamp: "Jan 1 06:25:43", - logsource: "mailserver14", - program: "postfix/cleanup", - pid: "21403", - queue_id: "BEF25A72965", - syslog_message: "message-id=<20130101142543.5828399CCAF@mailserver14.example.com>" - } + ```mdtest-spq {data-layout="stacked"} + # spq + yield grok("%{SYSLOGBASE} %{POSTFIX_QUEUEID:queue_id}: %{GREEDYDATA:syslog_message}", + this, + "POSTFIX_QUEUEID [0-9A-F]{10,11}") + # input + "Jan 1 06:25:43 mailserver14 postfix/cleanup[21403]: BEF25A72965: message-id=<20130101142543.5828399CCAF@mailserver14.example.com>" + # expected output + {timestamp:"Jan 1 06:25:43",logsource:"mailserver14",program:"postfix/cleanup",pid:"21403",queue_id:"BEF25A72965",syslog_message:"message-id=<20130101142543.5828399CCAF@mailserver14.example.com>"} ``` 4. The Grok implementation for Logstash uses the @@ -111,7 +104,8 @@ issue describing your use case. avoid compatibility issues, we recommend building configurations starting from the RE2-based [included patterns](#included-patterns). -:::tip Note +{{% tip "Note" %}} + If you absolutely require features of Logstash's Grok that are not currently present in SuperPipe, you can create a Logstash-based preprocessing pipeline that uses its @@ -121,7 +115,8 @@ and send its output as JSON to SuperPipe. Issue getting started. If you pursue this approach, please add a comment to the issue describing your use case or come talk to us on [community Slack](https://www.brimdata.io/join-slack/). -::: + +{{% /tip %}} ### Debugging @@ -150,18 +145,14 @@ on the [community Slack](https://www.brimdata.io/join-slack/). ### Examples Parsing a simple log line using the built-in named patterns: -```mdtest-command -echo '"2020-09-16T04:20:42.45+01:00 DEBUG This is a sample debug log message"' | - super -Z -c 'yield grok("%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}", - this)' - -``` -=> -```mdtest-output -{ - timestamp: "2020-09-16T04:20:42.45+01:00", - level: "DEBUG", - message: "This is a sample debug log message" -} +```mdtest-spq {data-layout="stacked"} +# spq +yield grok("%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:level} %{GREEDYDATA:message}", + this) +# input +"2020-09-16T04:20:42.45+01:00 DEBUG This is a sample debug log message" +# expected output +{timestamp:"2020-09-16T04:20:42.45+01:00",level:"DEBUG",message:"This is a sample debug log message"} ``` As with any [string literal](../expressions.md#literals), the @@ -169,14 +160,14 @@ leading backslash in escape sequences in string arguments must be doubled, such as changing the `\d` to `\\d` if we repurpose the [included pattern](#included-patterns) for `NUMTZ` as a `definitions` argument: -```mdtest-command -echo '"+7000"' | - super -z -c 'yield grok("%{MY_NUMTZ:tz}", - this, - "MY_NUMTZ [+-]\\d{4}")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield grok("%{MY_NUMTZ:tz}", + this, + "MY_NUMTZ [+-]\\d{4}") +# input +"+7000" +# expected output {tz:"+7000"} ``` @@ -184,14 +175,14 @@ In addition to using `\n` newline escapes to separate multiple named patterns in the `definitions` argument, string concatenation via `+` may further enhance readability. -```mdtest-command -echo '"(555)-1212"' | - super -z -c 'yield grok("\\(%{PH_PREFIX:prefix}\\)-%{PH_LINE_NUM:line_number}", - this, - "PH_PREFIX \\d{3}\n" + - "PH_LINE_NUM \\d{4}")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield grok("\\(%{PH_PREFIX:prefix}\\)-%{PH_LINE_NUM:line_number}", + this, + "PH_PREFIX \\d{3}\n" + + "PH_LINE_NUM \\d{4}") +# input +"(555)-1212" +# expected output {prefix:"555",line_number:"1212"} ``` diff --git a/docs/language/functions/has.md b/docs/language/functions/has.md index 3a7459f660..523e2d8bde 100644 --- a/docs/language/functions/has.md +++ b/docs/language/functions/has.md @@ -30,21 +30,56 @@ switch ( ### Examples -```mdtest-command -echo '{foo:10}' | super -z -c 'yield {yes:has(foo),no:has(bar)}' - -echo '{foo:[1,2,3]}' | super -z -c 'yield {yes: has(foo[0]),no:has(foo[3])}' - -echo '{foo:{bar:"value"}}' | - super -z -c 'yield {yes:has(foo.bar),no:has(foo.baz)}' - -echo '{foo:10}' | super -z -c 'yield {yes:has(foo+1),no:has(bar+1)}' - -echo 1 | super -z -c 'yield has(bar)' - -echo '{x:error("missing")}' | super -z -c 'yield has(x)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield {yes:has(foo),no:has(bar)} +# input +{foo:10} +# expected output {yes:true,no:false} +``` + +```mdtest-spq +# spq +yield {yes: has(foo[0]),no:has(foo[3])} +# input +{foo:[1,2,3]} +# expected output {yes:true,no:false} +``` + +```mdtest-spq +# spq +yield {yes:has(foo.bar),no:has(foo.baz)} +# input +{foo:{bar:"value"}} +# expected output {yes:true,no:false} +``` + +```mdtest-spq +# spq +yield {yes:has(foo+1),no:has(bar+1)} +# input +{foo:10} +# expected output {yes:true,no:false} +``` + +```mdtest-spq +# spq +yield has(bar) +# input +1 +# expected output false +``` + +```mdtest-spq +# spq +yield has(x) +# input +{x:error("missing")} +# expected output false ``` diff --git a/docs/language/functions/has_error.md b/docs/language/functions/has_error.md index d7af8be356..53abea6258 100644 --- a/docs/language/functions/has_error.md +++ b/docs/language/functions/has_error.md @@ -16,12 +16,20 @@ into value's leaves to determine if there is an error in the value. ### Examples -```mdtest-command -echo '{a:{b:"foo"}}' | super -z -c 'yield has_error(this)' - -echo '{a:{b:"foo"}}' | super -z -c 'a.x := a.y + 1 |> yield has_error(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield has_error(this) +# input +{a:{b:"foo"}} +# expected output false +``` + +```mdtest-spq +# spq +a.x := a.y + 1 | yield has_error(this) +# input +{a:{b:"foo"}} +# expected output true ``` diff --git a/docs/language/functions/hex.md b/docs/language/functions/hex.md index a9c5df9ab3..0d99037a00 100644 --- a/docs/language/functions/hex.md +++ b/docs/language/functions/hex.md @@ -17,34 +17,39 @@ a hexadecimal string or decodes a hexadecimal string `s` into a bytes value. ### Examples Encode a simple bytes sequence as a hexadecimal string: -```mdtest-command -echo '0x0102ff' | super -z -c 'yield hex(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield hex(this) +# input +0x0102ff +# expected output "0102ff" ``` Decode a simple hex string: -```mdtest-command -echo '"0102ff"' | super -z -c 'yield hex(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield hex(this) +# input +"0102ff" +# expected output 0x0102ff ``` Encode the bytes of an ASCII string as a hexadecimal string: -```mdtest-command -echo '"hello, world"' | super -z -c 'yield hex(bytes(this))' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield hex(bytes(this)) +# input +"hello, world" +# expected output "68656c6c6f2c20776f726c64" ``` + Decode hex string representing ASCII into its string form: -```mdtest-command -echo '"68656c6c6f20776f726c64"' | super -z -c 'yield string(hex(this))' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield string(hex(this)) +# input +"68656c6c6f20776f726c64" +# expected output "hello world" ``` diff --git a/docs/language/functions/is.md b/docs/language/functions/is.md index 765aa4f13a..84f20b4740 100644 --- a/docs/language/functions/is.md +++ b/docs/language/functions/is.md @@ -16,39 +16,51 @@ is omitted, it defaults to `this`. The _is_ function is shorthand for `typeof(v ### Examples Test simple types: -```mdtest-command -echo '1.' | super -z -c 'yield {yes:is(),no:is()}' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield {yes:is(),no:is()} +# input +1. +# expected output {yes:true,no:false} ``` Test for a given input's record type or "shape": -```mdtest-command -echo '{s:"hello"}' | super -z -c 'yield is(<{s:string}>)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield is(<{s:string}>) +# input +{s:"hello"} +# expected output true ``` -If you test a named type with it's underlying type, the types are different, + +If you test a named type with its underlying type, the types are different, but if you use the type name or typeunder function, there is a match: -```mdtest-command -echo '{s:"hello"}(=foo)' | super -z -c 'yield is(<{s:string}>)' - -echo '{s:"hello"}(=foo)' | super -z -c 'yield is()' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield is(<{s:string}>) +# input +{s:"hello"}(=foo) +# expected output false +``` + +```mdtest-spq +# spq +yield is() +# input +{s:"hello"}(=foo) +# expected output true ``` To test the underlying type, just use `==`: -```mdtest-command -echo '{s:"hello"}(=foo)' | super -z -c 'yield typeunder(this)==<{s:string}>' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield typeunder(this)==<{s:string}> +# input +{s:"hello"}(=foo) +# expected output true ``` diff --git a/docs/language/functions/is_error.md b/docs/language/functions/is_error.md index 6b9552e250..7f242cb06f 100644 --- a/docs/language/functions/is_error.md +++ b/docs/language/functions/is_error.md @@ -11,35 +11,38 @@ is_error(val: any) -> bool ### Description The _is_error_ function returns true if its argument's type is error. -`is_error(v)` is shortcut for `kind(v)=="error"`, +`is_error(v)` is shorthand for `kind(v)=="error"`, ### Examples A simple value is not an error: -```mdtest-command -echo 1 | super -z -c 'yield is_error(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield is_error(this) +# input +1 +# expected output false ``` An error value is an error: -```mdtest-command -echo "error(1)" | super -z -c 'yield is_error(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield is_error(this) +# input +error(1) +# expected output true ``` Convert an error string into a record with an indicator and a message: -```mdtest-command -echo '"not an error" error("an error")' | - super -z -c 'yield {err:is_error(this),message:under(this)}' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield {err:is_error(this),message:under(this)} +# input +"not an error" +error("an error") +# expected output {err:false,message:"not an error"} {err:true,message:"an error"} ``` diff --git a/docs/language/functions/join.md b/docs/language/functions/join.md index 297e04b4e0..83f0a75d74 100644 --- a/docs/language/functions/join.md +++ b/docs/language/functions/join.md @@ -16,21 +16,23 @@ string. The string `sep` is placed between each value in the resulting string. #### Example: Join a symbol array of strings: -```mdtest-command -echo '["a","b","c"]' | super -z -c 'yield join(this, ",")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield join(this, ",") +# input +["a","b","c"] +# expected output "a,b,c" ``` Join non-string arrays by first casting: -```mdtest-command -echo '[1,2,3] [10.0.0.1,10.0.0.2]' | - super -z -c 'yield join(cast(this, <[string]>), "...")' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield join(cast(this, <[string]>), "...") +# input +[1,2,3] +[10.0.0.1,10.0.0.2] +# expected output "1...2...3" "10.0.0.1...10.0.0.2" ``` diff --git a/docs/language/functions/kind.md b/docs/language/functions/kind.md index 3c876a052c..6528194279 100644 --- a/docs/language/functions/kind.md +++ b/docs/language/functions/kind.md @@ -17,23 +17,31 @@ then the type category of the referenced type is returned. #### Example: A primitive value's kind is "primitive": -```mdtest-command -echo '1 "a" 10.0.0.1' | super -z -c 'yield kind(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield kind(this) +# input +1 +"a" +10.0.0.1 +# expected output "primitive" "primitive" "primitive" ``` -A complex value's kind is it's complex type category. Try it on +A complex value's kind is its complex type category. Try it on these empty values of various complex types: -```mdtest-command -echo '{} [] |[]| |{}| 1((int64,string))' | super -z -c 'yield kind(this)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield kind(this) +# input +{} +[] +|[]| +|{}| +1((int64,string)) +# expected output "record" "array" "set" @@ -41,20 +49,22 @@ echo '{} [] |[]| |{}| 1((int64,string))' | super -z -c 'yield kind(this)' - "union" ``` -A Zed error has kind "error": -```mdtest-command -echo null | super -z -c 'yield kind(1/0)' - -``` -=> -```mdtest-output +An error has kind "error": +```mdtest-spq +# spq +yield kind(1/this) +# input +0 +# expected output "error" ``` -A Zed type's kind is the kind of the type: -```mdtest-command -echo '<{s:string}>' | super -z -c 'yield kind(this)' - -``` -=> -```mdtest-output +A type's kind is the kind of the type: +```mdtest-spq +# spq +yield kind(this) +# input +<{s:string}> +# expected output "record" ``` diff --git a/docs/language/functions/ksuid.md b/docs/language/functions/ksuid.md index a6359c77b6..ec1497ef3f 100644 --- a/docs/language/functions/ksuid.md +++ b/docs/language/functions/ksuid.md @@ -21,11 +21,11 @@ returned as a bytes value. #### Example: -```mdtest-command -echo '{id:0x0dfc90519b60f362e84a3fdddd9b9e63e1fb90d1}' | - super -z -c 'id := ksuid(id)' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +id := ksuid(id) +# input +{id:0x0dfc90519b60f362e84a3fdddd9b9e63e1fb90d1} +# expected output {id:"1zjJzTWWCJNVrGwqB8kZwhTM2fR"} ``` diff --git a/docs/language/functions/len.md b/docs/language/functions/len.md index 7c918f1de4..e27be30e34 100644 --- a/docs/language/functions/len.md +++ b/docs/language/functions/len.md @@ -29,12 +29,16 @@ Supported types include: Take the length of various types: -```mdtest-command -echo '[1,2,3] |["hello"]| {a:1,b:2} "hello" 10.0.0.1 1' | - super -z -c 'yield {this,len:len(this)}' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield {this,len:len(this)} +# input +[1,2,3] +|["hello"]| +{a:1,b:2} +"hello" +10.0.0.1 1 +# expected output {this:[1,2,3],len:3} {this:|["hello"]|,len:1} {this:{a:1,b:2},len:2} diff --git a/docs/language/functions/levenshtein.md b/docs/language/functions/levenshtein.md index 3b518159ce..d5be640d2a 100644 --- a/docs/language/functions/levenshtein.md +++ b/docs/language/functions/levenshtein.md @@ -16,10 +16,11 @@ distance](https://en.wikipedia.org/wiki/Levenshtein_distance) between strings ### Examples -```mdtest-command -echo '{a:"kitten",b:"sitting"}' | super -z -c 'yield levenshtein(a, b)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield levenshtein(a, b) +# input +{a:"kitten",b:"sitting"} +# expected output 3 ``` diff --git a/docs/language/functions/log.md b/docs/language/functions/log.md index ea5e4949f8..0ce438f5b8 100644 --- a/docs/language/functions/log.md +++ b/docs/language/functions/log.md @@ -16,11 +16,15 @@ must be numeric. The return value is a float64 or an error. ### Examples The logarithm of various numbers: -```mdtest-command -echo '4 4.0 2.718 -1' | super -z -c 'yield log(this)' - -``` -=> -```mdtest-output +```mdtest-spq {data-layout="stacked"} +# spq +yield log(this) +# input +4 +4.0 +2.718 +-1 +# expected output 1.3862943611198906 1.3862943611198906 0.999896315728952 @@ -28,11 +32,17 @@ error({message:"log: illegal argument",on:-1}) ``` The largest power of 10 smaller than the input: -```mdtest-command -echo '9 10 20 1000 1100 30000' | super -z -c 'yield int64(log(this)/log(10))' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield int64(log(this)/log(10)) +# input +9 +10 +20 +1000 +1100 +30000 +# expected output 0 1 1 diff --git a/docs/language/functions/lower.md b/docs/language/functions/lower.md index a8d0de6b16..31b885678a 100644 --- a/docs/language/functions/lower.md +++ b/docs/language/functions/lower.md @@ -15,10 +15,11 @@ to lower case and returns the result. ### Examples -```mdtest-command -echo '"Zed"' | super -z -c 'yield lower(this)' - -``` -=> -```mdtest-output -"zed" +```mdtest-spq +# spq +yield lower(this) +# input +"SuperDB" +# expected output +"superdb" ``` diff --git a/docs/language/functions/map.md b/docs/language/functions/map.md index c37b7ef261..d14f12f47a 100644 --- a/docs/language/functions/map.md +++ b/docs/language/functions/map.md @@ -17,27 +17,24 @@ only one argument. `f` may be a [user-defined function](../statements.md#func-st ### Examples Upper case each element of an array: - -```mdtest-command -echo '["foo","bar","baz"]' | super -z -c 'yield map(this, upper)' - -``` -=> -```mdtest-output +```mdtest-spq +# spq +yield map(this, upper) +# input +["foo","bar","baz"] +# expected output ["FOO","BAR","BAZ"] ``` -Using a user-defined function to convert an epoch float to a time: - -```mdtest-command -echo '[1697151533.41415,1697151540.716529]' | - super -z -c ' - func floatToTime(x): ( - cast(x*1000000000,