From 3da3ebbc94e35fb9fffaccaa7901f535856022e0 Mon Sep 17 00:00:00 2001
From: Meira Josephy <144697924+mjosephym@users.noreply.github.com>
Date: Wed, 27 Nov 2024 10:29:30 +0200
Subject: [PATCH 01/12] related content
---
.../data-export/continuous-data-export.md | 22 +++++----
...continuous-export-with-managed-identity.md | 16 +++----
.../data-export/create-alter-continuous.md | 24 +++++++---
.../data-export/disable-enable-continuous.md | 14 ++++--
.../data-export/drop-continuous-export.md | 10 +++-
.../export-data-to-an-external-table.md | 48 ++++++++++++-------
.../data-export/export-data-to-sql.md | 47 +++++++++++-------
.../data-export/export-data-to-storage.md | 34 ++++++++-----
.../kusto/management/data-export/index.md | 4 +-
.../data-export/show-continuous-artifacts.md | 37 ++++++++------
.../data-export/show-continuous-export.md | 38 +++++++++------
.../data-export/show-continuous-failures.md | 36 +++++++++-----
12 files changed, 212 insertions(+), 118 deletions(-)
diff --git a/data-explorer/kusto/management/data-export/continuous-data-export.md b/data-explorer/kusto/management/data-export/continuous-data-export.md
index b1b87836c6..59534c052c 100644
--- a/data-explorer/kusto/management/data-export/continuous-data-export.md
+++ b/data-explorer/kusto/management/data-export/continuous-data-export.md
@@ -39,8 +39,8 @@ All continuous export commands require at least [Database Admin](../../access-co
* **External table storage accounts**:
- * For best performance, the database and the storage account(s) should be colocated in the same Azure region.
- * Continuous export works in a distributed manner, such that all nodes are exporting concurrently. On large databases, and if the exported data volume is large, this might lead to storage throttling. It's recommended to configure multiple storage accounts for the external table. See [storage failures during export commands](export-data-to-storage.md#failures-during-export-commands) for more details.
+ * For best performance, the database and the storage accounts should be colocated in the same Azure region.
+ * Continuous export works in a distributed manner, such that all nodes are exporting concurrently. On large databases, and if the exported data volume is large, this might lead to storage throttling. The recommendation is to configure multiple storage accounts for the external table. For more information, see [storage failures during export commands](export-data-to-storage.md#failures-during-export-commands).
## Exactly once export
@@ -48,13 +48,13 @@ To guarantee "exactly once" export, continuous export uses [database cursors](..
[IngestionTime policy](../show-table-ingestion-time-policy-command.md) must be enabled on all tables referenced in the query that should be processed "exactly once" in the export. The policy is enabled by default on all newly created tables.
-The guarantee for "exactly once" export is only for files reported in the [show exported artifacts command](show-continuous-artifacts.md). Continuous export doesn't guarantee that each record will be written only once to the external table. If a failure occurs after export has begun and some of the artifacts were already written to the external table, the external table might contain duplicates. If a write operation was aborted before completion, the external table might contain corrupted files. In such cases, artifacts aren't deleted from the external table, but they won't be reported in the [show exported artifacts command](show-continuous-artifacts.md). Consuming the exported files using the `show exported artifacts command` guarantees no duplications and no corruptions.
+The guarantee for "exactly once" export is only for files reported in the [show exported artifacts command](show-continuous-artifacts.md). Continuous export doesn't guarantee that each record is written only once to the external table. If a failure occurs after export begins and some of the artifacts were already written to the external table, the external table might contain duplicates. If a write operation was aborted before completion, the external table might contain corrupted files. In such cases, artifacts aren't deleted from the external table, but they aren't reported in the [show exported artifacts command](show-continuous-artifacts.md). Consuming the exported files using the `show exported artifacts command` guarantees no duplications and no corruptions.
## Export from fact and dimension tables
By default, all tables referenced in the export query are assumed to be [fact tables](../../concepts/fact-and-dimension-tables.md). As such, they're scoped to the database cursor. The syntax explicitly declares which tables are scoped (fact) and which aren't scoped (dimension). See the `over` parameter in the [create command](create-alter-continuous.md) for details.
-The export query includes only the records that joined since the previous export execution. The export query might contain [dimension tables](../../concepts/fact-and-dimension-tables.md) in which all records of the dimension table are included in all export queries. When using joins between fact and dimension tables in continuous-export, keep in mind that records in the fact table are only processed once. If the export runs while records in the dimension tables are missing for some keys, records for the respective keys will either be missed or include null values for the dimension columns in the exported files. Returning missed or null records depends on whether the query uses inner or outer join. The `forcedLatency` property in the continuous-export definition can be useful in such cases, where the fact and dimensions tables are ingested during the same time for matching records.
+The export query includes only the records that joined since the previous export execution. The export query might contain [dimension tables](../../concepts/fact-and-dimension-tables.md) in which all records of the dimension table are included in all export queries. When using joins between fact and dimension tables in continuous-export, keep in mind that records in the fact table are only processed once. If the export runs while records in the dimension tables are missing for some keys, records for the respective keys are either missed or include null values for the dimension columns in the exported files. Returning missed or null records depends on whether the query uses inner or outer join. The `forcedLatency` property in the continuous-export definition can be useful in such cases, where the fact and dimensions tables are ingested during the same time for matching records.
> [!NOTE]
> Continuous export of only dimension tables isn't supported. The export query must include at least a single fact table.
@@ -63,7 +63,7 @@ The export query includes only the records that joined since the previous export
Monitor the health of your continuous export jobs using the following [export metrics](/azure/data-explorer/using-metrics#export-metrics):
-* `Continuous export max lateness` - Max lateness (in minutes) of continuous exports in the databsae. This is the time between now and the min `ExportedTo` time of all continuous export jobs in database. For more information, see [`.show continuous export`](show-continuous-export.md) command.
+* `Continuous export max lateness` - Max lateness (in minutes) of continuous exports in the database. This is the time between now and the min `ExportedTo` time of all continuous export jobs in database. For more information, see [`.show continuous export`](show-continuous-export.md) command.
* `Continuous export result` - Success/failure result of each continuous export execution. This metric can be split by the continuous export name.
Use the [`.show continuous export failures`](show-continuous-failures.md) command to see the specific failures of a continuous export job.
@@ -76,7 +76,7 @@ Use the [`.show continuous export failures`](show-continuous-failures.md) comman
### Resource consumption
* The impact of the continuous export on the database depends on the query the continuous export is running. Most resources, such as CPU and memory, are consumed by the query execution.
-* The number of export operations that can run concurrently is limited by the database's data export capacity. For more information, see [Management commands throttling](../../management/capacity-policy.md#management-commands-throttling). If the database doesn't have sufficient capacity to handle all continuous exports, some will start lagging behind.
+* The number of export operations that can run concurrently is limited by the database's data export capacity. For more information, see [Management commands throttling](../../management/capacity-policy.md#management-commands-throttling). If the database doesn't have sufficient capacity to handle all continuous exports, some start lagging behind.
* The [show commands-and-queries command](../commands-and-queries.md) can be used to estimate the resources consumption.
* Filter on `| where ClientActivityId startswith "RunContinuousExports"` to view the commands and queries associated with continuous export.
@@ -135,8 +135,8 @@ To define continuous export to a delta table, do the following steps:
**General**:
* The following formats are allowed on target tables: `CSV`, `TSV`, `JSON`, and `Parquet`.
-* Continuous export isn't designed to work over [materialized views](../materialized-views/materialized-view-overview.md), since a materialized view might be updated, while data exported to storage is always append only and never updated.
-* Continuous export cannot be created on [follower databases](/azure/data-explorer/follower) since follower databases are read-only and continuous export requires write operations.
+* Continuous export isn't designed to work over [materialized views](../materialized-views/materialized-view-overview.md), since a materialized view might be updated, while data exported to storage is always appended and never updated.
+* Continuous export can't be created on [follower databases](/azure/data-explorer/follower) since follower databases are read-only and continuous export requires write operations.
* Records in source table must be ingested to the table directly, using an [update policy](../update-policy.md), or [ingest from query commands](../data-ingestion/ingest-from-query.md). If records are moved into the table using [.move extents](../move-extents.md) or using [.rename table](../rename-table-command.md), continuous export might not process these records. See the limitations described in the [Database Cursors](../database-cursor.md#restrictions) page.
* If the artifacts used by continuous export are intended to trigger Event Grid notifications, see the [known issues section in the Event Grid documentation](/azure/data-explorer/ingest-data-event-grid-overview#known-event-grid-issues).
@@ -162,7 +162,13 @@ To define continuous export to a delta table, do the following steps:
## Related content
+:::moniker range="microsoft-fabric"
* [Create or alter continuous export](create-alter-continuous.md)
+* [External tables](../../query/schema-entities/external-tables.md)
+:::monker-end
+
:::moniker-range="azure-data-explorer"
+* [Create or alter continuous export](create-alter-continuous.md)
+* [External tables](../../query/schema-entities/external-tables.md)
* [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md)
:::monker-end
diff --git a/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md b/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md
index aabf413159..be84af65c9 100644
--- a/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md
+++ b/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md
@@ -8,15 +8,15 @@ monikerRange: "azure-data-explorer"
---
# Use a managed identity to run a continuous export job
-> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
+> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
A [continuous export job](continuous-data-export.md) exports data to an [external table](../../query/schema-entities/external-tables.md) with a periodically run query.
The continuous export job should be configured with a [managed identity](/azure/data-explorer/managed-identities-overview) in the following scenarios:
-* When the external table uses impersonation authentication.
-* When the query references tables in other databases.
-* When the query references tables with an enabled [row level security policy](../row-level-security-policy.md).
+* When the external table uses impersonation authentication
+* When the query references tables in other databases
+* When the query references tables with an enabled [row level security policy](../row-level-security-policy.md)
A continuous export job configured with a managed identity is performed on behalf of the managed identity.
@@ -99,7 +99,7 @@ Select one of the following tabs to set up your preferred managed identity type.
## Set up an external table
-External tables refer to data located in Azure Storage, such as Azure Blob Storage, Azure Data Lake Gen1, and Azure Data Lake Gen2, or SQL Server.
+External tables refer to data located in Azure Storage, such as Azure Blob Storage, Azure Data Lake Storage Gen1, Azure Data Lake Storage Gen2, or SQL Server.
Select one of the following tabs to set up an Azure Storage or SQL Server external table.
@@ -107,7 +107,7 @@ Select one of the following tabs to set up an Azure Storage or SQL Server extern
1. Create a connection string based on the [storage connection string templates](../../api/connection-strings/storage-connection-strings.md#storage-connection-string-templates). This string indicates the resource to access and its authentication information. For continuous export flows, we recommend [impersonation authentication](../../api/connection-strings/storage-connection-strings.md#impersonation).
-1. Run the [.create or .alter external table](../external-sql-tables.md) to create the table. Use the connection string from the previous step as the *storageConnectionString* argument.
+1. Run the [.create or .alter external table](../external-sql-tables.md) command to create the table. Use the connection string from the previous step as the *storageConnectionString* argument.
For example, the following command creates `MyExternalTable` that refers to CSV-formatted data in `mycontainer` of `mystorageaccount` in Azure Blob Storage. The table has two columns, one for an integer `x` and one for a string `s`. The connection string ends with `;impersonate`, which indicates to use impersonation authentication to access the data store.
@@ -130,7 +130,7 @@ Select one of the following tabs to set up an Azure Storage or SQL Server extern
1. Create a SQL Server connection string. This string indicates the resource to access and its authentication information. For continuous export flows, we recommend [Microsoft Entra integrated authentication](../../api/connection-strings/storage-connection-strings.md#impersonation), which is impersonation authentication.
-1. Run the [.create or .alter external table](../external-sql-tables.md) to create the table. Use the connection string from the previous step as the *sqlServerConnectionString* argument.
+1. Run the [.create or .alter external table](../external-sql-tables.md) command to create the table. Use the connection string from the previous step as the *sqlServerConnectionString* argument.
For example, the following command creates `MySqlExternalTable` that refers to `MySqlTable` table in `MyDatabase` of SQL Server. The table has two columns, one for an integer `x` and one for a string `s`. The connection string contains `;Authentication=Active Directory Integrated`, which indicates to use impersonation authentication to access the table.
@@ -147,7 +147,7 @@ Select one of the following tabs to set up an Azure Storage or SQL Server extern
## Create a continuous export job
-Select one of the following tabs to create a continuous export job that will run on behalf of a user-assigned or system-assigned managed identity.
+Select one of the following tabs to create a continuous export job that runs on behalf of a user-assigned or system-assigned managed identity.
### [User-assigned](#tab/user-assigned)
diff --git a/data-explorer/kusto/management/data-export/create-alter-continuous.md b/data-explorer/kusto/management/data-export/create-alter-continuous.md
index 064bff8760..34e7786ff2 100644
--- a/data-explorer/kusto/management/data-export/create-alter-continuous.md
+++ b/data-explorer/kusto/management/data-export/create-alter-continuous.md
@@ -3,7 +3,7 @@ title: Create or alter continuous data export
description: This article describes how to create or alter continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/26/2024
---
# Create or alter continuous export
@@ -28,7 +28,7 @@ You must have at least [Database Admin](../../access-control/role-based-access-c
| *continuousExportName* | `string` | :heavy_check_mark: | The name of the continuous export. Must be unique within the database. |
| *externalTableName* | `string` | :heavy_check_mark: | The name of the [external table](../../query/schema-entities/external-tables.md) export target. |
| *query* | `string` | :heavy_check_mark: | The query to export. |
-| *T1*, *T2* | `string` | | A comma-separated list of fact tables in the query. If not specified, all tables referenced in the query are assumed to be fact tables. If specified, tables *not* in this list are treated as dimension tables and won't be scoped, so all records will participate in all exports. See [continuous data export overview](continuous-data-export.md) for details. |
+| *T1*, *T2* | `string` | | A comma-separated list of fact tables in the query. If not specified, all tables referenced in the query are assumed to be fact tables. If specified, tables *not* in this list are treated as dimension tables and aren't scoped, so all records participate in all exports. See [continuous data export overview](continuous-data-export.md) for details. |
| *propertyName*, *propertyValue* | `string` | | A comma-separated list of optional [properties](#supported-properties).|
> [!NOTE]
@@ -39,15 +39,17 @@ You must have at least [Database Admin](../../access-control/role-based-access-c
| Property | Type | Description |
|--|--|--|
| `intervalBetweenRuns` | `Timespan` | The time span between continuous export executions. Must be greater than 1 minute. |
-| `forcedLatency` | `Timespan` | An optional period of time to limit the query to records that were ingested only prior to this period (relative to current time). This property is useful if, for example, the query performs some aggregations/joins and you would like to make sure all relevant records have already been ingested before running the export. |
-| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact being written (prior to compression). Valid range: 100 MB (default) to 1 GB. |
-| `distributed` | `bool` | Disable/enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. |
+| `forcedLatency` | `Timespan` | An optional period of time to limit the query to records ingested before a specified period relative to the current time. This property is useful if, for example, the query performs some aggregations or joins, and you want to make sure all relevant records have been ingested before running the export. |
+| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 1 GB. |
+| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. |
| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. |
-| `managedIdentity` | `string` | The managed identity on behalf of which the continuous export job will run. The managed identity can be an object ID, or the `system` reserved word. For more information, see [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md#use-a-managed-identity-to-run-a-continuous-export-job).|
-| `isDisabled` | `bool` | Disable/enable the continuous export. Default is false. |
+| `managedIdentity` | `string` | The managed identity for which the continuous export job runs. The managed identity can be an object ID, or the `system` reserved word. For more information, see [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md#use-a-managed-identity-to-run-a-continuous-export-job). |
+| `isDisabled` | `bool` | Disable or enable the continuous export. Default is false. |
## Example
+The following example creates or alters a continuous export `MyExport` that exports data from the `T` table to `ExternalBlob`. The data exports occur every hour, and have a defined forced latency and size limit per storage artifact.
+
```kusto
.create-or-alter continuous-export MyExport
over (T)
@@ -62,3 +64,11 @@ with
| Name | ExternalTableName | Query | ForcedLatency | IntervalBetweenRuns | CursorScopedTables | ExportProperties |
|--|--|--|--|--|--|--|
| MyExport | ExternalBlob | S | 00:10:00 | 01:00:00 | [
"['DB'].['S']"
] | {
"SizeLimit": 104857600
} |
+
+## Related content
+
+* [External tables](../../query/schema-entities/external-tables.md)
+* [Disable or enable continuous export](disable-enable-continuous.md)
+* [Show continuous export](show-continuous-export.md)
+* [Drop continuous export](drop-continuous-export.md)
+* [Show continuous export failures](show-continuous-failures.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/disable-enable-continuous.md b/data-explorer/kusto/management/data-export/disable-enable-continuous.md
index b2475784db..661eb75812 100644
--- a/data-explorer/kusto/management/data-export/disable-enable-continuous.md
+++ b/data-explorer/kusto/management/data-export/disable-enable-continuous.md
@@ -9,9 +9,9 @@ ms.date: 08/11/2024
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
-Disables or enables the continuous-export job. A disabled continuous export won't be executed, but its current state is persisted and can be resumed when the continuous export is enabled.
+Disables or enables the continuous-export job. A disabled continuous export isn't executed, but its current state is persisted and can be resumed when the continuous export is enabled.
-When enabling a continuous export that has been disabled for a long time, exporting will continue from where it last stopped when the exporting was disabled. This continuation may result in a long running export, blocking other exports from running, if there isn't sufficient database capacity to serve all processes. Continuous exports are executed by last run time in ascending order (oldest export will run first, until catch up is complete).
+When enabling a continuous export that was disabled for a long time, exporting continues from where it last stopped when the exporting was disabled. This continuation might result in a long running export, blocking other exports from running, if there isn't sufficient database capacity to serve all processes. Continuous exports are executed by last run time in ascending order so the oldest export runs first, until catch up is complete.
## Permissions
@@ -33,4 +33,12 @@ You must have at least [Database Admin](../../access-control/role-based-access-c
## Returns
-The result of the [show continuous export command](show-continuous-export.md) of the altered continuous export.
+The result of the [show continuous export command](show-continuous-export.md) of the altered continuous export.
+
+## Related content
+
+* [External tables](../../query/schema-entities/external-tables.md)
+* [Create or alter continuous export](create-alter-continuous.md)
+* [Drop continuous export](drop-continuous-export.md)
+* [Show continuous export](show-continuous-export.md)
+* [Show continuous export failures](show-continuous-failures.md)
diff --git a/data-explorer/kusto/management/data-export/drop-continuous-export.md b/data-explorer/kusto/management/data-export/drop-continuous-export.md
index b0a2342b70..62160b8fb4 100644
--- a/data-explorer/kusto/management/data-export/drop-continuous-export.md
+++ b/data-explorer/kusto/management/data-export/drop-continuous-export.md
@@ -3,7 +3,7 @@ title: Drop continuous data export
description: This article describes how to drop continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/27/2024
---
# Drop continuous export
@@ -30,3 +30,11 @@ You must have at least [Database Admin](../../access-control/role-based-access-c
## Returns
The remaining continuous exports in the database (post deletion). Output schema as in the [show continuous export command](show-continuous-export.md).
+
+## Related content
+
+* [External tables](../../query/schema-entities/external-tables.md)
+* [Create or alter continuous export](create-alter-continuous.md)
+* [Disable or enable continuous export](disable-enable-continuous.md)
+* [Show continuous export](show-continuous-export.md)
+* [Show continuous export failures](show-continuous-failures.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md b/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md
index 7989cc841a..2581f5251d 100644
--- a/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md
+++ b/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md
@@ -3,7 +3,7 @@ title: Export data to an external table
description: This article describes Export data to an external table.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/26/2024
---
# Export data to an external table
@@ -36,22 +36,22 @@ The following properties are supported as part of the export to external table c
| Property | Type | Description | Default |
|--|--|--|--|
-| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact being written (prior to compression). A full row group of size `parquetRowGroupSize` will be written before checking whether this row group has reached the size limit and should start a new artifact. Valid range: 100 MB (default) to 1 GB. | |
-| `distributed` | `bool` | Disable/enable distributed export. Setting to false is equivalent to `single` distribution hint. | Default is `true`. |
-| `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). See more details in [Distribution settings](#distribution-settings) | Default is `per_node`. |
+| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. A full row group of size `parquetRowGroupSize` is written before checking whether this row group reaches the size limit and should start a new artifact. Valid range: 100 MB (default) to 1 GB. | |
+| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. | Default is `true`. |
+| `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). See more details in [Distribution settings](#distribution-settings) | Default is `per_node`. |
| `distributionKind` | `string` | Optionally switches to uniform distribution when the external table is partitioned by string partition. Valid values are `uniform` or `default`. See more details in [Distribution settings](#distribution-settings) | |
| `concurrency` | *Number* | Hints the system how many partitions to run in parallel. See more details in [Distribution settings](#distribution-settings) | The default value is 16. |
| `spread` | *Number* | Hints the system how to distribute the partitions among nodes. See more details in [Distribution settings](#distribution-settings) | The default value is `Min(64, number-of-nodes)`. |
-| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. This value takes precedence over `sizeLimit`, meaning a full row group will be exported before checking whether this row group has reached the size limit and should start a new artifact. | Default row group size is 100,000 records. |
+| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. This value takes precedence over `sizeLimit`, meaning a full row group will be exported before checking whether this row group reaches the size limit and should start a new artifact. | Default row group size is 100,000 records. |
### Distribution settings
The distribution of an export to external table operation indicates the number of nodes and threads that are writing to storage concurrently. The default distribution depends on the external table partitioning:
-| External table partitioning | Default distribution
+| External table partitioning | Default distribution |
|---|---|
-|External table isn't partitioned, or partitioned by `datetime` column only|Export is distributed `per_node` - all nodes are exporting concurrently. Each node writes the data assigned to that node. The number of files exported by a node will be greater than one, only if the size of the data from that node exceeds `sizeLimit`.|
-|External table is partitioned by a string column|The data to export is moved between the nodes, such that each node writes a subset of the partition values. A single partition is always written by a single node. The number of files written per partition should be greater than one only if the data exceeds `sizeLimit`. If the external table includes several string partitions, then data is partitioned between the node based on the first partition. Therefore, it is recommended to define the partition with most uniform distribution as the first one.|
+|External table isn't partitioned, or partitioned by `datetime` column only|Export is distributed `per_node` - all nodes are exporting concurrently. Each node writes the data assigned to that node. The number of files exported by a node is greater than one, only if the size of the data from that node exceeds `sizeLimit`.|
+|External table is partitioned by a string column|The data to export is moved between the nodes, such that each node writes a subset of the partition values. A single partition is always written by a single node. The number of files written per partition should be greater than one only if the data exceeds `sizeLimit`. If the external table includes several string partitions, then data is partitioned between the node based on the first partition. Therefore, the recommendation is to define the partition with most uniform distribution as the first one.|
#### Change the default distribution settings
@@ -60,8 +60,8 @@ Changing the default distribution settings can be useful in the following cases:
| Use case | Description | Recommendation |
|--|--|--|
| **Reduce the number of exported files** | Export is creating too many small files, and you would like it to create a smaller number of larger files. | Set `distribution`=`single` or `distributed`=`false` (both are equivalent) in the command properties. Only a single thread performs the export. The downside of this is that the export operation can be slower, as concurrency is much reduced. |
-| **Reduce the export duration** | Increasing the concurrency of the export operation, to reduce its duration. | Set `distribution`=`per_shard` in the command properties. This means concurrency of the write operations is per data shard, instead of per node. This is only relevant when exporting to an external table that isn't partitioned by string partition. This might create too much load on storage, potentially resulting in throttling. See [Storage failures](export-data-to-storage.md#storage-failures). |
-| **Reduce the export duration for external tables that are partitioned by a string partition** | If the partitions are not uniformly distributed between the nodes, export might take a longer time to run. For example, if there is a single partition that's much larger than others, most of the export work is done by the single node assigned to that partition, while other nodes will be mostly idle. See [Distribution settings](#distribution-settings). | There are several settings you can change:
* If there is more than one string partition, define the one with best distribution first.
* Set `distributionKind`=`uniform` in the command properties. This setting disables the default distribution settings for string-partitioned external tables. Export will run with `per-node` distribution and each node will export the data assigned to the node. A single partition might be written by several nodes, and the number of files increases accordingly. To increase concurrency even further, set `distributionKind`=`uniform` along with `distribution`=`per_shard` for highest concurrency (at the cost of potentially many more files written)
* If the cause for slow export isn't outliers in the data, reduce duration by increasing concurrency, without changing partitioning settings. Use the `hint.spread` and `hint.concurrency` properties, which determine the concurrency of the partitioning. See [partition operator](../../query/partition-operator.md). By default, the number of nodes exporting concurrently (the `spread`) will be the minimum value between 64 and the number of nodes. Setting `spread` to a higher number than number of nodes increases the concurrency on each node (max value for `spread` is 64). |
+| **Reduce the export duration** | Increasing the concurrency of the export operation, to reduce its duration. | Set `distribution`=`per_shard` in the command properties. Doing so means concurrency of the write operations is per data shard, instead of per node. This is only relevant when exporting to an external table that isn't partitioned by string partition. This might create too much load on storage, potentially resulting in throttling. See [Storage failures](export-data-to-storage.md#storage-failures). |
+| **Reduce the export duration for external tables that are partitioned by a string partition** | If the partitions aren't uniformly distributed between the nodes, export might take a longer time to run. If one partition is much larger than the others, the node assigned to that partition does most of the export work, while the other nodes remain mostly idle. For more information, see [Distribution settings](#distribution-settings). | There are several settings you can change:
* If there's more than one string partition, define the one with best distribution first.
* Set `distributionKind`=`uniform` in the command properties. This setting disables the default distribution settings for string-partitioned external tables. Export runs with `per-node` distribution and each node exports the data assigned to the node. A single partition might be written by several nodes, and the number of files increases accordingly. To increase concurrency even further, set `distributionKind`=`uniform` along with `distribution`=`per_shard` for highest concurrency (at the cost of potentially many more files written)
* If the cause for slow export isn't outliers in the data, reduce duration by increasing concurrency, without changing partitioning settings. Use the `hint.spread` and `hint.concurrency` properties, which determine the concurrency of the partitioning. See [partition operator](../../query/partition-operator.md). By default, the number of nodes exporting concurrently (the `spread`) is the minimum value between 64 and the number of nodes. Setting `spread` to a higher number than number of nodes increases the concurrency on each node (max value for `spread` is 64). |
## Authentication and authorization
@@ -81,40 +81,42 @@ In order to export to an external table, you must set up write permissions. For
* It isn't possible to override the external table properties using the export command. For example, you can't export data in Parquet format to an external table whose data format is CSV.
-* If the external table is partitioned, exported artifacts are written to their respective directories according to the partition definitions as seen in the [partitioned external table example](#partitioned-external-table-example).
+* If the external table is partitioned, exported artifacts are written to their respective directories according to the partition definitions. For an example, see [partitioned external table example](#partitioned-external-table-example).
* If a partition value is null/empty or is an invalid directory value, per the definitions of the target storage, the partition value is replaced with a default value of `__DEFAULT_PARTITION__`.
* For suggestions to overcome storage errors during export commands, see [failures during export commands](export-data-to-storage.md#failures-during-export-commands).
* External table columns are mapped to suitable target format data types, according to [data types mapping](export-data-to-storage.md#data-types-mapping) rules.
-* Parquet native export is a more performant, resource light export mechanism. An exported 'datetime' column is currently unsupported by Synapse SQL 'COPY'.
+* Parquet native export is a more performant, resource light export mechanism. An exported `datetime` column is currently unsupported by Synapse SQL `COPY`.
### Number of files
The number of files written per partition depends on the [distribution settings](#distribution-settings) of the export operation:
- * If the external table includes datetime partitions only, or no partitions at all, the number of files written (for each partition, if exists) should be similar to the number of nodes (or more, if `sizeLimit` is reached). When the export operation is distributed, all nodes export concurrently. To disable distribution, so that only a single node does the writes, set `distributed` to false. This process creates fewer files, but will reduce the export performance.
+* If the external table includes `datetime` partitions only, or no partitions at all, the number of files written for each partition that exists, should be similar to the number of nodes (or more, if `sizeLimit` is reached). When the export operation is distributed, all nodes export concurrently. To disable distribution, so that only a single node does the writes, set `distributed` to false. This process creates fewer files, but reduces the export performance.
-* If the external table includes a partition by a string column, the number of exported files should be a single file per partition (or more, if `sizeLimit` is reached). All nodes still participate in the export (operation is distributed), but each partition is assigned to a specific node. Setting `distributed` to false, will cause only a single node to do the export, but behavior will remain the same (a single file written per partition).
+* If the external table includes a partition by a string column, the number of exported files should be a single file per partition (or more, if `sizeLimit` is reached). All nodes still participate in the export (operation is distributed), but each partition is assigned to a specific node. Setting `distributed` to false, causes only a single node to do the export, but behavior remains the same (a single file written per partition).
## Examples
### Non-partitioned external table example
-ExternalBlob is a non-partitioned external table.
+The following example exports data from table `T` to the `ExternalBlob` table. `ExternalBlob` is a non-partitioned external table.
```kusto
.export to table ExternalBlob <| T
```
+**Output**
+
|ExternalTableName|Path|NumRecords|
|---|---|---|
|ExternalBlob|http://storage1.blob.core.windows.net/externaltable1cont1/1_58017c550b384c0db0fea61a8661333e.csv|10|
### Partitioned external table example
-PartitionedExternalBlob is an external table, defined as follows:
+The following example first creates a partitioned external table, `PartitionedExternalBlob` with a specified blob storage location. The data is stored in CSV format with a path format which organizes the data by customer name and date.
```kusto
.create external table PartitionedExternalBlob (Timestamp:datetime, CustomerName:string)
@@ -127,13 +129,25 @@ dataformat=csv
)
```
+It then exports data from table `T` to the `PartitionedExternalBlob` external table.
+
```kusto
.export to table PartitionedExternalBlob <| T
```
+**Output**
+
|ExternalTableName|Path|NumRecords|
|---|---|---|
|ExternalBlob|http://storageaccount.blob.core.windows.net/container1/CustomerName=customer1/2019/01/01/fa36f35c-c064-414d-b8e2-e75cf157ec35_1_58017c550b384c0db0fea61a8661333e.csv|10|
|ExternalBlob|http://storageaccount.blob.core.windows.net/container1/CustomerName=customer2/2019/01/01/fa36f35c-c064-414d-b8e2-e75cf157ec35_2_b785beec2c004d93b7cd531208424dc9.csv|10|
-If the command is executed asynchronously (by using the `async` keyword), the output is available using the [show operation details](../show-operation-details.md) command.
+If the command is executed asynchronously by using the `async` keyword, the output is available using the [show operation details](../show-operation-details.md) command.
+
+## Related content
+
+* [Management commands overview](../index.md)
+* [External tables](../../query/schema-entities/external-tables.md)
+* [Export data to SQL](export-data-to-sql.md)
+* [Export data to cloud storage](export-data-to-storage.md)
+* [Continuous data export](continuous-data-export.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/export-data-to-sql.md b/data-explorer/kusto/management/data-export/export-data-to-sql.md
index 56b12fe4ea..74c52e2fce 100644
--- a/data-explorer/kusto/management/data-export/export-data-to-sql.md
+++ b/data-explorer/kusto/management/data-export/export-data-to-sql.md
@@ -3,7 +3,7 @@ title: Export data to SQL
description: This article describes Export data to SQL.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/26/2024
---
# Export data to SQL
@@ -32,13 +32,13 @@ You must have at least [Table Admin](../../access-control/role-based-access-cont
## Supported properties
-|Name |Values |Description|
-|-------------------|-----------------|-----------|
-|`firetriggers` |`true` or `false`|If `true`, instructs the target system to fire INSERT triggers defined on the SQL table. The default is `false`. For more information, see [BULK INSERT](/sql/t-sql/statements/bulk-insert-transact-sql) and [System.Data.SqlClient.SqlBulkCopy](/dotnet/api/system.data.sqlclient.sqlbulkcopy).|
-|`createifnotexists`|`true` or `false`|If `true`, the target SQL table is created if it doesn't already exist; the `primarykey` property must be provided in this case to indicate the result column that is the primary key. The default is `false`.|
-|`primarykey` | |If `createifnotexists` is `true`, this property indicates the name of the column in the result that will be used as the SQL table's primary key if it's created by this command.|
-|`persistDetails` |`bool` |Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results. Defaults to `false` in synchronous executions, but can be turned on. |
-|`token` |`string` |The Microsoft Entra access token that Kusto will forward to the SQL endpoint for authentication. When set, the SQL connection string shouldn't include authentication information like `Authentication`, `User ID`, or `Password`.|
+| Name | Values | Description |
+|--|--|--|
+| `firetriggers` | `true` or `false` | If `true`, instructs the target system to fire INSERT triggers defined on the SQL table. The default is `false`. For more information, see [BULK INSERT](/sql/t-sql/statements/bulk-insert-transact-sql) and [System.Data.SqlClient.SqlBulkCopy](/dotnet/api/system.data.sqlclient.sqlbulkcopy). |
+| `createifnotexists` | `true` or `false` | If `true`, the target SQL table is created if it doesn't already exist; the `primarykey` property must be provided in this case to indicate the result column that is the primary key. The default is `false`. |
+| `primarykey` | | If `createifnotexists` is `true`, this property indicates the name of the column in the result that is used as the SQL table's primary key if it's created by this command. |
+| `persistDetails` | `bool` | Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results. Defaults to `false` in synchronous executions, but can be turned on. |
+| `token` | `string` | The Microsoft Entra access token that Kusto forwards to the SQL endpoint for authentication. When set, the SQL connection string shouldn't include authentication information like `Authentication`, `User ID`, or `Password`. |
## Authentication and authorization
@@ -70,10 +70,10 @@ There are some limitations and restrictions when exporting data to an SQL databa
identity itself.
3. If the target table in the SQL database exists, it must match the query result
- schema. Note that in some cases (such as Azure SQL Database) this means
+ schema. In some cases, such as Azure SQL Database, this means
that the table has one column marked as an identity column.
-4. Exporting large volumes of data may take a long time. It's recommended that
+4. Exporting large volumes of data might take a long time. It's recommended that
the target SQL table is set for minimal logging during bulk import.
See [SQL Server Database Engine > ... > Database Features > Bulk Import and Export of Data](/sql/relational-databases/import-export/prerequisites-for-minimal-logging-in-bulk-import).
@@ -87,32 +87,43 @@ There are some limitations and restrictions when exporting data to an SQL databa
8. The primary key property on the column can be specified when creating
a new SQL table. If the column is of type `string`, then SQL might refuse to create the
- table due to other limitations on the primary key column. The workaround is to manually create the table in SQL before exporting the data. The reason for this limitation is that primary key columns in SQL can't be of unlimited size, but Kusto table columns
- have no declared size limitations.
+ table due to other limitations on the primary key column. The workaround is to manually create the table in SQL before exporting the data. This limitation exists because primary key columns in SQL can’t be of unlimited size, but Kusto table columns don't have declared size limitations.
-## Azure DB Microsoft Entra integrated authentication Documentation
+## Azure database Microsoft Entra integrated authentication Documentation
* [Use Microsoft Entra authentication for authentication with SQL Database](/azure/sql-database/sql-database-aad-authentication)
## Examples
-In this example, Kusto runs the query and then exports the first record set produced by the query to the `MySqlTable` table in the `MyDatabase` database in server `myserver`.
+### Asynchronous export to SQL table
-```kusto
+In the following example, Kusto runs the query and then exports the first record set produced by the query to the `MySqlTable` table in the `MyDatabase` database in server `myserver`.
+
+```kusto
.export async to sql MySqlTable
h@"Server=tcp:myserver.database.windows.net,1433;Authentication=Active Directory Integrated;Initial Catalog=MyDatabase;Connection Timeout=30;"
<| print Id="d3b68d12-cbd3-428b-807f-2c740f561989", Name="YSO4", DateOfBirth=datetime(2017-10-15)
```
-In this example, Kusto runs the query and then exports the first record set produced by the query to the `MySqlTable` table in the `MyDatabase` database in server `myserver`.
-If the target table doesn't exist in the target database, it's created.
+### Export to SQL table if it doesn't exist
+
+In the following example, Kusto runs the query and then exports the first record set produced by the query to the `MySqlTable` table in the `MyDatabase` database in server `myserver`.
+The target table is created if it doesn't exist in the target database.
-```kusto
+```kusto
.export async to sql ['dbo.MySqlTable']
h@"Server=tcp:myserver.database.windows.net,1433;Authentication=Active Directory Integrated;Initial Catalog=MyDatabase;Connection Timeout=30;"
with (createifnotexists="true", primarykey="Id")
<| print Message = "Hello World!", Timestamp = now(), Id=12345678
```
+
+## Related content
+
+* [Ingest from query](../data-ingestion/ingest-from-query.md)
+* [Management commands overview](../index.md)
+* [Export to an external table](export-data-to-an-external-table.md)
+* [Export data to cloud storage](export-data-to-storage.md)
+* [Continuous data export](continuous-data-export.md)
diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md
index e64fde135f..b48fb7044c 100644
--- a/data-explorer/kusto/management/data-export/export-data-to-storage.md
+++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md
@@ -3,7 +3,7 @@ title: Export data to cloud storage
description: Learn how to export data to cloud storage.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/26/2024
---
# Export data to cloud storage
@@ -25,10 +25,10 @@ You must have at least [Database Viewer](../../access-control/role-based-access-
| Name | Type | Required | Description |
|--|--|--|--|
-| `async` | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode).|
-| `compressed` | `string` | | If specified, the output storage artifacts are compressed as `.gz` files. See the `compressionType` [supported property](#supported-properties) for compressing Parquet files as snappy.|
-| *OutputDataFormat* | `string` | :heavy_check_mark: | Indicates the data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`.|
-| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that indicate which storage to write the data to. More than one storage connection string may be specified for scalable writes. Each such connection string must indicate the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs.|
+| `async` | `string` | | If specified, the command runs in asynchronous mode. See [asynchronous mode](#asynchronous-mode). |
+| `compressed` | `string` | | If specified, the output storage artifacts are compressed as `.gz` files. See the `compressionType` [supported property](#supported-properties) for compressing Parquet files as snappy. |
+| *OutputDataFormat* | `string` | :heavy_check_mark: | Indicates the data format of the storage artifacts written by the command. Supported values are: `csv`, `tsv`, `json`, and `parquet`. |
+| *StorageConnectionString* | `string` | | One or more [storage connection strings](../../api/connection-strings/storage-connection-strings.md) that indicate which storage to write the data to. More than one storage connection string might be specified for scalable writes. Each such connection string must indicate the credentials to use when writing to storage. For example, when writing to Azure Blob Storage, the credentials can be the storage account key, or a shared access key (SAS) with the permissions to read, write, and list blobs. |
| *PropertyName*, *PropertyValue* | `string` | | A comma-separated list of key-value property pairs. See [supported properties](#supported-properties).|
> [!NOTE]
@@ -45,9 +45,9 @@ You must have at least [Database Viewer](../../access-control/role-based-access-
| `compressionType` | `string` | Indicates the type of compression to use. Possible values are `gzip` or `snappy`. Default is `gzip`. `snappy` can (optionally) be used for `parquet` format. |
| `distribution` | `string` | Distribution hint (`single`, `per_node`, `per_shard`). If value equals `single`, a single thread writes to storage. Otherwise, export writes from all nodes executing the query in parallel. See [evaluate plugin operator](../../query/evaluate-operator.md). Defaults to `per_shard`. |
| `persistDetails` | `bool` | Indicates that the command should persist its results (see `async` flag). Defaults to `true` in async runs, but can be turned off if the caller doesn't require the results). Defaults to `false` in synchronous executions, but can be turned on in those as well. |
-| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact being written (prior to compression). Valid range: 100 MB (default) to 4 GB. |
+| `sizeLimit` | `long` | The size limit in bytes of a single storage artifact written before compression. Valid range: 100 MB (default) to 4 GB. |
| `parquetRowGroupSize` | `int` | Relevant only when data format is Parquet. Controls the row group size in the exported files. Default row group size is 100,000 records. |
-| `distributed` | `bool` | Disable/enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. |
+| `distributed` | `bool` | Disable or enable distributed export. Setting to false is equivalent to `single` distribution hint. Default is true. |
| `parquetDatetimePrecision` | `string` | Specifies the precision to use when exporting `datetime` values to Parquet. Possible values are millisecond and microsecond. Default is millisecond. |
## Authentication and authorization
@@ -93,7 +93,7 @@ For example, after a successful completion, you can retrieve the results using:
## Examples
-In this example, Kusto runs the query and then exports the first recordset produced by the query to one or more compressed CSV blobs, up to 1GB before compression.
+In this example, Kusto runs the query and then exports the first recordset produced by the query to one or more compressed CSV blobs, up to 1 GB before compression.
Column name labels are added as the first row for each blob.
```kusto
@@ -114,22 +114,22 @@ Column name labels are added as the first row for each blob.
## Failures during export commands
-Export commands can transiently fail during execution. [Continuous export](continuous-data-export.md) will automatically retry the command. Regular export commands ([export to storage](export-data-to-storage.md), [export to external table](export-data-to-an-external-table.md)) don't perform any retries.
+Export commands can transiently fail during execution. [Continuous export](continuous-data-export.md) automatically retries the command. Regular export commands ([export to storage](export-data-to-storage.md), [export to external table](export-data-to-an-external-table.md)) don't perform any retries.
* When the export command fails, artifacts that were already written to storage aren't deleted. These artifacts remain in storage. If the command fails, assume the export is incomplete, even if some artifacts were written.
* The best way to track both completion of the command and the artifacts exported upon successful completion is by using the [`.show operations`](../show-operations.md) and [`.show operation details`](../show-operation-details.md) commands.
### Storage failures
-By default, export commands are distributed such that there may be many concurrent writes to storage. The level of distribution depends on the type of export command:
+By default, export commands are distributed such that there might be many concurrent writes to storage. The level of distribution depends on the type of export command:
* The default distribution for regular `.export` command is `per_shard`, which means all [extents](../extents-overview.md) that contain data to export write to storage concurrently.
* The default distribution for [export to external table](export-data-to-an-external-table.md) commands is `per_node`, which means the concurrency is the number of nodes.
-When the number of extents/nodes is large, this may lead to high load on storage that results in storage throttling, or transient storage errors. The following suggestions may overcome these errors (by order of priority):
+When the number of extents/nodes is large, this might lead to high load on storage that results in storage throttling, or transient storage errors. The following suggestions might overcome these errors (by order of priority):
-* Increase the number of storage accounts provided to the export command or to the [external table definition](../external-tables-azure-storage.md) (the load will be evenly distributed between the accounts).
+* Increase the number of storage accounts provided to the export command or to the [external table definition](../external-tables-azure-storage.md). The load is evenly distributed between the accounts.
* Reduce the concurrency by setting the distribution hint to `per_node` (see command properties).
* Reduce concurrency of number of nodes exporting by setting the [client request property](../../api/rest/request-properties.md) `query_fanout_nodes_percent` to the desired concurrency (percent of nodes). The property can be set as part of the export query. For example, the following command limits the number of nodes writing to storage concurrently to 50% of the nodes:
@@ -160,7 +160,7 @@ When the number of extents/nodes is large, this may lead to high load on storage
```
* If exporting to a partitioned external table, setting the `spread`/`concurrency` properties can reduce concurrency (see details in the [command properties](export-data-to-an-external-table.md#syntax).
-* If neither of the above work, you can completely disable distribution by setting the `distributed` property to false. However, we don't recommend doing so, as it may significantly affect the command performance.
+* If neither of the previous recommendations work, you can completely disable distribution by setting the `distributed` property to false. However, we don't recommend doing so, as it might significantly affect the command performance.
### Authorization failures
@@ -184,3 +184,11 @@ On export, Kusto data types are mapped to Parquet data types using the following
| `string` | `BYTE_ARRAY` | UTF-8 | |
| `timespan` | `INT64` | | Stored as ticks (100-nanosecond units) count |
| `decimal` | `FIXED_LENGTH_BYTE_ARRAY` | DECIMAL | |
+
+## Related content
+
+* [Ingest from storage](../data-ingestion/ingest-from-storage.md)
+* [Management commands overview](../index.md)
+* [Export to an external table](export-data-to-an-external-table.md)
+* [Export to a SQL table](export-data-to-sql.md)
+* [Continuous data export](continuous-data-export.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/index.md b/data-explorer/kusto/management/data-export/index.md
index 52915fe666..d560914102 100644
--- a/data-explorer/kusto/management/data-export/index.md
+++ b/data-explorer/kusto/management/data-export/index.md
@@ -3,7 +3,7 @@ title: Data export
description: Learn how to export data.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/27/2024
---
# Data export
@@ -15,7 +15,7 @@ For examples on data export, see [Related content](#related-content).
## Client-side export
-Client-side export gives you control over saving query results either to the local file system or pushing them to a preferred storage location. This flexibility is facilitated through the use of [Kusto client libraries](../../api/client-libraries.md). You can [create an app to run queries](../../api/get-started/app-basic-query.md), read the desired data, and implement an export process tailored to your requirements.
+Client-side export gives you control over saving query results either to the local file system or pushing them to a preferred storage location. This flexibility is facilitated by using [Kusto client libraries](../../api/client-libraries.md). You can [create an app to run queries](../../api/get-started/app-basic-query.md), read the desired data, and implement an export process tailored to your requirements.
::: moniker range="azure-data-explorer"
Alternatively, you can use a client tool like the Azure Data Explorer web UI to export data from your Kusto cluster. For more information, see [Share queries](/azure/data-explorer/web-share-queries).
diff --git a/data-explorer/kusto/management/data-export/show-continuous-artifacts.md b/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
index 076a620cf5..9a35b4ee0a 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
@@ -3,7 +3,7 @@ title: Show continuous data export artifacts
description: This article describes how to show continuous data export artifacts.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/27/2024
---
# Show continuous export artifacts
@@ -12,14 +12,13 @@ ms.date: 08/11/2024
Returns all artifacts exported by the continuous-export in all runs. Filter the results by the Timestamp column in the command to view only records of interest. The history of exported artifacts is retained for 14 days.
::: moniker range="azure-data-explorer"
-The command will not return any results if executed on a [follower database](/azure/data-explorer/follower), it must be executed against the leader database.
+The command doesn't return any results if executed on a [follower database](/azure/data-explorer/follower), it must be executed against the leader database.
::: moniker-end
-::: moniker range="azure-data-explorer"
-The command will not return any results if executed on a [database shortcut](/fabric/real-time-intelligence/database-shortcut), it must be executed against the leader database.
+::: moniker range="microsoft-fabric"
+The command doesn't return any results if executed on a [database shortcut](/fabric/real-time-intelligence/database-shortcut), it must be executed against the leader database.
::: moniker-end
-
## Permissions
You must have at least Database Monitor or Database Admin permissions to run this command. For more information, see [role-based access control](../../access-control/role-based-access-control.md).
@@ -38,19 +37,29 @@ You must have at least Database Monitor or Database Admin permissions to run thi
## Returns
-| Output parameter | Type | Description |
-|-------------------|----------|----------------------------------------|
-| Timestamp | `datetime` | Timestamp of the continuous export run |
-| ExternalTableName | `string` | Name of the external table |
-| Path | `string` | Output path |
-| NumRecords | `long` | Number of records exported to path |
+| Output parameter | Type | Description |
+|--|--|--|
+| Timestamp | `datetime` | THe tTimestamp of the continuous export run |
+| ExternalTableName | `string` | Name of the external table |
+| Path | `string` | Output path |
+| NumRecords | `long` | Number of records exported to path |
## Example
+The following example shows retrieved artifacts from the continuous export `MyExport` that were exported within the last hour.
+
```kusto
.show continuous-export MyExport exported-artifacts | where Timestamp > ago(1h)
```
-| Timestamp | ExternalTableName | Path | NumRecords | SizeInBytes |
-|-----------------------------|-------------------|------------------|------------|-------------|
-| 2018-12-20 07:31:30.2634216 | ExternalBlob | `http://storageaccount.blob.core.windows.net/container1/1_6ca073fd4c8740ec9a2f574eaa98f579.csv` | 10 | 1024 |
+**Output**
+
+| Timestamp | ExternalTableName | Path | NumRecords | SizeInBytes |
+|--|--|--|--|--|
+| 2018-12-20 07:31:30.2634216 | ExternalBlob | `http://storageaccount.blob.core.windows.net/container1/1_6ca073fd4c8740ec9a2f574eaa98f579.csv` | 10 | 1024 |
+
+## Related content
+
+* [External tables](../../query/schema-entities/external-tables.md)
+* [Continuous data export overview](continuous-data-export.md)
+* [Show continuous export failures](show-continuous-failures.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/show-continuous-export.md b/data-explorer/kusto/management/data-export/show-continuous-export.md
index 3f934fda38..f0c7916187 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-export.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-export.md
@@ -31,18 +31,26 @@ You must have at least Database User, Database Viewer, or Database Monitor permi
## Returns
-| Output parameter | Type | Description |
-|---------------------|----------|-------------------------------------------------------------------------|
-| CursorScopedTables | `string` | List of explicitly scoped (fact) tables (JSON serialized) |
-| ExportProperties | `string` | Export properties (JSON serialized) |
-| ExportedTo | `datetime` | The last datetime (ingestion time) that was exported successfully |
-| ExternalTableName | `string` | Name of the external table |
-| ForcedLatency | `timeSpan` | Forced latency (null if not provided) |
-| IntervalBetweenRuns | `timeSpan` | Interval between runs |
-| IsDisabled | `bool` | True if the continuous export is disabled |
-| IsRunning | `bool` | True if the continuous export is currently running |
-| LastRunResult | `string` | The results of the last continuous-export run (`Completed` or `Failed`) |
-| LastRunTime | `datetime` | The last time the continuous export was executed (start time) |
-| Name | `string` | Name of the continuous export |
-| Query | `string` | Export query |
-| StartCursor | `string` | Starting point of the first execution of this continuous export |
\ No newline at end of file
+| Output parameter | Type | Description |
+|--|--|--|
+| CursorScopedTables | `string` | The list of explicitly scoped (fact) tables (JSON serialized). |
+| ExportProperties | `string` | The export properties (JSON serialized). |
+| ExportedTo | `datetime` | The last datetime (ingestion time) that was exported successfully. |
+| ExternalTableName | `string` | The external table name. |
+| ForcedLatency | `timeSpan` | The forced latency timespan, if defined. Returns `Null` if no timespan is defined. |
+| IntervalBetweenRuns | `timeSpan` | The interval between runs. |
+| IsDisabled | `bool` | A boolean value indicating whether the continuous export is disabled. |
+| IsRunning | `bool` | A boolean value indicating whether the continuous export is currently running. |
+| LastRunResult | `string` | The results of the last continuous-export run (`Completed` or `Failed`). |
+| LastRunTime | `datetime` | The last time the continuous export was executed (start time) |
+| Name | `string` | The name of the continuous export. |
+| Query | `string` | The export query. |
+| StartCursor | `string` | The starting point of the first execution of this continuous export. |
+
+## Related content
+
+* [External tables](../../query/schema-entities/external-tables.md)
+* [Create or alter continuous export](create-alter-continuous.md)
+* [Disable or enable continuous export](disable-enable-continuous.md)
+* [Drop continuous export](drop-continuous-export.md)
+* [Show continuous export failures](show-continuous-failures.md)
diff --git a/data-explorer/kusto/management/data-export/show-continuous-failures.md b/data-explorer/kusto/management/data-export/show-continuous-failures.md
index 3fca9ae696..81bde2c053 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-failures.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-failures.md
@@ -3,20 +3,20 @@ title: Show continuous data export failures
description: This article describes how to show continuous data export failures.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 11/27/2024
---
# Show continuous export failures
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
-Returns all failures logged as part of the continuous export. Filter the results by the Timestamp column in the command to view only time range of interest.
+Returns all failures logged as part of the continuous export. To view only a specific time range, filter the results by the **Timestamp** column.
::: moniker range="azure-data-explorer"
-The command will not return any results if executed on a [follower database](/azure/data-explorer/follower), it must be executed against the leader database.
+The command doesn't return any results if executed on a [follower database](/azure/data-explorer/follower), it must be executed against the leader database.
::: moniker-end
::: moniker range="microsoft-fabric"
-The command will not return any results if executed on a [database shortcut](/fabric/real-time-intelligence/database-shortcut), it must be executed against the leader database.
+The command doesn't return any results if executed on a [database shortcut](/fabric/real-time-intelligence/database-shortcut), it must be executed against the leader database.
::: moniker-end
## Permissions
@@ -37,21 +37,33 @@ You must have at least Database Monitor or Database Admin permissions to run thi
## Returns
-| Output parameter | Type | Description |
-|------------------|-----------|-----------------------------------------------------|
-| Timestamp | `datetime` | Timestamp of the failure. |
-| OperationId | `string` | Operation ID of the failure. |
-| Name | `string` | Continuous export name. |
-| LastSuccessRun | Timestamp | The last successful run of the continuous export. |
-| FailureKind | `string` | Failure/PartialFailure. PartialFailure indicates some artifacts were exported successfully before the failure occurred. |
-| Details | `string` | Failure error details. |
+| Output parameter | Type | Description |
+|--|--|--|
+| Timestamp | `datetime` | Timestamp of the failure. |
+| OperationId | `string` | Operation ID of the failure. |
+| Name | `string` | Continuous export name. |
+| LastSuccessRun | Timestamp | The last successful run of the continuous export. |
+| FailureKind | `string` | Failure/PartialFailure. PartialFailure indicates some artifacts were exported successfully before the failure occurred. |
+| Details | `string` | Failure error details. |
## Example
+The following example shows failures from the continuous export `MyExport`.
+
```kusto
.show continuous-export MyExport failures
```
+**Output**
+
| Timestamp | OperationId | Name | LastSuccessRun | FailureKind | Details |
|-----------------------------|--------------------------------------|----------|-----------------------------|-------------|------------|
| 2019-01-01 11:07:41.1887304 | ec641435-2505-4532-ba19-d6ab88c96a9d | MyExport | 2019-01-01 11:06:35.6308140 | Failure | Details... |
+
+## Related content
+
+* [External tables](../../query/schema-entities/external-tables.md)
+* [Create or alter continuous export](create-alter-continuous.md)
+* [Disable or enable continuous export](disable-enable-continuous.md)
+* [Show continuous export](show-continuous-export.md)
+* [Drop continuous export](drop-continuous-export.md)
From 118ea9c1e5b1fccf4df28701ca1a9abf0424437a Mon Sep 17 00:00:00 2001
From: Meira Josephy <144697924+mjosephym@users.noreply.github.com>
Date: Wed, 27 Nov 2024 11:11:19 +0200
Subject: [PATCH 02/12] edits
---
.../kusto/management/data-export/continuous-data-export.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/data-explorer/kusto/management/data-export/continuous-data-export.md b/data-explorer/kusto/management/data-export/continuous-data-export.md
index 59534c052c..571153f364 100644
--- a/data-explorer/kusto/management/data-export/continuous-data-export.md
+++ b/data-explorer/kusto/management/data-export/continuous-data-export.md
@@ -165,7 +165,7 @@ To define continuous export to a delta table, do the following steps:
:::moniker range="microsoft-fabric"
* [Create or alter continuous export](create-alter-continuous.md)
* [External tables](../../query/schema-entities/external-tables.md)
-:::monker-end
+:::moniker-end
:::moniker-range="azure-data-explorer"
* [Create or alter continuous export](create-alter-continuous.md)
From 2d8df186b5cd7af5803fa074f86c32a558f9bb13 Mon Sep 17 00:00:00 2001
From: Meira Josephy <144697924+mjosephym@users.noreply.github.com>
Date: Wed, 27 Nov 2024 11:14:36 +0200
Subject: [PATCH 03/12] edits
---
.../kusto/management/data-export/continuous-data-export.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/data-explorer/kusto/management/data-export/continuous-data-export.md b/data-explorer/kusto/management/data-export/continuous-data-export.md
index 571153f364..5c6b4609ec 100644
--- a/data-explorer/kusto/management/data-export/continuous-data-export.md
+++ b/data-explorer/kusto/management/data-export/continuous-data-export.md
@@ -171,4 +171,4 @@ To define continuous export to a delta table, do the following steps:
* [Create or alter continuous export](create-alter-continuous.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md)
-:::monker-end
+:::moniker-end
From 6b8cbcd479b043acc1c1cb5b485990182cf98408 Mon Sep 17 00:00:00 2001
From: Meira Josephy <144697924+mjosephym@users.noreply.github.com>
Date: Wed, 27 Nov 2024 11:18:40 +0200
Subject: [PATCH 04/12] edits
---
.../kusto/management/data-export/continuous-data-export.md | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/data-explorer/kusto/management/data-export/continuous-data-export.md b/data-explorer/kusto/management/data-export/continuous-data-export.md
index 5c6b4609ec..9e9ac34fce 100644
--- a/data-explorer/kusto/management/data-export/continuous-data-export.md
+++ b/data-explorer/kusto/management/data-export/continuous-data-export.md
@@ -163,12 +163,15 @@ To define continuous export to a delta table, do the following steps:
## Related content
:::moniker range="microsoft-fabric"
+
* [Create or alter continuous export](create-alter-continuous.md)
* [External tables](../../query/schema-entities/external-tables.md)
:::moniker-end
-:::moniker-range="azure-data-explorer"
+:::moniker range="azure-data-explorer"
+
* [Create or alter continuous export](create-alter-continuous.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md)
+
:::moniker-end
From 16a3a72531bf78dec2bdf9d0908d6b5ca227a485 Mon Sep 17 00:00:00 2001
From: Meira Josephy <144697924+mjosephym@users.noreply.github.com>
Date: Sun, 1 Dec 2024 11:50:59 +0200
Subject: [PATCH 05/12] edits
---
.../data-export/continuous-export-with-managed-identity.md | 4 ++--
.../kusto/management/data-export/create-alter-continuous.md | 3 ++-
.../kusto/management/data-export/disable-enable-continuous.md | 3 ++-
.../kusto/management/data-export/drop-continuous-export.md | 3 ++-
.../data-export/export-data-to-an-external-table.md | 4 ++--
.../kusto/management/data-export/export-data-to-sql.md | 4 ++--
.../kusto/management/data-export/export-data-to-storage.md | 4 ++--
.../kusto/management/data-export/show-continuous-artifacts.md | 2 +-
.../kusto/management/data-export/show-continuous-export.md | 1 +
.../kusto/management/data-export/show-continuous-failures.md | 1 +
10 files changed, 17 insertions(+), 12 deletions(-)
diff --git a/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md b/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md
index be84af65c9..18bda0259b 100644
--- a/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md
+++ b/data-explorer/kusto/management/data-export/continuous-export-with-managed-identity.md
@@ -3,7 +3,7 @@ title: Use a managed identity to run a continuous export job
description: This article describes how to use a managed identity for continuous export.
ms.reviewer: shanisolomon
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 12/01/2024
monikerRange: "azure-data-explorer"
---
# Use a managed identity to run a continuous export job
@@ -173,6 +173,6 @@ For example, the following command creates a continuous export job named `MyExpo
## Related content
-* [.show continuous-exports](show-continuous-export.md)
* [Continuous export overview](continuous-data-export.md)
+* [.show continuous-exports](show-continuous-export.md)
* [Managed identities](/azure/data-explorer/managed-identities-overview)
diff --git a/data-explorer/kusto/management/data-export/create-alter-continuous.md b/data-explorer/kusto/management/data-export/create-alter-continuous.md
index 34e7786ff2..6b03a91db8 100644
--- a/data-explorer/kusto/management/data-export/create-alter-continuous.md
+++ b/data-explorer/kusto/management/data-export/create-alter-continuous.md
@@ -3,7 +3,7 @@ title: Create or alter continuous data export
description: This article describes how to create or alter continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 11/26/2024
+ms.date: 12/01/2024
---
# Create or alter continuous export
@@ -67,6 +67,7 @@ with
## Related content
+* [Continuous export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Disable or enable continuous export](disable-enable-continuous.md)
* [Show continuous export](show-continuous-export.md)
diff --git a/data-explorer/kusto/management/data-export/disable-enable-continuous.md b/data-explorer/kusto/management/data-export/disable-enable-continuous.md
index 661eb75812..2a8e3d4332 100644
--- a/data-explorer/kusto/management/data-export/disable-enable-continuous.md
+++ b/data-explorer/kusto/management/data-export/disable-enable-continuous.md
@@ -3,7 +3,7 @@ title: Enable or disable continuous data export
description: This article describes how to disable or enable continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 12/01/2024
---
# Disable or enable continuous export
@@ -37,6 +37,7 @@ The result of the [show continuous export command](show-continuous-export.md) of
## Related content
+* [Continuous export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Create or alter continuous export](create-alter-continuous.md)
* [Drop continuous export](drop-continuous-export.md)
diff --git a/data-explorer/kusto/management/data-export/drop-continuous-export.md b/data-explorer/kusto/management/data-export/drop-continuous-export.md
index 62160b8fb4..b7a9024ccc 100644
--- a/data-explorer/kusto/management/data-export/drop-continuous-export.md
+++ b/data-explorer/kusto/management/data-export/drop-continuous-export.md
@@ -3,7 +3,7 @@ title: Drop continuous data export
description: This article describes how to drop continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 11/27/2024
+ms.date: 12/01/2024
---
# Drop continuous export
@@ -33,6 +33,7 @@ The remaining continuous exports in the database (post deletion). Output schema
## Related content
+* [Continuous export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Create or alter continuous export](create-alter-continuous.md)
* [Disable or enable continuous export](disable-enable-continuous.md)
diff --git a/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md b/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md
index 2581f5251d..1b143a6480 100644
--- a/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md
+++ b/data-explorer/kusto/management/data-export/export-data-to-an-external-table.md
@@ -3,7 +3,7 @@ title: Export data to an external table
description: This article describes Export data to an external table.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 11/26/2024
+ms.date: 12/01/2024
---
# Export data to an external table
@@ -146,8 +146,8 @@ If the command is executed asynchronously by using the `async` keyword, the outp
## Related content
+* [Continuous data export](continuous-data-export.md)
* [Management commands overview](../index.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Export data to SQL](export-data-to-sql.md)
* [Export data to cloud storage](export-data-to-storage.md)
-* [Continuous data export](continuous-data-export.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/export-data-to-sql.md b/data-explorer/kusto/management/data-export/export-data-to-sql.md
index 74c52e2fce..7aac410618 100644
--- a/data-explorer/kusto/management/data-export/export-data-to-sql.md
+++ b/data-explorer/kusto/management/data-export/export-data-to-sql.md
@@ -3,7 +3,7 @@ title: Export data to SQL
description: This article describes Export data to SQL.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 11/26/2024
+ms.date: 12/01/2024
---
# Export data to SQL
@@ -126,4 +126,4 @@ The target table is created if it doesn't exist in the target database.
* [Management commands overview](../index.md)
* [Export to an external table](export-data-to-an-external-table.md)
* [Export data to cloud storage](export-data-to-storage.md)
-* [Continuous data export](continuous-data-export.md)
+* [Create and alter Azure SQL external tables](../external-sql-tables.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/export-data-to-storage.md b/data-explorer/kusto/management/data-export/export-data-to-storage.md
index b48fb7044c..0ce7bba2ca 100644
--- a/data-explorer/kusto/management/data-export/export-data-to-storage.md
+++ b/data-explorer/kusto/management/data-export/export-data-to-storage.md
@@ -3,7 +3,7 @@ title: Export data to cloud storage
description: Learn how to export data to cloud storage.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 11/26/2024
+ms.date: 12/01/2024
---
# Export data to cloud storage
@@ -187,8 +187,8 @@ On export, Kusto data types are mapped to Parquet data types using the following
## Related content
+* [Continuous data export](continuous-data-export.md)
* [Ingest from storage](../data-ingestion/ingest-from-storage.md)
* [Management commands overview](../index.md)
* [Export to an external table](export-data-to-an-external-table.md)
* [Export to a SQL table](export-data-to-sql.md)
-* [Continuous data export](continuous-data-export.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/show-continuous-artifacts.md b/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
index 9a35b4ee0a..5ae308f53d 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
@@ -60,6 +60,6 @@ The following example shows retrieved artifacts from the continuous export `MyEx
## Related content
-* [External tables](../../query/schema-entities/external-tables.md)
* [Continuous data export overview](continuous-data-export.md)
+* [External tables](../../query/schema-entities/external-tables.md)
* [Show continuous export failures](show-continuous-failures.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/show-continuous-export.md b/data-explorer/kusto/management/data-export/show-continuous-export.md
index f0c7916187..1336968386 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-export.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-export.md
@@ -49,6 +49,7 @@ You must have at least Database User, Database Viewer, or Database Monitor permi
## Related content
+* [Continuous data export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Create or alter continuous export](create-alter-continuous.md)
* [Disable or enable continuous export](disable-enable-continuous.md)
diff --git a/data-explorer/kusto/management/data-export/show-continuous-failures.md b/data-explorer/kusto/management/data-export/show-continuous-failures.md
index 81bde2c053..a1afbd0afb 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-failures.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-failures.md
@@ -62,6 +62,7 @@ The following example shows failures from the continuous export `MyExport`.
## Related content
+* [Continuous data export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Create or alter continuous export](create-alter-continuous.md)
* [Disable or enable continuous export](disable-enable-continuous.md)
From 64250f234de5f98251d7406b6156fae56b07a632 Mon Sep 17 00:00:00 2001
From: Meira Josephy <144697924+mjosephym@users.noreply.github.com>
Date: Sun, 8 Dec 2024 22:26:14 +0200
Subject: [PATCH 06/12] edit
---
.../data-export/continuous-data-export.md | 6 +++---
.../data-export/create-alter-continuous.md | 13 +++++++------
.../data-export/drop-continuous-export.md | 14 +++++++-------
.../kusto/management/data-export/index.md | 2 +-
.../data-export/show-continuous-artifacts.md | 8 ++++----
.../data-export/show-continuous-export.md | 14 +++++++-------
.../data-export/show-continuous-failures.md | 14 +++++++-------
data-explorer/kusto/management/toc.yml | 10 +++++-----
8 files changed, 41 insertions(+), 40 deletions(-)
diff --git a/data-explorer/kusto/management/data-export/continuous-data-export.md b/data-explorer/kusto/management/data-export/continuous-data-export.md
index 9e9ac34fce..e52817257b 100644
--- a/data-explorer/kusto/management/data-export/continuous-data-export.md
+++ b/data-explorer/kusto/management/data-export/continuous-data-export.md
@@ -3,7 +3,7 @@ title: Continuous data export
description: This article describes Continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 12/08/2024
---
# Continuous data export overview
@@ -164,13 +164,13 @@ To define continuous export to a delta table, do the following steps:
:::moniker range="microsoft-fabric"
-* [Create or alter continuous export](create-alter-continuous.md)
+* [.create or alter continuous-export](create-alter-continuous.md)
* [External tables](../../query/schema-entities/external-tables.md)
:::moniker-end
:::moniker range="azure-data-explorer"
-* [Create or alter continuous export](create-alter-continuous.md)
+* [.create or alter continuous-export](create-alter-continuous.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Use a managed identity to run a continuous export job](continuous-export-with-managed-identity.md)
diff --git a/data-explorer/kusto/management/data-export/create-alter-continuous.md b/data-explorer/kusto/management/data-export/create-alter-continuous.md
index 6b03a91db8..68e32eac7a 100644
--- a/data-explorer/kusto/management/data-export/create-alter-continuous.md
+++ b/data-explorer/kusto/management/data-export/create-alter-continuous.md
@@ -1,11 +1,11 @@
---
-title: Create or alter continuous data export
+title: .create or alter continuous-export
description: This article describes how to create or alter continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 12/01/2024
+ms.date: 12/08/2024
---
-# Create or alter continuous export
+# .create or alter continuous-export
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
@@ -70,6 +70,7 @@ with
* [Continuous export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
* [Disable or enable continuous export](disable-enable-continuous.md)
-* [Show continuous export](show-continuous-export.md)
-* [Drop continuous export](drop-continuous-export.md)
-* [Show continuous export failures](show-continuous-failures.md)
\ No newline at end of file
+* [.show continuous-exports](show-continuous-export.md)
+* [.show continuous-export](show-continuous-export.md)
+* [.drop continuous-export](drop-continuous-export.md)
+* [.show continuous-export failures](show-continuous-failures.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/drop-continuous-export.md b/data-explorer/kusto/management/data-export/drop-continuous-export.md
index b7a9024ccc..e1b8cd9b4c 100644
--- a/data-explorer/kusto/management/data-export/drop-continuous-export.md
+++ b/data-explorer/kusto/management/data-export/drop-continuous-export.md
@@ -1,11 +1,11 @@
---
-title: Drop continuous data export
+title: .drop continuous-export
description: This article describes how to drop continuous data export.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 12/01/2024
+ms.date: 12/08/2024
---
-# Drop continuous export
+# .drop continuous-export
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
@@ -35,7 +35,7 @@ The remaining continuous exports in the database (post deletion). Output schema
* [Continuous export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
-* [Create or alter continuous export](create-alter-continuous.md)
-* [Disable or enable continuous export](disable-enable-continuous.md)
-* [Show continuous export](show-continuous-export.md)
-* [Show continuous export failures](show-continuous-failures.md)
\ No newline at end of file
+* [.create or alter continuous-export](create-alter-continuous.md)
+* [.disable or enable continuous-export](disable-enable-continuous.md)
+* [.show continuous-export](show-continuous-export.md)
+* [.show continuous-export failures](show-continuous-failures.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/index.md b/data-explorer/kusto/management/data-export/index.md
index d560914102..c99354ec7d 100644
--- a/data-explorer/kusto/management/data-export/index.md
+++ b/data-explorer/kusto/management/data-export/index.md
@@ -39,4 +39,4 @@ For scalable data export, the service offers various `.export` management comman
* [Export to cloud storage](export-data-to-storage.md)
* [Export to an external table](export-data-to-an-external-table.md)
* [Export to a SQL table](export-data-to-sql.md)
-* [Continuous data export](continuous-data-export.md)
+* [Continuous data export overview](continuous-data-export.md)
diff --git a/data-explorer/kusto/management/data-export/show-continuous-artifacts.md b/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
index 5ae308f53d..b4a89c9137 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-artifacts.md
@@ -1,11 +1,11 @@
---
-title: Show continuous data export artifacts
+title: .show continuous-export exported-artifacts
description: This article describes how to show continuous data export artifacts.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 11/27/2024
+ms.date: 12/08/2024
---
-# Show continuous export artifacts
+# .show continuous-export exported-artifacts
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
@@ -62,4 +62,4 @@ The following example shows retrieved artifacts from the continuous export `MyEx
* [Continuous data export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
-* [Show continuous export failures](show-continuous-failures.md)
\ No newline at end of file
+* [.show continuous-export failures](show-continuous-failures.md)
\ No newline at end of file
diff --git a/data-explorer/kusto/management/data-export/show-continuous-export.md b/data-explorer/kusto/management/data-export/show-continuous-export.md
index 1336968386..2cddf74d5f 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-export.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-export.md
@@ -1,11 +1,11 @@
---
-title: Show continuous data export
+title: .show continuous-export
description: This article describes how to show continuous data export properties.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 12/08/2024
---
-# Show continuous export
+# .show continuous-export
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
@@ -51,7 +51,7 @@ You must have at least Database User, Database Viewer, or Database Monitor permi
* [Continuous data export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
-* [Create or alter continuous export](create-alter-continuous.md)
-* [Disable or enable continuous export](disable-enable-continuous.md)
-* [Drop continuous export](drop-continuous-export.md)
-* [Show continuous export failures](show-continuous-failures.md)
+* [.create or alter continuous-export](create-alter-continuous.md)
+* [.disable or enable continuous-export](disable-enable-continuous.md)
+* [.drop continuous-export](drop-continuous-export.md)
+* [.show continuous-export failures](show-continuous-failures.md)
diff --git a/data-explorer/kusto/management/data-export/show-continuous-failures.md b/data-explorer/kusto/management/data-export/show-continuous-failures.md
index a1afbd0afb..8802d2ed16 100644
--- a/data-explorer/kusto/management/data-export/show-continuous-failures.md
+++ b/data-explorer/kusto/management/data-export/show-continuous-failures.md
@@ -1,11 +1,11 @@
---
-title: Show continuous data export failures
+title: .show continuous data-export failures
description: This article describes how to show continuous data export failures.
ms.reviewer: yifats
ms.topic: reference
-ms.date: 11/27/2024
+ms.date: 12/08/2024
---
-# Show continuous export failures
+# .show continuous-export failures
> [!INCLUDE [applies](../../includes/applies-to-version/applies.md)] [!INCLUDE [fabric](../../includes/applies-to-version/fabric.md)] [!INCLUDE [azure-data-explorer](../../includes/applies-to-version/azure-data-explorer.md)]
@@ -64,7 +64,7 @@ The following example shows failures from the continuous export `MyExport`.
* [Continuous data export overview](continuous-data-export.md)
* [External tables](../../query/schema-entities/external-tables.md)
-* [Create or alter continuous export](create-alter-continuous.md)
-* [Disable or enable continuous export](disable-enable-continuous.md)
-* [Show continuous export](show-continuous-export.md)
-* [Drop continuous export](drop-continuous-export.md)
+* [.create or alter continuous-export](create-alter-continuous.md)
+* [.disable or enable continuous-export](disable-enable-continuous.md)
+* [.show continuous-export](show-continuous-export.md)
+* [.drop continuous-export](drop-continuous-export.md)
diff --git a/data-explorer/kusto/management/toc.yml b/data-explorer/kusto/management/toc.yml
index f06d605b6f..0c4554f9bf 100644
--- a/data-explorer/kusto/management/toc.yml
+++ b/data-explorer/kusto/management/toc.yml
@@ -854,19 +854,19 @@ items:
items:
- name: Continuous data export overview
href: data-export/continuous-data-export.md
- - name: Create or alter continuous export
+ - name: .create or alter continuous-export
href: data-export/create-alter-continuous.md
- name: Use a managed identity
href: data-export/continuous-export-with-managed-identity.md
- name: Disable or enable continuous export
href: data-export/disable-enable-continuous.md
- - name: Drop continuous export
+ - name: .drop continuous-export
href: data-export/drop-continuous-export.md
- - name: Show continuous export
+ - name: .show continuous-export
href: data-export/show-continuous-export.md
- - name: Show continuous export artifacts
+ - name: .show continuous-export exported-artifacts
href: data-export/show-continuous-artifacts.md
- - name: Show continuous export failures
+ - name: .show continuous-export failures
href: data-export/show-continuous-failures.md
- name: Database cursors
href: database-cursor.md
From 2cce7ca2b1d6207dbc33f60253de8389dae22a86 Mon Sep 17 00:00:00 2001
From: andykop <165172502+AndyKop@users.noreply.github.com>
Date: Mon, 16 Dec 2024 13:00:14 +0200
Subject: [PATCH 07/12] new examples
---
data-explorer/kusto/query/extract-function.md | 47 +++++++++++++++----
1 file changed, 39 insertions(+), 8 deletions(-)
diff --git a/data-explorer/kusto/query/extract-function.md b/data-explorer/kusto/query/extract-function.md
index b7bf8cefff..f2457b3dab 100644
--- a/data-explorer/kusto/query/extract-function.md
+++ b/data-explorer/kusto/query/extract-function.md
@@ -3,7 +3,7 @@ title: extract()
description: Learn how to use the extract() function to get a match for a regular expression from a source string.
ms.reviewer: alexans
ms.topic: reference
-ms.date: 08/11/2024
+ms.date: 12/16/2024
---
# extract()
@@ -36,23 +36,54 @@ If there's no match, or the type conversion fails: `null`.
## Examples
-The example string `Trace` is searched for a definition for `Duration`.
-The match is converted to `real`, then multiplied it by a time constant (`1s`) so that `Duration` is of type `timespan`. In this example, it's equal to 123.45 seconds:
+The following query extracts the month from the string `Dates` and returns a table with the date string and the month.
+
+:::moniker range="azure-data-explorer"
+> [!div class="nextstepaction"]
+> Run the query
+::: moniker-end
```kusto
-T
-| extend Trace="A=1, B=2, Duration=123.45, ..."
-| extend Duration = extract("Duration=([0-9.]+)", 1, Trace, typeof(real)) * time(1s)
+let Dates = datatable(DateString: string)
+[
+ "15-12-2024",
+ "21-07-2023",
+ "10-03-2022"
+];
+Dates
+| extend Month = extract(@"-(\d{2})-", 1, DateString, typeof(int))
+| project DateString, Month
```
-This example is equivalent to `substring(Text, 2, 4)`:
+**Output**
+
+| DateString | Month |
+| ... | ... |
+| 15-12-2024 | 12 |
+| 21-07-2023 | 7 |
+| 10-03-2022 | 3 |
+
+The following example returns the username from the string. The regular expression `User: ([^,]+)` matches the text following "User: " up to the next comma, effectively extracting the username.
+
+:::moniker range="azure-data-explorer"
+> [!div class="nextstepaction"]
+> Run the query
+::: moniker-end
```kusto
-extract("^.{2,2}(.{4,4})", 1, Text)
+let Text = "User: JohnDoe, Email: johndoe@example.com, Age: 29";
+| print UserName = extract("User: ([^,]+)", 1, Text)
```
+**Output**
+
+| UserName |
+| ... |
+| JohnDoe |
+
## Related content
* [extract-all function](extract-all-function.md)
* [extract-json function](extract-json-function.md)
* [parse operator](parse-operator.md)
+* [regular expression](regex.md)
From cc388e5ae53f1119050ad5324a199e5ca6def3dd Mon Sep 17 00:00:00 2001
From: andykop <165172502+AndyKop@users.noreply.github.com>
Date: Mon, 16 Dec 2024 14:12:41 +0200
Subject: [PATCH 08/12] build fix
---
data-explorer/kusto/query/extract-function.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/data-explorer/kusto/query/extract-function.md b/data-explorer/kusto/query/extract-function.md
index f2457b3dab..a218214a9e 100644
--- a/data-explorer/kusto/query/extract-function.md
+++ b/data-explorer/kusto/query/extract-function.md
@@ -58,7 +58,7 @@ Dates
**Output**
| DateString | Month |
-| ... | ... |
+| --- | --- |
| 15-12-2024 | 12 |
| 21-07-2023 | 7 |
| 10-03-2022 | 3 |
@@ -78,7 +78,7 @@ let Text = "User: JohnDoe, Email: johndoe@example.com, Age: 29";
**Output**
| UserName |
-| ... |
+| --- |
| JohnDoe |
## Related content
From 8274fcb2f617d515ba1821186464329e6561125e Mon Sep 17 00:00:00 2001
From: andykop <165172502+AndyKop@users.noreply.github.com>
Date: Mon, 16 Dec 2024 15:41:51 +0200
Subject: [PATCH 09/12] text tweak
---
data-explorer/kusto/query/extract-function.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/data-explorer/kusto/query/extract-function.md b/data-explorer/kusto/query/extract-function.md
index a218214a9e..24ce497a61 100644
--- a/data-explorer/kusto/query/extract-function.md
+++ b/data-explorer/kusto/query/extract-function.md
@@ -63,7 +63,7 @@ Dates
| 21-07-2023 | 7 |
| 10-03-2022 | 3 |
-The following example returns the username from the string. The regular expression `User: ([^,]+)` matches the text following "User: " up to the next comma, effectively extracting the username.
+The following example returns the username from the string. The regular expression `([^,]+)` matches the text following "User: " up to the next comma, effectively extracting the username.
:::moniker range="azure-data-explorer"
> [!div class="nextstepaction"]
From 175095cf38af4111d00d05bbe9a7692c4e17a767 Mon Sep 17 00:00:00 2001
From: andykop <165172502+AndyKop@users.noreply.github.com>
Date: Tue, 17 Dec 2024 09:34:19 +0200
Subject: [PATCH 10/12] add subheadings for examples
---
data-explorer/kusto/query/extract-function.md | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/data-explorer/kusto/query/extract-function.md b/data-explorer/kusto/query/extract-function.md
index 24ce497a61..376cb1a301 100644
--- a/data-explorer/kusto/query/extract-function.md
+++ b/data-explorer/kusto/query/extract-function.md
@@ -36,6 +36,8 @@ If there's no match, or the type conversion fails: `null`.
## Examples
+**Extract value from datetime data**
+
The following query extracts the month from the string `Dates` and returns a table with the date string and the month.
:::moniker range="azure-data-explorer"
@@ -63,6 +65,8 @@ Dates
| 21-07-2023 | 7 |
| 10-03-2022 | 3 |
+**Extract from a text string**
+
The following example returns the username from the string. The regular expression `([^,]+)` matches the text following "User: " up to the next comma, effectively extracting the username.
:::moniker range="azure-data-explorer"
From 4221104721c056a031d54ae85d5767be0e19e7e6 Mon Sep 17 00:00:00 2001
From: Yael Schuster-Davidi
Date: Tue, 17 Dec 2024 09:46:27 +0200
Subject: [PATCH 11/12] fixed incorrect link
---
.../data-ingestion/streaming-ingestion-schema-changes.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/data-explorer/kusto/management/data-ingestion/streaming-ingestion-schema-changes.md b/data-explorer/kusto/management/data-ingestion/streaming-ingestion-schema-changes.md
index b35a5e7dde..9ab8801f53 100644
--- a/data-explorer/kusto/management/data-ingestion/streaming-ingestion-schema-changes.md
+++ b/data-explorer/kusto/management/data-ingestion/streaming-ingestion-schema-changes.md
@@ -3,7 +3,7 @@ title: Streaming ingestion and schema changes
description: This article discusses options of handling schema changes with streaming ingestion.
ms.reviewer: orspodek
ms.topic: reference
-ms.date: 10/13/2024
+ms.date: 12/17/2024
---
# Streaming ingestion and schema changes
@@ -14,7 +14,7 @@ Cluster nodes cache the schema of databases that get data through [streaming ing
::: moniker-end
::: moniker range="microsoft-fabric"
-Eventhouse nodes cache the schema of databases that get data through [streaming ingestion](/azure/data-explorer/ingest-data-streaming), boosting performance and resource use. However, when there are schema changes, it can lead to delays in updates.
+Eventhouse nodes cache the schema of databases that get data through streaming ingestion, boosting performance and resource use. However, when there are schema changes, it can lead to delays in updates.
::: moniker-end
If schema changes and streaming ingestion aren't synchronized, you can encounter failures like schema-related errors or incomplete and distorted data in the table.
From b62a2b4b2f947437812b663bdde7bf6ddd8c9398 Mon Sep 17 00:00:00 2001
From: Shlomo Sagir <51323195+shsagir@users.noreply.github.com>
Date: Tue, 17 Dec 2024 10:11:59 +0200
Subject: [PATCH 12/12] Apply suggestions from code review
---
data-explorer/kusto/query/extract-function.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/data-explorer/kusto/query/extract-function.md b/data-explorer/kusto/query/extract-function.md
index 376cb1a301..164ff80f9d 100644
--- a/data-explorer/kusto/query/extract-function.md
+++ b/data-explorer/kusto/query/extract-function.md
@@ -36,7 +36,7 @@ If there's no match, or the type conversion fails: `null`.
## Examples
-**Extract value from datetime data**
+### Extract month from datetime string
The following query extracts the month from the string `Dates` and returns a table with the date string and the month.
@@ -65,7 +65,7 @@ Dates
| 21-07-2023 | 7 |
| 10-03-2022 | 3 |
-**Extract from a text string**
+### Extract username from a string
The following example returns the username from the string. The regular expression `([^,]+)` matches the text following "User: " up to the next comma, effectively extracting the username.