Add Delete blocks #144
Open
Add Delete blocks #144
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
tothmano
commented
Nov 7, 2024
tothmano
commented
Nov 8, 2024
…into mano/block-deletion
mhr3
reviewed
Dec 17, 2024
| sidebarTitle: 'Delete blocks' | ||
| --- | ||
|
|
||
| After data ingestion, Axiom stores events in blocks of tens of thousands of events. You cannot delete individual events after you send them to Axiom, but you can delete entire blocks of data. With the `/v2/datasets/_apl/delete` endpoint, you can specify an APL query and delete all blocks that contain matching events. |
There was a problem hiding this comment.
You cannot delete individual events after you send them to Axiom
right now you cannot, that will hopefully change in the future
mhr3
reviewed
Dec 17, 2024
| After data ingestion, Axiom stores events in blocks of tens of thousands of events. You cannot delete individual events after you send them to Axiom, but you can delete entire blocks of data. With the `/v2/datasets/_apl/delete` endpoint, you can specify an APL query and delete all blocks that contain matching events. | ||
|
|
||
| <Warning> | ||
| The `/v2/datasets/_apl/delete` endpoint deletes all blocks containing events that match your query. For example, even if a block only contains one event that matches your query, the endpoint removes all 65,536 events in the block. For this reason, using this endpoint may remove more data than your query specifies. |
There was a problem hiding this comment.
all 65,536 events
there could be anywhere from 1 to a few million events in a block
mhr3
reviewed
Dec 17, 2024
| <Warning> | ||
| The `/v2/datasets/_apl/delete` endpoint deletes all blocks containing events that match your query. For example, even if a block only contains one event that matches your query, the endpoint removes all 65,536 events in the block. For this reason, using this endpoint may remove more data than your query specifies. | ||
|
|
||
| Deleting data using this endpoint is slow and expensive. Only use this endpoint when you have sent sensitive data to Axiom by mistake and you need to immediately delete it for regulatory reasons. |
There was a problem hiding this comment.
for regulatory reasons
do we need to specify reason they have for deletion?
mhr3
reviewed
Dec 17, 2024
|
|
||
| Deleting data using this endpoint is slow and expensive. Only use this endpoint when you have sent sensitive data to Axiom by mistake and you need to immediately delete it for regulatory reasons. | ||
|
|
||
| Use the block deletion endpoint for deleting less than one million rows in one request. To delete more than one million rows, send a number of requests and delete a limited number of rows in each request. If you delete too many rows in one request, the request might fail unexpectedly. |
There was a problem hiding this comment.
might fail unexpectedly.
If we're telling them about it, is it unexpected?
mhr3
reviewed
Dec 17, 2024
|
|
||
| ## Best practices | ||
|
|
||
| - Start with a dry run. Set `commit` to `false` to safely review which events are to be deleted by your request. |
There was a problem hiding this comment.
Suggested change
| - Start with a dry run. Set `commit` to `false` to safely review which events are to be deleted by your request. | |
| - Start with a dry run. Set `commit` to `false` to safely review the number of events to be deleted by your request. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Preview: https://axiom-mano-block-deletion.mintlify.app/restapi/delete-blocks