Skip to content

Commit

Permalink
Remove sqlite3-specific info from main README, make it more storage a…
Browse files Browse the repository at this point in the history
…gnostic and point to plugin-specific README (ros2#1193)

Signed-off-by: Emerson Knapp <[email protected]>
  • Loading branch information
emersonknapp authored Dec 2, 2022
1 parent 1accc18 commit 1ce1e01
Show file tree
Hide file tree
Showing 2 changed files with 46 additions and 37 deletions.
52 changes: 15 additions & 37 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -116,31 +116,12 @@ It is recommended to use this feature with the splitting options.
#### Recording with a storage configuration

Storage configuration can be specified in a YAML file passed through the `--storage-config-file` option.
This can be used to optimize performance for specific use-cases.
This can be used to optimize performance for specific use cases.

For the default storage plugin (sqlite3), the file has a following syntax:
```
read:
pragmas: <list of pragma settings for read-only>
write:
pragmas: <list of pragma settings for read/write>
```

By default, SQLite settings are significantly optimized for performance.
This might have consequences of bag data being corrupted after an application or system-level crash.
This consideration only applies to current bagfile in case bag splitting is on (through `--max-bag-*` parameters).
If increased crash-caused corruption resistance is necessary, use `resilient` option for `--storage-preset-profile` setting.

Settings are fully exposed to the user and should be applied with understanding.
Please refer to [documentation of pragmas](https://www.sqlite.org/pragma.html).

An example configuration file could look like this:
See storage plugin documentation for more detail:
* [sqlite3](rosbag2_storage_sqlite3/README.md#storage-configuration-file)
* [mcap](rosbag2_storage_mcap/README.md#writer-configuration)

```
write:
pragmas: ["journal_mode = MEMORY", "synchronous = OFF", "schema.cache_size = 1000", "schema.page_size = 4096"]
```

### Replaying data

Expand Down Expand Up @@ -207,7 +188,7 @@ This example notes all fields that can have an effect, with a comment on the req
```
output_bags:
- uri: /output/bag1 # required
storage_id: sqlite3 # required
storage_id: "" # will use the default storage plugin, if unspecified
max_bagfile_size: 0
max_bagfile_duration: 0
storage_preset_profile: ""
Expand All @@ -233,7 +214,6 @@ $ ros2 bag convert -i bag1 -i bag2 -o out.yaml
# out.yaml
output_bags:
- uri: merged_bag
storage_id: sqlite3
all: true
```

Expand All @@ -245,10 +225,8 @@ $ ros2 bag convert -i bag1 -o out.yaml
# out.yaml
output_bags:
- uri: split1
storage_id: sqlite3
topics: [/topic1, /topic2]
- uri: split2
storage_id: sqlite3
topics: [/topic3]
```

Expand All @@ -260,7 +238,6 @@ $ ros2 bag convert -i bag1 -o out.yaml
# out.yaml
output_bags:
- uri: compressed
storage_id: sqlite3
all: true
compression_mode: file
compression_format: zstd
Expand Down Expand Up @@ -340,20 +317,21 @@ $ ros2 launch record_all.launch.xml

## Storage format plugin architecture

Looking at the output of the `ros2 bag info` command, we can see a field called `storage id:`.
rosbag2 specifically was designed to support multiple storage formats.
This allows a flexible adaptation of various storage formats depending on individual use cases.
As of now, this repository comes with two storage plugins.
The first plugin, sqlite3 is chosen by default.
If not specified otherwise, rosbag2 will store and replay all recorded data in an SQLite3 database.
Looking at the output of the `ros2 bag info` command, we can see a field `Storage id:`.
Rosbag2 was designed to support multiple storage formats to adapt to individual use cases.
This repository provides two storage plugins, `mcap` and `sqlite3`.
The default is `sqlite3`, which is provided to code by [`rosbag2_storage::get_default_storage_id()`](rosbag2_storage/include/rosbag2_storage/default_storage_id.hpp) and defined in [`default_storage_id.cpp`](rosbag2_storage/src/rosbag2_storage/default_storage_id.cpp#L21)

In order to use a specified (non-default) storage format plugin, rosbag2 has a command line argument for it:
If not specified otherwise, rosbag2 will write data using the default plugin.

In order to use a specified (non-default) storage format plugin, rosbag2 has a command line argument `--storage`:

```
$ ros2 bag <record> | <play> | <info> -s <sqlite3> | <rosbag2_v2> | <custom_plugin>
$ ros2 bag record --storage <storage_id>
```

Have a look at each of the individual plugins for further information.
Bag reading commands can detect the storage plugin automatically, but if for any reason you want to force a specific plugin to read a bag, you can use the `--storage` option on any `ros2 bag` verb.


## Serialization format plugin architecture

Expand Down
31 changes: 31 additions & 0 deletions rosbag2_storage_sqlite3/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# rosbag2_storage_sqlite3

Storage implementation plugin for rosbag2 providing SQLite3 `.db3` files as the bag storage backend.


## Storage Configuration File

The `--storage-config-file` option for this plugin takes files in the following format:

```
read:
pragmas: <list of SQLite pragma settings for read-only>
write:
pragmas: <list of SQLite pragma settings for write modes>
```

By default, SQLite settings are significantly optimized for performance.
This might have consequences of bag data being corrupted after an application or system-level crash.
This consideration only applies to current bagfile in case bag splitting is on (through `--max-bag-*` parameters).
If increased crash-caused corruption resistance is necessary, use `resilient` option for `--storage-preset-profile` setting.

Settings are fully exposed to the user and should be applied with understanding.
Please refer to [documentation of pragmas](https://www.sqlite.org/pragma.html).

An example configuration file could look like this:

```
write:
pragmas: ["journal_mode = MEMORY", "synchronous = OFF", "schema.cache_size = 1000", "schema.page_size = 4096"]
```

0 comments on commit 1ce1e01

Please sign in to comment.