Skip to content

Commit

Permalink
Finalized all the text
Browse files Browse the repository at this point in the history
  • Loading branch information
@abey79
abey79 committed Oct 14, 2024
1 parent 3a51fc6 commit affd8f8
Show file tree
Hide file tree
Showing 7 changed files with 133 additions and 223 deletions.
6 changes: 4 additions & 2 deletions docs/content/getting-started/data-out.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,14 @@ title: Get data out of Rerun
order: 700
---

At its core, Rerun is a database. The Rerun SDK includes an API to export data as dataframes from recording. This can be used, for example, to perform analysis on the data and even log back the results to the original recording.
At its core, Rerun is a database. The viewer includes the [dataframe view](../reference/types/views/dataframe_view) to explore data in tabular form, and the SDK includes an API to export data as dataframes from recording. These features can be used, for example, to perform analysis on the data and log back the results to the original recording.

In this series of articles, we explore this workflow by implementing an "open jaw detector" on top of our [face tracking example](https://rerun.io/examples/video-image/face_tracking). This process is split into three steps:
In this three-part guide, we explore such a workflow by implementing an "open jaw detector" on top of our [face tracking example](https://rerun.io/examples/video-image/face_tracking). This process is split into three steps:

1. [Explore a recording with the dataframe view](data-out/explore-as-dataframe)
2. [Export the dataframe](data-out/export-dataframe)
3. [Analyze the data and log the results](data-out/analyze-and-log)

Note: this guide uses [Pandas](https://pandas.pydata.org) dataframes because of how popular this package is. The same concept however applies in the same way for alternative dataframe packages such as [Polars](https://pola.rs).

For the fast track, jump to the [complete script](data-out/analyze-and-log.md#complete-script) at the end of the third section.
28 changes: 19 additions & 9 deletions docs/content/getting-started/data-out/analyze-and-log.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,15 +5,15 @@ order: 3



In the previous sections, we explored our data and exported it into a Pandas dataframe. In this section, we will analyze the data to extract a "jaw open state" signal and log it back to the viewer.
In the previous sections, we explored our data and exported it to a Pandas dataframe. In this section, we will analyze the data to extract a "jaw open state" signal and log it back to the viewer.



## Analyze the data

Well, this is not the most complicated part, as we already identified that thresholding the `jawOpen` signal at 0.15 is all we need. Recall that we already flattened that signal into a `"jawOpen"` dataframe column in the [previous section](export-dataframe.md#inspect-the-dataframe)
This is admittedly not the most complicated part of this guide. We already identified that thresholding the `jawOpen` signal at 0.15 is all we need to produce a binary "jaw open state" signal.

Let's add a boolean column to our Pandas dataframe to hold our jaw open state:
In the [previous section](export-dataframe.md#inspect-the-dataframe), we prepared a flat, floating point column with the signal of interest called `"jawOpen"`. Let's add a boolean column to our Pandas dataframe to hold our jaw open state:

```python
df["jawOpenState"] = df["jawOpen"] > 0.15
Expand All @@ -22,16 +22,19 @@ df["jawOpenState"] = df["jawOpen"] > 0.15

## Log the result back to the viewer

The first step to log the data is to initialize the logging such that the data we log is routed to the exact same recording that we just analyzed. For this, both the application ID and the recording ID must match. Here is how it is done:
The first step to log the data is to initialize the logging SDK to direct it to the exact same recording that we just analyzed. For this, both the application ID and the recording ID must match. Here is how it is done:

```python
rr.init(recording.application_id(), recording_id=recording.recording_id())
rr.init(
recording.application_id()
recording_id=recording.recording_id(),
)
rr.connect()
```

Note: When automating data analysis, you should typically log the results to an RRD file distinct from the source RRD (using `rr.save()`). It is also valid to use the same app ID and recording ID in such a case. In particular, this allows opening both the source and result RRDs in the viewer, which will display both data under the same recording.
_Note_: When automating data analysis, it is typically preferable to log the results to an distinct RRD file next to the source RRD (using `rr.save()`). In such case, it is also valid to use the same app ID and recording ID. This allows opening both the source and result RRDs in the viewer, which will display both data under the same recording.

Let's log our jaw open state data in two forms:
We will log our jaw open state data in two forms:
1. As a standalone `Scalar` component, to hold the raw data.
2. As a `Text` component on the existing bounding box entity, such that we obtain a textual representation of the state in the visualization.

Expand Down Expand Up @@ -63,14 +66,21 @@ rr.send_columns(
)
```

Here we first log the [`ShowLabel`](../../reference/types/components/show_labels.md) component as static to enable the display of the label. Then, we use `rr.send_column()` again to send an entire batch of text labels. We use the [`np.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html) to produce a label that matches the state for each timestamp.
Here we first log the [`ShowLabel`](../../reference/types/components/show_labels.md) component as static to enable the display of the label. Then, we use `rr.send_column()` again to send an entire batch of text labels. We use the [`np.where()`](https://numpy.org/doc/stable/reference/generated/numpy.where.html) to produce a label matching the state for each timestamp.

### Final result

TODO: screen shot
With some adjustments to the viewer blueprint, we obtain the following result:

<video width="100%" autoplay loop muted controls>
<source src="https://static.rerun.io/getting-started-data-out/data-out-final.webm" type="video/webm" />
</video>

The OPEN/CLOSE label is displayed along the bounding box on the 2D view, and the `/jaw_open_state` signal is visible in both the timeseries and dataframe views.


### Complete script

Here is the complete script used by this guide to load data, analyze it, and log the result back:

snippet: tutorials/data_out
45 changes: 19 additions & 26 deletions docs/content/getting-started/data-out/explore-as-dataframe.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,20 +3,10 @@ title: Explore a recording with the dataframe view
order: 1
---

**OVERVIEW**
- introduce RRD for this series (-> face tracking)
- create a dataframe view with all the data
- explain:
- column meaning
- do not explain:
- latest at & pov -> link to something else
- "[Next](export-dataframe): export the dataframe to Pandas"


<hr>


For this series of guides, we use the [face tracking example](https://rerun.io/examples/video-image/face_tracking) to explore the Rerun viewer's dataframe view and the Rerun SDK's dataframe API. Our goal is to implement a "jaw open" detector in Python and log its result back to the viewer.
In this first part of the guide, we run the [face tracking example](https://rerun.io/examples/video-image/face_tracking) and explore the data in the viewer.

## Create a recording

Expand All @@ -33,9 +23,9 @@ A person's face is visible and being tracked. Their jaws occasionally open and c

## Explore the data

The [MediaPipe Face Landmark](https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker) package used by the face tracking example outputs, amongst other things, so-called blendshapes signals, which provide infomation about various aspects of the face expression. These signals are logged under the `/blendshapes` root entity by the face tracking example.
Amongst other things, the [MediaPipe Face Landmark](https://ai.google.dev/edge/mediapipe/solutions/vision/face_landmarker) package used by the face tracking example outputs so-called blendshapes signals, which provide information on various aspects of the face expression. These signals are logged under the `/blendshapes` root entity by the face tracking example.

One signal, `jawOpen` (logged under the `/blendshapes/0/jawOpen` entity as a [`Scalar`](../../reference/types/components/scalar.md)) component), is of particular interest for our purpose. Let's inspect it further using a timeseries view:
One signal, `jawOpen` (logged under the `/blendshapes/0/jawOpen` entity as a [`Scalar`](../../reference/types/components/scalar.md) component), is of particular interest for our purpose. Let's inspect it further using a timeseries view:


<picture>
Expand All @@ -46,34 +36,37 @@ One signal, `jawOpen` (logged under the `/blendshapes/0/jawOpen` entity as a [`S
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/data-out-jaw-open-signal/258f5ffe043b8affcc54d5ea1bc864efe7403f2c/1200w.png">
</picture>

This signal indeed seems to jump from approx. 0.0 to approx. 0.5 whenever the jaws are open. We can also notice a discontinuity in the middle of the recording. This is due to the blendshapes being [`Clear`](../../reference/types/archetypes/clear.md)ed when no face is being detected.
This signal indeed seems to jump from approximately 0.0 to 0.5 whenever the jaws are open. We also notice a discontinuity in the middle of the recording. This is due to the blendshapes being [`Clear`](../../reference/types/archetypes/clear.md)ed when no face is detected.

Let's create a dataframe view to further inspect the data:

<picture>
<img src="https://static.rerun.io/data-out-jaw-open-dataframe/52c4f78e8b462365e65ca397a37ee737543de62c/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/52c4f78e8b462365e65ca397a37ee737543de62c/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/52c4f78e8b462365e65ca397a37ee737543de62c/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/52c4f78e8b462365e65ca397a37ee737543de62c/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/52c4f78e8b462365e65ca397a37ee737543de62c/1200w.png">
<img src="https://static.rerun.io/data-out-jaw-open-dataframe/bde18eb7b159e3ea1166a61e4a334eaedf2e04f8/full.png" alt="">
<source media="(max-width: 480px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/bde18eb7b159e3ea1166a61e4a334eaedf2e04f8/480w.png">
<source media="(max-width: 768px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/bde18eb7b159e3ea1166a61e4a334eaedf2e04f8/768w.png">
<source media="(max-width: 1024px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/bde18eb7b159e3ea1166a61e4a334eaedf2e04f8/1024w.png">
<source media="(max-width: 1200px)" srcset="https://static.rerun.io/data-out-jaw-open-dataframe/bde18eb7b159e3ea1166a61e4a334eaedf2e04f8/1200w.png">
</picture>

Here is how this view is configured:
- Its content is set to `/blendshapes/0/jawOpen`. As a result, the table only contains columns pertaining to that entity (along with any timeline(s)). For this entity, a single column exists in the table, corresponding to entity's single component (of `Scalar` type).
- The `frame_nr` timeline is used as index for the table. This means that the table will contain one row for each distinct value of `frame_nr` where data was logged.
- Its content is set to `/blendshapes/0/jawOpen`. As a result, the table only contains columns pertaining to that entity (along with any timeline(s)). For this entity, a single column exists in the table, corresponding to entity's single component (a `Scalar`).
- The `frame_nr` timeline is used as index for the table. This means that the table will contain one row for each distinct value of `frame_nr` for which data is available.
- The rows can further be filtered by time range. In this case, we keep the default "infinite" boundaries, so no filtering is applied.
- The dataframe view has other advanced features which we are not using here, including filtering rows based on the existence of data for a given column, or filling empty cells with latest-at data.

The dataframe view has other advanced features which we are not using here, including filtering rows based on the existence of data for a given column or filling empty cells with latest-at data. You can read more about these here (TODO: ADD LINK).
<!-- TODO(#7499): add link to more information on filter-is-not-null and fill with latest-at -->

Now, let's look at the actual data as represented in the above screenshot. At around frame #140, the jaws are open, and, accordingly, the `jawOpen` signal has values around 0.55. Shortly after, they close again and the signal decreases to below 0.1. Then, the signal becomes empty. This happens in rows corresponding to the period of time when the face cannot be tracked and all the signals are cleared.


## Next steps

Our exploration of the data in the viewer so far provided us with the information we require to implement the jaw open detector in two important ways.
Our exploration of the data in the viewer so far provided us with two important pieces of information useful to implement the jaw open detector.

First, we identified that the `Scalar` value contained in `/blendshapes/0/jawOpen` contains relevant data. In particular, thresholding this signal with a value of 0.15 should provide us with a closed/opened jaw state binary indicator.

First, we identified that the `Scalar` value contained in `/blendshapes/0/jawOpen` contains the information we require. In particular, thresholding this signal with a value of 0.15 should provide us with a closed/opened jaw state binary indicator.
Then, we explored the numerical data in a dataframe view. Importantly, the way we configured this view for our needs informs us on how to query the recording from code such as to obtain the correct output.

Then, we explored the numerical data in a dataframe view. Importantly, the way we configured this view for such that it displays the data of interest informs us on how we should query the recording to extract that data.
<!-- TODO(#7462): improve the previous paragraph to mention copy-as-code instead -->

From there, our next step is to query the recording and extract the data as a Pandas dataframe in Python, such that it can then be analyzed. This is covered in the [next section](export-dataframe.md) of this guide.
From there, our next step is to query the recording and extract the data as a Pandas dataframe in Python. This is covered in the [next section](export-dataframe.md) of this guide.
Loading

0 comments on commit affd8f8

Please sign in to comment.