Determine how to improve DataCatalog documentation page: review/propose ticket

[stichbury]

This ticket picks up where #1742 left off. We need to do a similar piece of work to that done to the Config docs to split it into separate pages for different audience types/goals and structure it better for people needing to find "How to" information.

It may be sensible to move examples out of the narrative docs and back into the API docs too, as suggested by @noklam in #1742.

The first step of this issue is to review what is there, then propose a set of revisions. There can be a new issue to implement those changes

[stichbury]

Here are some ideas for improving

• Data Catalog basics
  • Configuration and the data catalog
  • Dataset locations
  • Parameters with basic examples 1-3
  • Error handling (improve what is on Kedro IO page)
• Basic Data Catalog How-tos
  • How to create a Data Catalog YAML configuration file via CLI
  • ... intervening sections down to
  • How to do transcoding
  • How to version datasets from Data Catalog page and Kedro IO page
  • How to use the Data Catalog with the API
• Example usage of the YAML API
  • Examples 1-17
• Advanced Data Catalog How-to guides
  • How to work with partitioned datasets
  • How to create a custom dataset <-- may consider whether to move from "Extend Kedro" into this section, or otherwise possibly just introduces AbstractDataSet and some of the key steps, then points through to the details

[stichbury]

@merelcht @noklam Here ⬆️ is a rough suggestion of a set of pages to replace the two pages currently in the "Data Catalog" section of the docs (the DataCatalog one is one of the most popular pages).

The suggestion is to break into four pages:

• Data Catalog basics
• How to guides for basic/standard activities
• Examples (considering whether this should go into API docs somewhere, but otherwise keep in a separate page)
• How to guides for more advanced usage

What do you think?

• Should we add more sections?
• Have I split stuff out of the Kedro IO page into the right places?
• What about writing a custom dataset -- should that come into this section?

The language is also quite dense/confusing in places so the overhaul would also do a set of improvements/additions to make the pages more readable, but this is more about getting your views on re-sectioning it.

Happy to discuss in person if it's easier.

[merelcht]

Wow looking at the existing pages is actually quite painful. What a mess 🤯

My thoughts on the breakdown:

• Data Catalog basics
  • Configuration and the data catalog
  • Dataset locations
  • Parameters with basic examples 1-3
  • Error handling (improve what is on Kedro IO page)

That error handling page right now is useless.. We'll have to rethink what information we'd actually like to give here.

I think in general we need to make a very clear separation between using the YAML API and the Code API. I would be inclined to move anything Code API to the advanced section. I'm thinking it might be good to add a section on "how to use the catalog" where we describe the YAML API and link to the examples.

• Basic Data Catalog How-tos
  • How to create a Data Catalog YAML configuration file via CLI
  • ... intervening sections down to
  • How to do transcoding
  • How to version datasets from Data Catalog page and Kedro IO page
  • How to use the Data Catalog with the API

If it's possible to clearly separate all Code API vs YAML API sections (it might be hard because we've mixed these a lot in the catalog docs..) then I'd probably move this to advanced use. The main reason for this is that I've more frequently seen beginner users skip the regular Kedro flow and immediately try to use components directly in code and they end up being confused about what to do. If we make it clear that the YAML API is the way to go this might help users understand better how to get started with Kedro and only start using the code APIs when they have a clear need.

• Example usage of the YAML API
  • Examples 1-17
• Advanced Data Catalog How-to guides
  • How to work with partitioned datasets
  • How to create a custom dataset <-- may consider whether to move from "Extend Kedro" into this section, or otherwise possibly just introduces AbstractDataSet and some of the key steps, then points through to the details

To me it makes sense to move the custom dataset section here and then also add the info about versioning a custom dataset to here instead of the general section on versioning.

[noklam]

Thank you for the proposal.

Here are some ideas for improving

• Data Catalog basics

  • Configuration and the data catalog
  • Dataset locations
  • Parameters with basic examples 1-3
  • Error handling (improve what is on Kedro IO page)

The error handling section is quite useless indeed, I think we can just remove it. I am fine with the structure, but I would comment [Configuration and the data catalog] isn't very intuitive, it talks a lot about different paths and is confusing. I know Kedro well enough so I understand it, but maybe we need some example explaining how that config merging helps. Or just link it to a more detailed example later.

• Basic Data Catalog How-tos

  • How to create a Data Catalog YAML configuration file via CLI

I don't think this is the first thing users need to learn and would defer it to advance examples.

• ... intervening sections down to
• How to do transcoding
  I think we can move this down a bit or to the advance section, it's topic that unlikely a beginner need until they work with spark.

• How to version datasets from Data Catalog page and Kedro IO page

• How to use the Data Catalog with the API

• Example usage of the YAML API

  • Examples 1-17

• Advanced Data Catalog How-to guides

  • How to work with partitioned datasets
  • How to create a custom dataset <-- may consider whether to move from "Extend Kedro" into this section, or otherwise possibly just introduces AbstractDataSet and some of the key steps, then points through to the details

[stichbury]

Thank you @merelcht and @noklam

With your help, I have this as a possible structure now:

• Data Catalog basics

  • How to use the Data Catalog: YAML API
  • Configuration and the data catalog (isn't very intuitive, it talks a lot about different paths and is confusing so maybe we need some example explaining how that config merging helps. Or just link it to a more detailed example later.)
  • Dataset locations
  • Parameters with basic examples 1-3
  • Error handling (improve what is on Kedro IO page) That error handling page right now is useless.. We'll have to rethink what information we'd actually like to give here. Just remove it.

• Basic Data Catalog How-tos ordered by usefulness, and moving anything using the Code API into the Advanced Usage section on a separate page. To include sections (but not in this order) from

  • How to create a Data Catalog YAML configuration file via CLI
  • ... intervening sections down to transcoding but not including it as that’s advanced

• Example usage of the YAML API

  • Examples 1-17

• Advanced Data Catalog How-to guides

  • Code API (explain to use it only when you’ve a clear need for it) from How to use the Data Catalog with the API and anything from DataCatalog page that is specific to Code API
  • How to do transcoding
  • How to work with partitioned datasets
  • How to create a custom dataset <-- Move from "Extend Kedro" into this section and add the info about versioning a custom dataset to here instead of the general section on versioning. Repurpose “How to version datasets” from Data Catalog page and Kedro IO page

[stichbury]

This completes this ticket as a review of the docs to propose a restructure. I'll make a separate ticket to execute the changes and will make a draft PR with some of the revisions so that they can be reviewed docs accuracy/correctness improved by the engineering team.