Restructure Docs #27235

TohnJhomas · 2022-10-24T19:16:10Z

This PR is designed to restructure the overcrowded and confusing sidebar of the Astronomer Documentation. This is intended to allow for further scalability of the airflow docs, and to increase user-friendliness to new users.

The current docs have 28 top-level pages and two sub-sections of substance: "How-to Guides" and "Concepts." Here I've attempted to clean up the sidebar and make the structure more clear by both splitting "Concepts" into three new top-level pages ("Core Concepts," "Authoring and Scheduling" and "Administration and Deployment") and folding many of the current top-level pages into these categories.

The structure of the docs as they currently are is visually here: The left column is representative of the top-level sidebar, with the concepts subsection broken out to the right. How-to guides is another large section, but because it didn't undergo any substantial change I've hidden the sub-pages for ease of understanding.

The structure of the refactored docs is below, with the same format as above:

boring-cyborg · 2022-10-24T19:16:14Z

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
Consider using Breeze environment for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
Be sure to read the Airflow Coding style.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: [email protected]
Slack: https://s.apache.org/airflow-slack

TohnJhomas · 2022-10-24T19:16:50Z

This version has known issues with cross-provider label referencing

uranusjr · 2022-10-25T08:35:03Z

Thanks for taking the time to do this! The new structure makes much more sense to me.

dstandish · 2022-10-25T17:31:42Z

Thanks for taking the time to do this! The new structure makes much more sense to me.

I second this emotion. Thanks

potiuk · 2022-10-26T01:29:00Z

Conflicts to resolve I am avraid :( . Emotions are the same :)

potiuk · 2022-10-26T01:29:29Z

Would be great also if you generate the docs breeze build-docs and share some screenshots.

BasPH · 2022-10-26T08:17:48Z

Looking forward to this much-needed change! 👍

potiuk · 2022-10-31T04:09:18Z

Looking forward to this much-needed change! 👍

Likewise.

mik-laj · 2022-11-03T01:32:05Z

Can you also update the redirects.txt file? It is essential for Google to be able to find the new page locations and also for users to be properly redirected when they change Apache Airflow versions in the documentation.

TohnJhomas · 2022-11-03T20:37:58Z

Yeah - just pointing all the old locations at their new locations?

mik-laj · 2022-11-04T11:45:03Z

@TohnJhomas Yes. Exactly.

dstandish · 2022-11-05T01:52:30Z

has conflicts

BasPH · 2022-12-14T12:41:51Z

@TohnJhomas I'd love to see this PR merged, any chance you could resolve the conflicts?

potiuk · 2022-12-14T16:11:59Z

@TohnJhomas I'd love to see this PR merged, any chance you could resolve the conflicts?

Oh yeah. I would very, very much love that too. Maybe you need some help @TohnJhomas ? I am happy to spend some time on helping to solve issue with docs building (I know it is a very daunting thing to fight with Sphinx. The way the tags etc. are defined are rather non-intuitive and there are many, many traps you can fall in.

TohnJhomas · 2022-12-15T19:19:02Z

@potiuk / @BasPH Honestly - I got a little stuck working on some of the sphynx roles. I'm getting redirects.txt and the merge conflicts fixed today and everything pushed up to date. a few broken links to/from KubernetesPodOperator probably shouldn't hold everything else up

potiuk · 2022-12-16T22:22:41Z

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

Let me merge it quickly before there are any conflicts

🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉 🎉

boring-cyborg · 2022-12-16T22:24:36Z

Awesome work, congrats on your first merged pull request!

A big restructuring of our docs to make them easier to reason about - in a much more logical and consistent way. Likel other updates and content changes will be needed to make it even better.

A big restructuring of our docs to make them easier to reason about - in a much more logical and consistent way. Likel other updates and content changes will be needed to make it even better. (cherry picked from commit 401fc57)

Add a broad-strokes description of the security expectations operator should expect. This will get included into https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/index.html I'm no Airflow expert so help definitely welcome. I would like this section to be a bit more prominent rather than hidden away under 'Administration and Deployment', but it looks like the structure was carefully considered in apache#27235 so this is probably fine. Eventually we could point the 'Security' link on the main pages like https://airflow.apache.org/ to this page.

Add a broad-strokes description of the security expectations operator should expect. This will get included into https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/index.html I'm no Airflow expert so help definitely welcome. I would like this section to be a bit more prominent rather than hidden away under 'Administration and Deployment', but it looks like the structure was carefully considered in #27235 so this is probably fine. Eventually we could point the 'Security' link on the main pages like https://airflow.apache.org/ to this page.

Add a broad-strokes description of the security expectations operator should expect. This will get included into https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/index.html I'm no Airflow expert so help definitely welcome. I would like this section to be a bit more prominent rather than hidden away under 'Administration and Deployment', but it looks like the structure was carefully considered in #27235 so this is probably fine. Eventually we could point the 'Security' link on the main pages like https://airflow.apache.org/ to this page. (cherry picked from commit f20c08a)

Add a broad-strokes description of the security expectations operator should expect. This will get included into https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/index.html I'm no Airflow expert so help definitely welcome. I would like this section to be a bit more prominent rather than hidden away under 'Administration and Deployment', but it looks like the structure was carefully considered in apache/airflow#27235 so this is probably fine. Eventually we could point the 'Security' link on the main pages like https://airflow.apache.org/ to this page. GitOrigin-RevId: f20c08a77c9daefe54c14c5013fcf53d84b5744d

Add a broad-strokes description of the security expectations operator should expect. This will get included into https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/index.html I'm no Airflow expert so help definitely welcome. I would like this section to be a bit more prominent rather than hidden away under 'Administration and Deployment', but it looks like the structure was carefully considered in apache/airflow#27235 so this is probably fine. Eventually we could point the 'Security' link on the main pages like https://airflow.apache.org/ to this page. (cherry picked from commit f20c08a77c9daefe54c14c5013fcf53d84b5744d) GitOrigin-RevId: bb86b468ae9396ece41f105db6104ee9282b4ed1

Add a broad-strokes description of the security expectations operator should expect. This will get included into https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/security/index.html I'm no Airflow expert so help definitely welcome. I would like this section to be a bit more prominent rather than hidden away under 'Administration and Deployment', but it looks like the structure was carefully considered in apache/airflow#27235 so this is probably fine. Eventually we could point the 'Security' link on the main pages like https://airflow.apache.org/ to this page. GitOrigin-RevId: f20c08a77c9daefe54c14c5013fcf53d84b5744d

Restructure Docs- known errors in label crosslinks

edf3256

TohnJhomas requested review from jedcunningham, dstandish, uranusjr, kaxil and potiuk as code owners October 24, 2022 19:16

boring-cyborg bot added area:core-operators Operators, Sensors and hooks within Core Airflow provider:cncf-kubernetes Kubernetes provider related issues area:providers kind:documentation labels Oct 24, 2022

TohnJhomas marked this pull request as draft October 24, 2022 19:17

jedcunningham closed this Oct 25, 2022

jedcunningham reopened this Oct 25, 2022

potiuk mentioned this pull request Nov 8, 2022

Remove taskflow code from Airflow intro example #26486

Closed

Spelling and link fixes

7f6c646

fix boring-cyborg links

66d622a

potiuk approved these changes Dec 16, 2022

View reviewed changes

fixup! fix boring-cyborg links

13c7d7c

potiuk merged commit 401fc57 into apache:main Dec 16, 2022

potiuk mentioned this pull request Dec 16, 2022

Add Public Interface description to Airflow documentation #28300

Merged

eladkal mentioned this pull request Jan 2, 2023

Status of testing Providers that were prepared on January 02, 2023 #28679

Closed

84 tasks

pierrejeambrun added this to the Airflow 2.5.1 milestone Jan 9, 2023

pierrejeambrun added the type:doc-only Changelog: Doc Only label Jan 9, 2023

ephraimbuddy modified the milestones: Airflow 2.5.1, Airflow 2.6.0 Jan 11, 2023

Taragolis mentioned this pull request Jan 11, 2023

Wrong GitHub Link provided in Documentation #28866

Closed

2 tasks

ephraimbuddy modified the milestones: Airflow 2.6.0, Airflow 2.5.1 Jan 12, 2023

ephraimbuddy added a commit that referenced this pull request Jan 12, 2023

fixup! Restructure Docs (#27235)

743c237

pierrejeambrun mentioned this pull request Jan 13, 2023

Sync v2-5-stable with v2-5-test to release 2.5.1 #28883

Merged

raboof mentioned this pull request Apr 24, 2023

Add broad-strokes 'security model' #30843

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Restructure Docs #27235

Restructure Docs #27235

TohnJhomas commented Oct 24, 2022

boring-cyborg bot commented Oct 24, 2022

TohnJhomas commented Oct 24, 2022

uranusjr commented Oct 25, 2022

dstandish commented Oct 25, 2022

potiuk commented Oct 26, 2022

potiuk commented Oct 26, 2022

BasPH commented Oct 26, 2022

potiuk commented Oct 31, 2022

mik-laj commented Nov 3, 2022

TohnJhomas commented Nov 3, 2022

mik-laj commented Nov 4, 2022

dstandish commented Nov 5, 2022

BasPH commented Dec 14, 2022

potiuk commented Dec 14, 2022

TohnJhomas commented Dec 15, 2022

potiuk commented Dec 16, 2022

boring-cyborg bot commented Dec 16, 2022

Restructure Docs #27235

Restructure Docs #27235

Conversation

TohnJhomas commented Oct 24, 2022

boring-cyborg bot commented Oct 24, 2022

TohnJhomas commented Oct 24, 2022

uranusjr commented Oct 25, 2022

dstandish commented Oct 25, 2022

potiuk commented Oct 26, 2022

potiuk commented Oct 26, 2022

BasPH commented Oct 26, 2022

potiuk commented Oct 31, 2022

mik-laj commented Nov 3, 2022

TohnJhomas commented Nov 3, 2022

mik-laj commented Nov 4, 2022

dstandish commented Nov 5, 2022

BasPH commented Dec 14, 2022

potiuk commented Dec 14, 2022

TohnJhomas commented Dec 15, 2022

potiuk commented Dec 16, 2022

boring-cyborg bot commented Dec 16, 2022