Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add option to generate a sitemap #59

Merged
merged 2 commits into from
Aug 29, 2023
Merged

Add option to generate a sitemap #59

merged 2 commits into from
Aug 29, 2023

Conversation

mortenpi
Copy link
Member

Just wanted to open a draft PR here with this naive implementation, which takes all the HTML files and generates the sitemap with all of them.

However, I realized that this way we'll end up with all the old versions also in the sitemap, and I don't this that's a good idea, since sitemaps should also point to canonical content. So we need to be a bit smarter about it.

We could try to interpret the file system paths, but I figured a better way is to re-use canonical URLs, which we can parse from the HTML. The sitemap will be just the canonical URLs, but we'll also make sure that the canonical URL is reasonable. HTML files with missing or broken canonical URLs get ignored.

This pushes the responsibility of having good canonical URLs to the HTML generation. So I think we want a post-processing step here as well for each subpage, to catch and/or fix bad or missing canonical URLs.

@mortenpi mortenpi changed the title WIP: add option to generate a sitemap Add option to generate a sitemap Aug 28, 2023
@mortenpi mortenpi marked this pull request as ready for review August 28, 2023 02:50
@mortenpi
Copy link
Member Author

A few changes

  • canonical -> canonical_domain, since you should only have the domain part of the URL there. The root portion of the path comes from rootpath.
  • Adds sitemap::Bool and sitemap_filename (optional), to toggle the sitemap generation. Since the implementation is based on canonical URLs now, you need to set canonical_domain (and therefore enable canonical URL rewriting) when you set sitemap=true.
  • I reorganized the runtests.jl a bit, moving the MultiDocumenter run out of the testset, so that it would be easier to rapidly re-run the test suite (when developing).
  • It adds a couple of tests for canonical URLs as well.

@mortenpi mortenpi requested a review from pfitzseb August 28, 2023 02:54
src/MultiDocumenter.jl Outdated Show resolved Hide resolved
src/sitemap.jl Outdated Show resolved Hide resolved
src/sitemap.jl Show resolved Hide resolved
src/sitemap.jl Outdated Show resolved Hide resolved
@mortenpi mortenpi force-pushed the mp/sitemap branch 2 times, most recently from 58f3a34 to 83c1f59 Compare August 29, 2023 08:55
@mortenpi mortenpi requested a review from pfitzseb August 29, 2023 08:56
@mortenpi mortenpi merged commit 4babdb8 into main Aug 29, 2023
@delete-merged-branch delete-merged-branch bot deleted the mp/sitemap branch August 29, 2023 10:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants