Sitemap improvements

[robert-bryson]

User Story

In order to [improve catalog SEO], [datagov team] wants [fix/improve the current sitemap].

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN [visiting catalog sitemap]
  WHEN [the page load] happens
  THEN [a html page should be loaded]
  [AND the XML file not downloaded but with a download link]

• GIVEN [checking Google Search Console]
  WHEN [checking sitemap Discovered URLs] happens
  THEN [the number is growing]

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Based on the meeting with Freddie Blicher and the datagov team on 11/14, we need to take a couple of actions to improve the effectiveness of our sitemap for catalog.

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

[Notes or a checklist reflecting our understanding of the selected approach]

• A good example of a sitemap is at https://moz.com/sitemaps-1-sitemap.xml
• Review https://www.sitemaps.org/protocol.html#index and make sure our sitemap generation/link conforms to the correct standard
• Freddie should send a copy of a partial crawl he did with Screaming Frog to take a look at as an example

[hkdctol]

Would be good to pair to spread background on this issue.

[nickumia-reisys]

So, with the current sitemap design, I don't think is possible. Right now, we are hosting sitemap files in an S3 bucket. Since the bucket is not configured as a static site domain, I don't think it will be possible to use S3 as a single solution hosting strategy. We could build a sitemap template into ckan catalog, probably ckanext-datagovtheme that looks for the S3 files and then dynamically generates these pages on load. I'm not sure how time-consuming it will be, or if this added complexity is worth it, but I suppose I don't have another option for implementing this. @robert-bryson @jbrown-xentity @FuhuXia @btylerburton @Jin-Sun-tts thoughts?

[nickumia-reisys]

**Time-consuming in terms of: (1) implementation and (2) actual user load time.

[robert-bryson]

After experimenting with explicitly setting the Content-Type in nginx yesterday, I have been looking more into how s3 behaves with the cli. I uploaded the same sitemap.xml file with aws s3 cp.. with and without setting the type and both correctly return application/xml:

[robert-bryson]

With GSA/catalog.data.gov#702, the sitemaps should be getting created with the correct content-type and should be able to be consumed by Google. Once the PR is merged, a new sitemap job can be dispatched (or one will run overnight), and then Google Search Console should start showing numbers go up.

There are still a number of changes to the sitemaps we can and should make, but hopefully this will enable Google to start picking up URLs.

[robert-bryson]

Well, with GSA/catalog.data.gov#703:

[robert-bryson]

Sitemap urls are being generated correctly, and site map files are being correctly generated but the connection between the two is not working. It should be a simple ngnix fix (and probably is), but I haven't gotten it to work correctly yet. It's possible that there is an s3 config that needs to be changed as well.

[robert-bryson]

It's working! It's working!

(The 'Couldn't Fetch' ones are slowing flipping to 'Success' as the scan runs)

[robert-bryson]

After letting Google do its thing for the weekend, it has made some limited forward progress.

sitemap.xml remains at 5k URLs, but there is now a chart of index coverage, but it has not grabbed the rest of the sitemap files (showing 5 files successfully read, 76 as couldn't fetch, and the other ~300 not present in the UI for an unknown reason).

I submitted sitemap-20 by hand last week to check as an example of a missing file from above. Initially it was also showing 403, but has now flipped to success with it's 1k URLs, though no chart of index coverage.

I'm not sure what to conclude or which actions to take. It does seem like Google will slowly crawl and index these URLs, but (from what I can tell) on the limited set of 5 or 81 sitemap files. Will continue to see if there is a way to account for the missing (on the ui at least) sitemap files and overcome Google's alleged 403s.

[robert-bryson]

As a bump: Search Console is now showing 17k URLs and (perhaps importantly?) 86 sitemap files found in the index (as opposed to the 81 it has been finding, but still missing the total of 374).

Since it does look like Google is slowly making its way through the files, perhaps we should close this for now and make a 'Check back in a month'-type ticket? We do have a lot of URLs after all.

[hkdctol]

We will make a follow up ticket, but close this one as complete.