[feature] Use X-Robots-Tag headers to instruct scrapers/crawlers

[tsmethurst]

Description

If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.

If this is a documentation change, please briefly describe what you've changed and why.

This pull request updates our middleware stack and router initialization to inject appropriate X-Robots-Tag HTTP headers in router groups.

This includes specifying different X-Robots-Tag headers for .well-known/nodeinfo and /nodeinfo/2.0|2.1 endpoints depending on instance-stats-mode: if instance-stats-mode is serve then X-Robots-Tag will permit indexing on those endpoints, else noindex, nofollow will be used.

With instance-stats-mode = serve:

Otherwise:

Regardless of instance-stats-mode setting, noai, noimageai tags will be transmitted everywhere, as before.

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

• I/we have read the GoToSocial contribution guidelines.
• I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
• I/we have not leveraged AI to create the proposed changes.
• I/we have performed a self-review of added code.
• I/we have written code that is legible and maintainable by others.
• I/we have commented the added code, particularly in hard-to-understand areas.
• I/we have made any necessary changes to documentation.
• I/we have added tests that cover new code.
• I/we have run tests and they pass locally with the changes.
• I/we have run go fmt ./... and golangci-lint run.