[chore-adj] disallow /nodeinfo/ explicitly too while we're at it

[wolfyjade]

Description

If this is a code change, please include a summary of what you've coded, and link to the issue(s) it closes/implements.

If this is a documentation change, please briefly describe what you've changed and why.

This pull request mainly disallows the /nodeinfo/ endpoint for scrapers (since they might already know where to look instead of asking the /.well-known endpoint) and those parsing the /robots.txt (basically complementing #3718).

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

• I/we have read the GoToSocial contribution guidelines.
• I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
• I/we have not leveraged AI to create the proposed changes.
• I/we have performed a self-review of added code.
• I/we have written code that is legible and maintainable by others.
• I/we have commented the added code, particularly in hard-to-understand areas.
• I/we have made any necessary changes to documentation.
• I/we have added tests that cover new code.
• I/we have run tests and they pass locally with the changes.
• I/we have run go fmt ./... and golangci-lint run.

[wolfyjade]

The reason why I originally marked it as a draft is because I wanted to test if it still builds just fine (since I'm using a MacBook and don't have the dev tools for Go installed on it, but would've done so soon(tm) on my ThinkPad) but I forgot that the Git repo runs them too- 🥴

[tsmethurst]

Ah thanks :)