[chore] Update robots.txt with more AI bots

[daenney]

Description

Updates our robots.txt to include more AI bots.

Checklist

Please put an x inside each checkbox to indicate that you've read and followed it: [ ] -> [x]

If this is a documentation change, only the first checkbox must be filled (you can delete the others if you want).

• I/we have read the GoToSocial contribution guidelines.
• I/we have discussed the proposed changes already, either in an issue on the repository, or in the Matrix chat.
• I/we have not leveraged AI to create the proposed changes.
• I/we have performed a self-review of added code.
• I/we have written code that is legible and maintainable by others.
• I/we have commented the added code, particularly in hard-to-understand areas.
• I/we have made any necessary changes to documentation.
• I/we have added tests that cover new code.
• I/we have run tests and they pass locally with the changes.
• I/we have run go fmt ./... and golangci-lint run.

[NyaaaWhatsUpDoc]

we could probably write a script that automatically pulls in updates for this at some point :p

the place i'm pulling from for my header filters repo is here which we could use.

[daenney]

That repo doesn't seem to classify them though. I've only been adding AI crawlers and some ultra-creepy SEO/Marketing Intel stuff.

I've been manually syncing it with https://github.com/ai-robots-txt/ai.robots.txt from time to time.