gh-mirrors/scribe
1
0
Fork 0
mirror of https://git.sr.ht/~edwardloveall/scribe synced 2025-12-10 04:31:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
6ccea391ed5e05cdd65b78fc1c7ada40f22fffe3
scribe/public/robots.txt
Edward Loveall 6ccea391ed Add a bunch of well-known, LLM scrapers to robots.txt 
Unknown if this will actually stop them, but at least I can show my intent. User agents sourced from https://darkvisitors.com/
2024-01-15 14:55:14 -05:00

675 B
Raw Blame History

View Raw
Reference in New Issue View Git Blame Copy Permalink