gh-mirrors/scribe
1
0
Fork 0
mirror of https://git.sr.ht/~edwardloveall/scribe synced 2025-12-10 04:31:22 +00:00
Code Issues Packages Projects Releases Wiki Activity
Files
main
scribe/public
History
Edward Loveall 6ccea391ed Add a bunch of well-known, LLM scrapers to robots.txt 
Unknown if this will actually stop them, but at least I can show my intent. User agents sourced from https://darkvisitors.com/
2024-01-15 14:55:14 -05:00
..
assets/images
Initial app
2021-05-01 17:03:38 -04:00
favicon.ico
Initial app
2021-05-01 17:03:38 -04:00
robots.txt
Add a bunch of well-known, LLM scrapers to robots.txt
2024-01-15 14:55:14 -05:00