Edward Loveall 6ccea391ed Add a bunch of well-known, LLM scrapers to robots.txt ...

Unknown if this will actually stop them, but at least I can show my intent. User agents sourced from https://darkvisitors.com/

2024-01-15 14:55:14 -05:00

..

assets/images

Initial app

2021-05-01 17:03:38 -04:00

favicon.ico

Initial app

2021-05-01 17:03:38 -04:00

robots.txt

Add a bunch of well-known, LLM scrapers to robots.txt

2024-01-15 14:55:14 -05:00