disallow our works from being crawled by archive.org

LafayetteCollegeLibraries · Nov 13, 2019 · a657879 · a657879
1 parent 9ed3175
commit a657879
Showing 1 changed file with 5 additions and 1 deletion.
diff --git a/public/robots.txt b/public/robots.txt
@@ -1 +1,5 @@
-# See http://www.robotstxt.org/robotstxt.html for documentation on how to use the robots.txt file
+# Block the Archive-It bot from crawling (+ storing) our
+# repository items, which will be too much data for our budget.
+User-agent: archive.org_bot
+Disallow: /concern
+Disallow: /downloads