init
This commit is contained in:
5
robots.txt
Normal file
5
robots.txt
Normal file
@@ -0,0 +1,5 @@
|
||||
User-agent: *
|
||||
Disallow:
|
||||
Crawl-Delay: 5
|
||||
Request-rate: 18/1m 2100-0529
|
||||
Request-rate: 6/1m 0530-2059
|
||||
Reference in New Issue
Block a user