6 lines
98 B
Plaintext
6 lines
98 B
Plaintext
User-agent: *
|
|
Disallow:
|
|
Crawl-Delay: 5
|
|
Request-rate: 18/1m 2100-0529
|
|
Request-rate: 6/1m 0530-2059
|