You are viewing a single comment's thread from:

RE: Google's Bot Imposes a Large Impact on the Performance of SteemYY.com

in #witness-category2 years ago

I had the same issue!!! google bot and also Bing bot impacted my site.
I changed the robots.txt and limited the pages they can crawl and it seems to be working... not 100% because some bots ignore robots.txt

Sort:  

For me, I don't want to ban entirely the Bot, is there a way to rate limit bots in robots.txt?

I am not a robots.txt expert... but I think so...
at least you tell them which bots are allowed what pages and others are delayed crawling... or something like that

Coin Marketplace

STEEM 0.21
TRX 0.24
JST 0.038
BTC 94636.84
ETH 3278.73
USDT 1.00
SBD 3.16