Site audits
Setting crawl limits and exclusions
Control page limits, crawl rate and the paths you would rather we left alone.
Published Updated
Our crawler respects robots.txt, throttles requests and identifies itself clearly. You can tighten it further per site.
Was this article helpful?
Want the work done for you?
Our team can implement anything the platform recommends.