Facebook and Crawl-delay in Robots.txt? - Stack Overflow

We don't have a crawler. We have a scraper that scrapes meta data on pages that have like buttons/are shared on FB.

Lee mas