#Default robots.txt file (which asks all bots to crawl slowly but still index everything). User-agent: * Allow: /*.js* Allow: /*.css* Allow: /*.png* Allow: /*.jpg* Allow: /*.gif* Crawl-delay: 1 Disallow: