User-agent: PetalBot User-agent: GPTBot User-agent: Bytedance User-agent: Amazonbot User-agent: Amzn-SearchBot User-agent: ahrefs User-agent: meta-externalagent User-agent: externalhit_uatext User-agent: Lightpanda User-agent: PerplexityBot User-agent: Gort User-agent: Baiduspider User-agent: Baiduspider-video User-agent: Baiduspider-image User-agent: Yandex User-agent: Yandex 3.0 User-agent: coccocbot User-agent: youbot User-agent: applebot User-agent: Barkrowler Disallow: / User-agent: * Allow: / Disallow: /articles/* Disallow: /books-compact/* Disallow: /static/* # amazon crawler goes nuts. User-agent: * Disallow: /catid--567374/channel Disallow: /catid--567374/books/