User-agent: * Disallow: /wp-login.php Disallow: /xmlrpc.php Disallow: /readme.html Disallow: /license.txt Disallow: /go/ User-agent: AhrefsBot Disallow: / User-agent: Screaming Frog SEO Spider Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: /