Crawler
ReadyLayerBot
You are probably here because you found this in your server logs. Here is exactly what it is, why it visited, and how to stop it.
Our full User-Agent
Mozilla/5.0 (compatible; ReadyLayerBot/1.0; +https://readylayer.io/bot)Every request we make carries it — page fetches, robots.txt, sitemaps, link checks, and our headless browser. We never disguise ourselves as a normal browser.
Blocking
How to block us
Add this to your robots.txt. We read it before every crawl and honour it — no form to fill in, no waiting on us.
# Block ReadyLayer entirely User-agent: ReadyLayerBot Disallow: /
To restrict us to part of your site, use ordinary Disallow rules in that group — we apply them the same way. We also obey the User-agent: * group, so if you have already disallowed everything, we are already blocked.
Matching is case-insensitive and we accept any user-agent group containing readylayer, so you do not have to get the capitalisation right.
Behaviour
What it does, and what it will never do
What it does
- Requests publicly accessible pages and reads their HTML, in order to score how well AI agents can understand them.
- Fetches your robots.txt and sitemap first, and follows links found on the pages it is allowed to visit.
- Renders pages in a headless browser, so it loads the scripts, styles and images an ordinary visitor would.
- Visits one page at a time, never in parallel, and stops at a page limit.
What it never does
- Sends cookies, credentials or authentication of any kind — it sees only what a logged-out visitor sees.
- Submits forms, or issues anything other than read requests.
- Attempts to reach admin paths, private networks or APIs.
- Ignores robots.txt.
Why us
Why it visited your site
ReadyLayer only crawls a site when a customer adds it to their account, and our terms require that they own it or are authorised to act for it.
So the usual explanation is that you, a colleague, or an agency working for you signed up and added your domain. We audit it, then generate a machine-readable copy of your content so AI agents can read your site properly.
If you did not authorise this
Tell us and we will investigate, stop the crawl, and remove the stored data. Email privacy@readylayer.io with your domain. You do not need an account, and blocking us in robots.txt works immediately either way.
What we store and for how long is set out in our Privacy Policy, and the obligation on customers to only submit sites they control is in our Terms of Service.
Curious
Want to see what agents make of your site?
The same analysis this crawler runs is available to you. Scan any page free, no signup.