Resolving The Most Common Website Scan Errors
When AccessibleWebBot attempts to scan your website, it behaves differently than a standard web browser. Occasionally, security configurations or server errors will block our bot from successfully scanning your pages.
When this happens, you will receive an email notification with the subject line “Action Required: Scan Error for [Your Website Name]”.
This guide breaks down why these scan errors happen, how to identify them using HTTP status codes, how to fix them, and answers the most common questions regarding automated bot security.
Why does my site load in a browser, but your bot cannot access it?
This is the most common question our support team receives. A client will often say, "I can open my website perfectly fine on my phone and desktop, so why is your scanner throwing an error?"
In short, web browsers are designed to be highly forgiving; secure bots are not.
Standard web browsers (like Chrome, Safari, or Edge) are built for everyday consumer convenience. If a website has an underlying security omission or misconfiguration, a browser will often silently ignore or fix it behind the scenes, or at most show the user a warning screen with an option to proceed anyway.
By contrast, AccessibleWebBot is a verified, secure bot. The rules and etiquette for bots on the internet are different than for humans. We do our best to operate our bot in a well-behaved manner, following the preferences of website owners as expressed through robots.txt. If we cannot clearly determine that our bot traffic is welcome, we will not visit a website or page.
Because we process data programmatically, our bot must operate on a strict zero-trust model. If your web server fails the standard security handshake, our bot cannot guarantee the integrity of the data stream and will safely abort the scan rather than processing untrusted content.
Common Scan Errors and Status Codes
When our bot has trouble scanning your website, it is almost always caused by one of five things. If you are looking at your RAMP dashboard, you will likely see a 4xx (Client Error) or 5xx (Server Error) HTTP status code associated with the failure.
They are listed below in order of how frequently they occur:
1. Your website’s robots.txt file is blocking our bot
Common Status Code: “robots.txt is blocking access”
A robots.txt file tells automated visitors which parts of your site they are allowed to access. If your file contains a rule that blocks our bot, our bot will respect it and halt the scan.
If your robots.txt file cannot be retrieved due to a network failure or server error such as 503 Service Unavailable, we assume that no bot access is allowed. (We follow the RFC 9309 specification.)
How to fix: You need to grant our bot permission to scan your pages. Review our quick guide to allowing AccessibleWebBot to add the proper exception rules to your file.
If you don’t have a robots.txt file, you must either create one, or ensure that attempting to access robots.txt will result in a clear “unavailable” error such as 404 Not Found.
2. Bot blocking or rate limiting at the CDN level
Common Status Codes: 429 Too Many Requests or 403 Forbidden
Content Delivery Networks (CDNs) like Cloudflare, Akamai, or AWS CloudFront use Web Application Firewalls (WAF) to block bad automated traffic. A 429 Too Many Requests means our bot triggered the server's rate-limiting security, essentially asking our client to slow down. A 403 here means the CDN outright blocked the connection.
How to fix: Have your IT or development team add an exception rule for AccessibleWebBot in your CDN/WAF dashboard. Since we are a Cloudflare-verified bot, simply ensuring that your rule allows "Verified Bots" often resolves this immediately.
You can filter by our User-Agent string which will always contain AccessibleWebBot, or reach out to us for details about our bot’s IP addresses.
3. Firewalls and Hosting Plugins
Common Status Codes: 403 Forbidden, 405 Method Not Allowed, or 503 Service Unavailable
Network-level firewalls or security plugins (like Wordfence on WordPress, or host-level protections like WP Engine) may look at our crawling pattern and mistake it for a brute-force attack. They might drop the connection abruptly, block it with a 403, throw a 405 Method Not Allowed, or return a 503 Service Unavailable proxy error. Because the connection can drop abruptly, our system may sometimes initially interpret this as a robots.txt retrieval issue.
How to fix: Your infrastructure team will need to allow our bot’s User-Agent string (AccessibleWebBot) or IP addresses in your server's hosting firewall configurations. Reach out to us for details about our bot’s IP addresses.
4. The website is temporarily down, overloaded, or moved
Common Status Codes: 503 Service Unavailable, 504 Gateway Timeout, 500 Internal Server Error, or a robots.txt error
A 503 indicates the server was misconfigured, overloaded, or unavailable when we tried to access it. A 504 Gateway Timeout means the server proxy didn't get a response in time. If our scanner hits these or broken redirect chains, it cannot read your page content.
How to fix: If it’s a temporary outage, no action is needed. Our system will automatically attempt to resume scanning once your site is back online. You can check your server logs to see if it was experiencing downtime at the time of the request. However, if your website has permanently moved to a new domain name, make sure to update your domain settings inside RAMP.
5. There is an issue with your website’s SSL/TLS certificate
Common Status Code: “robots.txt is blocking access”
If your website’s security handshake fails, our scanner will safely abort the connection rather than risk accessing a compromised site.
This will likely show as a robots.txt issue, because we verify access by loading robots.txt before scanning the page itself.
How to fix: This requires your IT department or web development team to update, renew, or reconfigure the certificate on your web server (detailed below).
SSL/TLS Certificate Requirements & Fixes
When our scanner attempts a connection, it looks for a completely valid, publicly verifiable SSL/TLS certificate. Here are some specific certificate issues that will cause our bot to stop scanning, alongside instructions for your development team to fix them:
Incomplete Certificate Chain (Most Common)
Web servers are supposed to send your specific website certificate along with "intermediate" certificates that link your site back to a globally trusted Root Certificate Authority (CA). Modern browsers often cache intermediate certificates and piece the chain together on their own, but secure backend automated tools do not. If your server has an incomplete chain, our bot will abort the scan.
How to fix: Ask your developer or IT team to ensure all intermediary certificates are bundled, concatenated correctly, and uploaded to your web server configurations (e.g., Nginx, Apache, or IIS).
Expired Certificates
If your certificate's validity window has passed, the encryption can no longer be verified as secure.
How to fix: Renew your SSL/TLS certificate through your certificate authority (such as Let's Encrypt, DigiCert, etc.) and deploy the updated certificate to your production server.
Hostname Mismatch
This happens if your SSL certificate is registered for one domain (e.g., example.com), but you are trying to scan a different subdomain (e.g., staging.example.com) that isn't explicitly covered by the certificate or its Wildcard rules.
How to fix: Update your certificate to include the specific subdomain as a Subject Alternative Name (SAN), or issue a separate certificate specifically covering the exact domain URL added to RAMP.
Self-Signed or Untrusted Certificates
If your certificate was self-generated rather than issued by a recognized public certificate authority, our system will reject it. This is highly common in internal testing environments or pre-production staging sites.
How to fix: Replace the self-signed certificate with a free, valid public certificate (like Let's Encrypt) for the staging environment, or swap the RAMP settings to point to your live, publicly accessible production site.
How to Verify Your Fixes
Before reaching out to support or waiting for your next scheduled RAMP scan, you can check your website’s certificate health using third-party verification tools.
We highly recommend using the free Qualys SSL Labs Server Test.
Simply enter your website URL and run the test. If the results flag an "Incomplete Certificate Chain" or any trust issues, hand those technical specs directly over to your web development or IT infrastructure team. Once they clear those flags, RAMP will be able to successfully scan your site again.
Need Extra Help?
Details about our bot’s IP addresses are available upon request.
If your development team has verified that your certificates, firewall rules, and robots.txt files are completely clear, but the error persists, please reach out for support. You can click here to submit a ticket, email us at support@accessibleweb.com, or launch our Live Chat for instant troubleshooting. We are more than happy to dive into the technical details with your team.