robotstxt.org
The Web Robots Pages
Last scanned Aug 5, 2026, 2:00 PM · 9 views
Overall score
Fair
Summary
This website uses Apache hosted on MYTHIC - Mythic Beasts Ltd, GB. SEO is fair (55/100) and accessibility is good (75/100). Security scores poor (35/100) and performance excellent (100/100). Top issue: Missing meta description.
- United Kingdom
- Hosted on MYTHIC - Mythic Beasts Ltd, GB
Top issues to fix
13 issues found · 6 high priority
No valid HTTPS/TLS certificate
SecurityServe the site over HTTPS (a free Let's Encrypt certificate works) — browsers mark HTTP sites as Not Secure and Google demotes them.
Missing Strict-Transport-Security header
SecurityAdd HSTS so browsers always use HTTPS — this closes the protocol-downgrade attack window.
Missing Content-Security-Policy header
SecurityAdd a CSP to control which sources can run scripts — the strongest single defense against XSS.
Missing meta description
SEOAdd a ~150-character meta description to control your search-result snippet and lift click-through rate.
No H1 heading found
SEOAdd exactly one H1 describing the page's main topic — it anchors the document outline for search engines and screen readers.
Missing viewport meta tag
AccessibilityAdd <meta name="viewport" content="width=device-width, initial-scale=1"> — without it the site is broken on mobile.
+7 more issues — details in the sections below.
SEO
6 issue(s) found
Performance
239 ms
Security
HTTPS & header checks
Accessibility
3 issue(s) found
Score history
0 overall since Jul 26, 2026
- Overall
- 65
- SEO
- 55
- Security
- 35
- Performance
- 100
- Accessibility
- 75
Contact
Email, phone, and address published on this site's own pages.
Meta tags
- Title
- The Web Robots Pages
- Description
- Canonical
- Language
- Charset
- Viewport
- Robots meta
- Manifest
Technologies
Detected frameworks, languages, and tools.
Hosting & CDN
- Hosting provider
- MYTHIC - Mythic Beasts Ltd, GB
- CDN provider
- —
- IP address
- 93.93.131.3
- ASN
- AS44684 (MYTHIC - Mythic Beasts Ltd, GB)
DNS records
| Type | Value |
|---|---|
| A |
|
| AAAA |
|
| MX | — |
| TXT | — |
| NS | — |
| CAA | — |
| CNAME | — |
| IPv6 support | Yes |
TLS / SSL certificate
No TLS certificate information available.
Security headers
- Strict-Transport-SecurityMissing
- Content-Security-PolicyMissing
- X-Frame-OptionsMissing
- X-Content-Type-OptionsMissing
- Referrer-PolicyMissing
- Permissions-PolicyMissing
Compression: gzip · 0 cookie(s) observed
Content & assets
Headings
H1 0 · H2 1 · H3 0
Links
15 internal · 1 external · 0 broken
Images
1 total · 0 missing alt
Scripts / Styles
12 scripts · 3 stylesheets
JS / CSS size
843 KB · 7.0 KB
Page size
4.3 KB
Robots.txt & Sitemap
- robots.txt
- Found
- Disallow rules
- 0
- sitemap.xml
- Not found
- URLs in sitemap
- 0
SEO issues
- Missing meta description
- Missing canonical link
- No H1 heading found
- No structured data (JSON-LD) found
- Missing OpenGraph tags
- sitemap.xml not found
Accessibility issues
- No H1 heading found
- Missing <html lang> attribute
- Missing viewport meta tag
Scan details
- Requested URL
- https://robotstxt.org
- Final URL
- http://www.robotstxt.org/
- HTTP status
- 200
- HTTP version
- HTTP/1.1
- Response time
- 239 ms
- HTML size
- 4.3 KB
- Redirect chain
- https://robotstxt.org → http://www.robotstxt.org/
See something wrong on this report? Request a correction
Related websites
Same technology stack
hovbi.com
Hovbi Trends — Discover What People Are Searching For
worlddata.info
Worlddata: The world in numbers
midnightbsd.org
MidnightBSD | The BSD-Derived Desktop Operating System
digitaldreamdoor.com
DigitalDreamDoor.com - Greatest Music, Movie, and Book lists
ezit.hu
EZIT - Magyar Hosting
01.org
Open Ecosystem
Same hosting provider
openstrike.co.uk
Openstrike: Improving Business Through Open Source
fixmystreet.com
FixMyStreet – Down for maintenance
piwheels.org
piwheels - Home
in-the-sky.org
In-The-Sky.org
dotat.at
Tony Finch
hiveworldterra.co.uk
Frequently asked questions
- What technology does robotstxt.org use?
- robotstxt.org is built with Google Ads, Apache.
- Is robotstxt.org secure (HTTPS)?
- robotstxt.org did not present a valid TLS certificate at the time of the last scan.
- Who hosts robotstxt.org?
- robotstxt.org is hosted on MYTHIC - Mythic Beasts Ltd, GB.
- What is robotstxt.org's SEO score?
- robotstxt.org scored 55/100 for SEO in the latest analysis (fair).
- What is robotstxt.org's overall website score?
- robotstxt.org has an overall SiteGraph score of 65/100, combining SEO, performance, security, and accessibility.
- Has robotstxt.org's score changed over time?
- robotstxt.org's overall score has stayed steady at 65/100 since Jul 26, 2026.
OpenGraph & Twitter Card
OpenGraph
Twitter Card