HonestWriterBot
You are probably here because you found this in your logs:
Mozilla/5.0 (compatible; HonestWriterBot/1.0; +https://honestwriter.ai/bot)This page says what it is, exactly what it does, where it is less strict than you might expect, and how to stop it. If something here does not sit right with you, write to us and we will answer. See Contact.
What it does
HonestWriter measures how AI assistants answer questions about a market, and writes articles for companies in that market. When an assistant answers, it names the pages it drew on. This bot reads those pages, so that what we write can be checked against a real source instead of invented.
It fetches the single page at the URL an assistant cited and extracts the readable text, the title and the publication date. It does not crawl your site, follow your internal links, submit forms, or log in anywhere.
Two kinds of request, and they behave differently
We would rather tell you this than have you discover it.
Reading a page an assistant cited – what this page is
about, and what almost all of our traffic is. It checks
robots.txt first, obeys Crawl-delay, paces itself, and
reads one page.
Reading a site because someone asked us to – when a
customer points us at their own domain, or a visitor runs our free preview
against one. Those requests also go out as HonestWriterBot and,
today, do not consult robots.txt. We are closing
that gap. Until we have: if you see us on a site that is not ours to read and
your rules did not stop us, mail us and we will stop it by hand the same day.
How it behaves
- It checks
robots.txtbefore every cited-page fetch. If your rules disallow the path, the page is not fetched. We keep a record that the URL exists and was refused. We do not retry it as a way around the rule. - It obeys
Crawl-delay, and waits at least one second between page requests to the same host even when you ask for nothing. There is also a ceiling on how many pages we will take from one host in a day. - It sends conditional requests. If your server gave us an
ETagor aLast-Modified, every later visit carries it back asIf-None-MatchorIf-Modified-Since, so an unchanged page costs you a304and no body. - It revisits rarely. Between 10 and 90 days, depending on how often assistants cite the page.
- It identifies itself honestly and does not retry under another crawler's name to get past a refusal. If you say no, that is the answer we take.
- It stays on the host it was pointed at. A cited URL that redirects to a different host is not followed; we record the new address and check your rules for it separately, as its own page.
The Mozilla/5.0 prefix is the crawler convention Googlebot and
bingbot also use. The (compatible; ...) part is what marks it as a
bot rather than a browser.
How to stop it
Add this to your robots.txt:
User-agent: HonestWriterBot
Disallow: /Spell the token exactly that way. We cache robots.txt for a day,
so at most one further request may go out against a copy read up to 24 hours
earlier. After that we stop.
If you would rather not touch robots.txt, mail us and we will add
your host to our own block list. We will confirm when it is done. That list is
ours, not yours, so if you later want us back you will need to tell us.
Blocking stops us fetching. It does not by itself delete what we already read. When we see your refusal we retire the pages we hold from that host, so they stop being used as source material. If you want the stored text deleted as well, say so in one line and we will delete it.
Where it comes from
Requests originate from Google Cloud Run in europe-west1, so the source
address belongs to Google's shared range rather than to us. That is why the
user-agent, not the IP address, is what to match in robots.txt.
It is also worth knowing that a user-agent is just a header and anyone can send ours. If you see traffic claiming to be us that does not look like what this page describes, send us the log lines and the timestamps and we will tell you whether it was us.
What we do with what we read
We store the readable text of the page as it stood, its title and its publication date, so that a statement we publish can be attributed to your page and dated honestly. We do not publish copies of your pages. When we use something from one, we say it came from you, link to it, and give the date we read it.
Personal data
We do not go looking for personal data and we do not build profiles of people. But we store the readable text of a page as it stood, and a page can mention real people: an author byline, a quote, a team page. We keep that text to support and date statements we publish, under our legitimate interest in publishing material that can be checked. If a page of yours holds personal data you want removed from our copy, write to privacy@honestwriter.ai and we will delete it. Our privacy policy is at honestwriter.ai/privacy.
Contact
hello@honestwriter.ai for anything about this bot: stopping it, deleting what it read, or telling us it behaved badly. For personal data specifically, privacy@honestwriter.ai.