SteedBot

SteedBot is how Steed reads a website's look, words and page addresses. It visits a site only when someone building with Steed gives us the address of the site they have today, so their new one can start from it, or when they move that site's domain to Steed, so every old link leads to the right new page.

• The site's robots.txt, first.
• Its home page and every page of that site the home page links to, the stylesheets they use, its logo, and the pictures on those pages (up to 200 a page), from wherever the site keeps them.
• Where a site draws itself with scripts, up to 10 of its pages opened in a browser of Steed's with the site's own scripts running, so they read as its visitors see them.
• When the owner moves the domain to Steed, once: the page list the site publishes for search engines (its sitemap) or, without one, the links on its home page, and the title of each of those pages, up to 2,048.

SteedBot reads the site's pages only from that site's own address, and only over https; its pictures come from wherever the site keeps them.

So Steed's designer can see the site as it looks today, Steed then opens that one home page in a browser with every script taken out, so nothing of the site's runs. That browser can't reach the site: the page's stylesheets are already inside it, and its pictures don't load. Only web fonts load, from a public font service, and those requests say SteedBot too.

SteedBot reads nothing beyond that: no page of any other site, and no links past the site's own pages and page list. It collects nothing about the site's visitors.

When someone gives Wren, Steed's planner, the address of their site in the chat, Wren reads its pages so their new site can carry its words over: up to 30 pages in a reply, from any address in the conversation. Steed's team can also search the web for their business, and Steed's designer may read their current site and any site they name for its look. All of these go through the web reader of the AI service Steed uses, so they come from that service, not from SteedBot, and a SteedBot line in robots.txt doesn't cover them.

SteedBot follows robots.txt, and reads it first. If robots.txt keeps it off, Steed reads nothing and opens nothing. A page robots.txt closes to SteedBot is never read; if the site's own page list names it, its address is still kept, so the old link can lead to its new page. To keep it off your site, add:

User-agent: SteedBot
Disallow: /

Every request SteedBot makes to read or draw a site names itself: SteedBot/1.0 (+https://steed.biz/bot; reading the home page, styles, logo and page addresses of a site its owner asked Steed to rebuild or move).

Write to support@steed.biz.