What crawls a brand-new website in 2026

Launch-week notes from Sunshine Atlas — 3,833 destinations, 4,147 prerendered pages, one small VPS. Five days of server logs on who actually showed up: ChatGPT answers with the site daily, Meta read all of it in one sitting, Google is methodical, Bing hasn't really knocked. By the Sunshine Atlas team · 12 July 2026.

I run Sunshine Atlas, a climate atlas that answers "where is it sunny in November?" with data instead of listicles. It went live on 8 July 2026: an interactive WebGL globe on the homepage and a static, prerendered page for every destination and every month of the year. Five days later the logs tell a story I haven't seen written down with numbers attached: a new website's first real audience is machines — and the interesting question is which ones, and what each of them does with you.

Five days of crawlers, counted

40,863 requests hit the site between 8 and 12 July. Verified humans — browser user-agent plus asset fetches plus several requests, minus everything that reverse-DNSes to a datacenter — number about 40 a day. Nearly everything else is bots. The named ones:

Method: nginx combined logs, bucketed by user-agent; humans separated by UA + CSS/JS asset fetches + ≥3 requests/day, with datacenter ranges removed via reverse DNS. Figures are a snapshot as of 12 July 2026.

Three kinds of machine visitor

The user-agents sort into three castes that behave nothing alike:

• Search indexers (Googlebot, bingbot) crawl so they can maybe send you humans later. Googlebot was polite and complete: the full 4,147-URL sitemap within three days, then a steady 300–900 pages a day of re-checking. The payoff is deferred: Search Console shows 1,810 impressions, 23 clicks, average position 17.6 in the first three days of data — page two, where nobody scrolls. Fine. New domain, no links yet; that's the deal with Google, and the 382 distinct queries it's already testing me on suggest the long tail will come.
• Corpus crawlers (meta-externalagent, GoogleOther, Amazonbot, most of ClaudeBot's volume) read everything and send nothing. Meta ignored the site for four days, then read 4,153 pages in one sitting. These visits will never appear in an analytics dashboard as a referral — the site simply becomes part of what their models know. Publishing under CC BY 4.0, that's not theft, it's the strategy: being in the corpus is the distribution.
• Live answer fetchers (ChatGPT-User, Claude-User) are the new thing, and the reason this post exists. When someone asks ChatGPT "where's sunny in November?", it fetches a page from my five-day-old site right then, reads the table, and answers with a citation. That's been happening about 50 times a day since day two. Google took three days to crawl me and will take months to rank me; assistants started sending my pages to real users' eyeballs in 48 hours.

The Bing paradox

The site pings IndexNow on every deploy, and Bing Webmaster Tools happily confirms receipt — 16,000+ URL submissions accepted in one 15-hour stretch. The crawler then came thirteen times in five days, and Bing's search results show nothing at all. Submission pipes and crawl appetite are entirely separate budgets; IndexNow gets you into a queue, not into an index. Meanwhile OpenAI — whose ChatGPT Search is supposedly built on Bing's index — ran its own crawler over ~800 of my pages. Even Microsoft's flagship AI customer doesn't wait for bingbot.

Why machines could eat the site this fast

None of this is luck; the site is built to be read by things that read HTML. The stack is deliberately boring:

• One selection rule. Every destination is a metro-primary airport — one per metropolitan area, so London appears once, not five times. That yields 3,833 real, flyable places and kills the "top 10 hidden gems you can't actually reach" genre at the data level.
• One number. Each destination-month gets a 0–100 Sunshine Score: 100 × warmth × (0.5 + 0.5 × (0.55 × dryness + 0.45 × sunniness)), computed from long-term climate normals (CRU climatology; sea temperatures via Open-Meteo; places from GeoNames & OurAirports). Warmth multiplies the whole thing, so a clear −5 °C morning scores zero — it's opinionated, and the methodology page says exactly how.
• No framework, no CMS, no database. The homepage globe is React+WebGL, but the other 4,146 pages come out of a single 2,300-line Node script that runs after vite build and writes plain HTML: real <table> elements, the answer stated in the first paragraph, JSON-LD (Place, FAQPage, ItemList, Dataset) on every page. A page that needs a JavaScript runtime to show its content is invisible to ChatGPT-User — it reads raw HTML, and so does everything else in the table above.
• Honest sitemap dates. <lastmod> tracks when the underlying climate data changed, not when I last deployed. Stamping 4,000 URLs "today" on every deploy teaches crawlers your dates are noise; Googlebot's tight re-crawl cadence suggests it trusts them.
• Deploys are a tarball. tar | ssh to a timestamped release directory, one atomic symlink swap, two old releases kept for rollback. The VPS also hosts my other map sites; nginx never gets touched.
• llms.txt, for the record: three fetches, none from a major AI crawler. It's one static file so it stays, but as of July 2026 nobody's reading the courtesy map — they just crawl.

Feeding the machines on purpose

Two launch-week decisions treat assistants as a first-class channel rather than a side effect:

The dataset is open. Every number on the site — monthly scores, temperatures, rainfall, sea temperature for 3,833 destinations — is a CSV/JSON download, CC BY 4.0, with a Zenodo DOI, mirrored to GitHub, Hugging Face and Kaggle. If a model is going to learn "Faro gets 3,044 hours of sun," I'd like it to learn it from me, with attribution baked into the license.

The site speaks MCP. A 256-line Model Context Protocol server runs next to nginx — stateless streamable-HTTP, no auth, rate-limited, reading the same published dataset through the release symlink (deploys refresh it with no restart). Point Claude or any MCP client at https://sunshineatlas.com/api/mcp and it can rank, look up and compare destinations live; every answer carries attribution and the destination URLs. It's listed in the official MCP registry. Whether assistants become a real referral channel is exactly the experiment — the marginal cost was an afternoon and 256 lines.

What I'd tell the next person launching a content site

• Prerender everything. Your first thousand readers are machines that don't run your JavaScript.
• State the answer in the first paragraph, in plain HTML, with a real table. That's what the live fetchers quote.
• Watch ChatGPT-User in your logs, not just Googlebot. One is your future ranking; the other is citations today.
• Corpus crawlers will take everything regardless — decide what your license says about it before launch, and make openness work for you.
• Don't confuse IndexNow acceptance with Bing caring. And don't confuse llms.txt with a channel.
• Humans are still the point. ~40 a day so far, page-two Google, one Hacker News submission that got a single point and sixteen curious visitors. The machines came first; the people are the compounding return the machines are supposed to bring. Ask me in six months.

The data behind everything here is free: download the dataset (CC BY 4.0) · connect your AI assistant · read the methodology · or just spin the globe.