Self-Hosting / Networking

Get Your Self-Hosted Site Actually Crawled and Indexed

Asked a simple question about a side project's public site — "is this actually being crawled?" — and had no way to answer it. No access logs, no Search Console, and it turned out both of the sitemap-ping URLs half the internet's guides still tell you to use have been dead for years. Here's the real, current state of getting a self-hosted site discovered, start to finish.

Check what you actually have before assuming

A site can serve traffic perfectly fine and still be completely invisible to search engines — those are unrelated facts. Two quick checks, neither of which assumes anything:

Do you have access logs at all? If your reverse proxy has no log directive configured (a genuinely common default — see the traffic analytics guide for how easy this is to miss for months), you have no way to see a crawler visit even if one happens. Turn logging on first; everything else in this guide is easier to verify once you can actually see requests hitting the box.
Search for your own domain. A site:yourdomain.com query, or just your bare domain in quotes, returning zero real results is a strong signal nothing's indexed yet — not proof of a bug, but a real starting point.

Both classic "ping" URLs are dead. google.com/ping?sitemap=... 404s — Google retired it in June 2023. bing.com/ping?sitemap=... returns 410 Gone — also fully retired. If a guide (including older ones) tells you to just curl one of these after publishing a sitemap, it's out of date. There is no replacement URL that works the same way for Google; see the Search Console section below. Bing's real replacement is IndexNow, covered next.

IndexNow — the actual modern replacement, for everyone except Google

IndexNow is a real, currently-working open protocol (originally Microsoft's) supported by Bing, Yandex, Naver, Seznam, and Yep. One submission to a shared endpoint fans out to every participating engine at once. It doesn't guarantee indexing — it guarantees the engine knows a URL changed and can prioritize crawling it, which is still a real improvement over waiting for a routine crawl to happen on its own schedule.

1. Generate a key and host it at your domain root

# Any random 32-128 character hex string works
python3 -c "import secrets; print(secrets.token_hex(32))"

Save that string as a plain text file named <key>.txt at your site's root — e.g. https://example.com/a1b2c3....txt, containing just the key itself, nothing else. Deploy it and confirm it's actually reachable before moving on:

curl https://example.com/a1b2c3....txt

2. Submit your URLs

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "a1b2c3...",
    "keyLocation": "https://example.com/a1b2c3....txt",
    "urlList": [
      "https://example.com/",
      "https://example.com/some-page.html"
    ]
  }'

A 202 Accepted response means it was received — that's confirmation of submission, not of indexing, but it's the whole job done on your end. Pull your URL list straight out of your existing sitemap.xml rather than typing them by hand if you have more than a handful of pages.

Google — no shortcut, Search Console is the only real path

Google doesn't support IndexNow. Its Indexing API is restricted to job-posting and event content for normal accounts — not usable for a general site. Search Console verification is genuinely the only lever for direct-to-Google submission.

You don't have to grant Google OAuth access to your DNS. If Search Console detects you're on Cloudflare (or another supported provider), it offers a one-click "Start Verification" button that authorizes Google to read/write your DNS account directly. You don't need that — switch the "Instructions for" dropdown to "Any DNS provider" instead, and it gives you a plain google-site-verification=... TXT record to add by hand. Same result, no standing integration granted to a third party.

Two property types exist — pick Domain, not URL-prefix, unless you have a specific reason not to:

Domain property — covers www, non-www, and both http/https in one property, verified via a single DNS TXT record.
URL-prefix property — covers exactly the one prefix you enter (e.g. only https://www.example.com/) and nothing else, including subdomains. Several supported verification methods (HTML file upload, meta tag, Google Analytics), but real coverage gaps if your site has more than one hostname pointing at it.

Check for this exact gap if a site's been live a while. A URL-prefix property verified early on (common — it's the default suggestion) silently doesn't cover any subdomain added later. Worth checking your property list for a narrower prefix property sitting where a domain property should be, especially on a site that's grown subdomains over time.

Once verified, submit your sitemap from the property's Sitemaps page — use the full URL (https://example.com/sitemap.xml), not just the relative path; the relative form can get rejected as an "invalid sitemap address" on a Domain-type property depending on which hostname it's ambiguous between. A freshly submitted sitemap will often show "Couldn't fetch" for a while immediately after submission — that's the normal pre-first-crawl placeholder status, not a real error, as long as the sitemap URL itself returns a clean 200 when you curl it directly (test with both a plain request and a spoofed Googlebot user-agent, in case anything's filtering by UA).

Cheap, real backlinks you're probably not using

Backlinks from other real, relevant sites are still the single biggest lever for both Google and Bing — nothing technical here substitutes for that. But two genuinely free, legitimate ones are easy to miss if a project has a public GitHub repo:

The repo's "Website" field (repo Settings, or gh api -X PATCH repos/OWNER/REPO -f homepage="https://example.com") — often left blank even on projects that have a real site.
A link in the README — github.com is an extremely heavily-crawled domain; a project whose own README never mentions its own site is leaving that for free.

Putting it together

For a typical self-hosted project's public site, the honest full checklist looks like:

Confirm access logging is actually on.
Generate and host an IndexNow key; submit every URL in your sitemap once.
Verify a Domain-type Search Console property via a manual DNS TXT record (not the OAuth auto-verify); submit your sitemap there too.
Add the two free GitHub backlinks if the project has a public repo.
Everything past this point is content and real backlinks — there's no more infrastructure to stand up.