Self-Hosting / Networking

Get a Self-Hosted Site Crawled and Indexed: Sitemaps, IndexNow, Search Console

I asked a simple question about a side project's public site. Is this being crawled? I had no way to answer it. No access logs, no Search Console. And both of the sitemap-ping URLs half the internet's guides still tell you to use have been dead for years. Here is the real, current state of getting a self-hosted site discovered, start to finish.

Check what you actually have before assuming

A site can serve traffic perfectly fine and still be completely invisible to search engines. Those are unrelated facts. Two quick checks, neither of which assumes anything:

Do you have access logs at all? If your reverse proxy has no log directive configured, you cannot see a crawler visit even when one happens. That is a genuinely common default. The traffic analytics guide covers how easy it is to miss for months. Turn logging on first. Everything else here is easier to verify once you can see requests hitting the box.
Search for your own domain. A site:yourdomain.com query, or your bare domain in quotes, returning zero real results is a strong signal nothing is indexed yet. Not proof of a bug. A real starting point.

BE WARNED: both classic "ping" URLs are dead. google.com/ping?sitemap=... 404s. Google retired it in June 2023. bing.com/ping?sitemap=... returns 410 Gone, also fully retired. If a guide tells you to curl one of these after publishing a sitemap, it is out of date. There is no replacement URL that works the same way for Google. See the Search Console section below. Bing's real replacement is IndexNow, covered next.

IndexNow: the modern replacement, for everyone except Google

IndexNow is a real, currently working open protocol, originally Microsoft's, supported by Bing, Yandex, Naver, Seznam and Yep. One submission to a shared endpoint fans out to every participating engine at once. It does not guarantee indexing. It guarantees the engine knows a URL changed and can prioritize crawling it. Still a real improvement over waiting for a routine crawl on its own schedule.

1. Generate a key and host it at your domain root

# Any random 32-128 character hex string works
python3 -c "import secrets; print(secrets.token_hex(32))"

Save that string as a plain text file named <key>.txt at your site's root, so https://example.com/a1b2c3....txt. It contains the key itself and nothing else. Deploy it and confirm it is reachable before you move on:

curl https://example.com/a1b2c3....txt

2. Submit your URLs

curl -X POST "https://api.indexnow.org/indexnow" \
  -H "Content-Type: application/json; charset=utf-8" \
  -d '{
    "host": "example.com",
    "key": "a1b2c3...",
    "keyLocation": "https://example.com/a1b2c3....txt",
    "urlList": [
      "https://example.com/",
      "https://example.com/some-page.html"
    ]
  }'

A 202 Accepted response means it was received. That confirms submission, not indexing, and it is the whole job done on your end. Pull your URL list straight out of your existing sitemap.xml rather than typing them by hand, if you have more than a handful of pages.

Google: no shortcut, Search Console is the only real path

Google does not support IndexNow. Its Indexing API is restricted to job-posting and event content for normal accounts, so it is not usable for a general site. Search Console verification is the only lever for direct-to-Google submission.

You do not have to grant Google OAuth access to your DNS. If Search Console detects you are on Cloudflare, or another supported provider, it offers a one-click "Start Verification" button that authorizes Google to read and write your DNS account directly. You do not need that. Switch the "Instructions for" dropdown to "Any DNS provider" and it gives you a plain google-site-verification=... TXT record to add by hand. Same result, no standing integration granted to a third party.

Two property types exist. Pick Domain, not URL-prefix, unless you have a specific reason not to:

Domain property. Covers www, non-www, and both http and https in one property, verified with a single DNS TXT record.
URL-prefix property. Covers exactly the one prefix you enter, say only https://www.example.com/, and nothing else, subdomains included. It supports several verification methods, HTML file upload, meta tag, Google Analytics. It also leaves real coverage gaps if your site has more than one hostname pointing at it.

BE WARNED: check for this exact gap if a site has been live a while. A URL-prefix property verified early on does not cover any subdomain added later. Verifying one early is common, since it is the default suggestion. Check your property list for a narrow prefix property sitting where a domain property should be. Especially on a site that has grown subdomains over time.

Once verified, submit your sitemap from the property's Sitemaps page. Use the full URL, https://example.com/sitemap.xml, not the relative path. The relative form gets rejected as an "invalid sitemap address" on a Domain-type property, depending which hostname it is ambiguous between. A freshly submitted sitemap often shows "Couldn't fetch" for a while. That is the normal pre-first-crawl placeholder status, not a real error, as long as the sitemap URL returns a clean 200 when you curl it directly. Test with a plain request and with a spoofed Googlebot user-agent, in case anything filters by UA.

Cheap, real backlinks you're probably not using

Backlinks from other real, relevant sites are still the single biggest lever for both Google and Bing. Nothing technical here substitutes for that. Two free, legitimate ones are easy to miss if a project has a public GitHub repo:

The repo's "Website" field. Set it in repo Settings, or with gh api -X PATCH repos/OWNER/REPO -f homepage="https://example.com". It gets left blank even on projects that have a real site.
A link in the README. github.com is an extremely heavily crawled domain. A project whose own README never mentions its own site is leaving that on the table.

Putting it together

For a typical self-hosted project's public site, the honest full checklist looks like:

Confirm access logging is actually on.
Generate and host an IndexNow key. Submit every URL in your sitemap once.
Verify a Domain-type Search Console property with a manual DNS TXT record, not the OAuth auto-verify. Submit your sitemap there too.
Add the two free GitHub backlinks if the project has a public repo.
Everything past this point is content and real backlinks. There is no more infrastructure to stand up.