Not all public websites are crawled by Google

I discovered quite by chance yesterday that not all Hugo websites are crawled by Google.

As Henry Leach explains when he was trying to search an old blog post in 2021

And that’s how I found out that I hadn’t configured a robots.txt file for my updated Hugo site, and Google was politely ignoring it.

It seems the same is true of 2024 as well. Maybe that is one reason search engines such as DuckDuckGo or Marginalia is better for searching blog posts.