You Probably Do Not Have a Crawl Budget Problem (Here Is What You Actually Have)

For a website under roughly 10,000 pages, crawl budget is almost never your problem. Yet the hours you spend chasing it are stolen from the one fix that would actually get your pages indexed.

If you have read a few technical SEO guides, you have probably met the crawl budget panic. Someone explains that Googlebot only crawls so many pages per site, that a bloated sitemap or slow server "wastes" your allocation, and that your unindexed pages are starving. It sounds authoritative. It sends people down a rabbit hole of log-file analysis and robots.txt surgery. And for the vast majority of small and mid-sized sites, it is the wrong diagnosis entirely.

What crawl budget actually is

Crawl budget is the number of URLs Googlebot can and wants to crawl on your site in a given window. Google's own documentation describes it as a combination of crawl capacity (how much your server can handle without slowing down) and crawl demand (how much Google wants your content). That is real. It is also, by Google's own framing, a concern for very large sites.

Here is where the numbers matter, and where you should hold them loosely:

• Google documents crawl budget as something to think about for sites with more than 10,000 URLs, or sites that change extremely frequently.
• Plenty of experienced SEOs go much further and treat crawl budget as a non-issue until a site approaches roughly 1 million pages.

Treat both as generalizations to verify against your own logs, not laws carved in stone. But notice the scale. If your site has 400 pages, or 4,000, you are not in the neighborhood where crawl budget becomes the bottleneck. Googlebot is not rationing visits to your blog.

The 400-page site that "had a crawl budget problem"

Picture a common scenario. Someone runs a 400-page site. A chunk of their pages will not show up in search, so they go looking for a cause. They land on a crawl budget guide, recognize the symptoms in the vaguest possible way, and conclude that Google simply cannot get to their pages.

So they optimize. They prune the sitemap, block a few parameter URLs, obsess over server response time, maybe hire someone to run a log-file audit. Weeks pass. Nothing changes.

The tell was in Search Console the whole time. Open the Pages report and the pages are marked as crawled. They sit in Discovered, currently not indexed or Crawled, currently not indexed. Read those statuses literally. Google found the page. In the "crawled" case, Google fetched it and read it. Crawl budget was spent successfully. Google then chose not to index the page. That is not a capacity failure. That is a judgment: the page did not clear the bar.

Is it really crawl budget? The 10-second flowchart

Before you touch a single robots.txt line, run this:

1. Do you have more than 10,000 URLs you actually want indexed? If no, stop. It is not crawl budget. Go to step 3.
2. If yes, are your important pages genuinely not being crawled at all (not just crawled-and-not-indexed)? If they are being crawled, it is still not a budget problem. Go to step 3.
3. Check the two levers that actually gate indexing on small sites: page value and internal links.

Almost everyone exits at step 1. That is the point. The flowchart is short because the answer, for most sites, is short.

The two levers that actually decide indexing

When Google crawls a page and declines to index it, it is making a bet about whether the page is worth storing and serving. Two things move that bet on a small site.

Lever one: perceived page value. Google is asking whether this page adds something a searcher would want that is not already covered better elsewhere, including elsewhere on your own site. Thin pages, near-duplicate pages, templated pages with a few words swapped, and pages that restate what ten other results already say are the ones that get crawled and left in limbo. The fix is not technical. It is making the page genuinely worth indexing: real depth, a specific answer, a reason to exist.

Lever two: internal links. Internal links are how you tell Google which of your pages matter. A page buried with no links pointing to it, or reachable only through a sitemap, reads as low priority. Pages that important, well-linked pages point to inherit that importance. If your unindexed pages are internal-link orphans, that is often the whole story. Link to them from relevant, established pages, using descriptive anchor text, and give Google a reason to treat them as part of the site rather than a stray URL.

That is the entire checklist for most sites:

• Page value: Is this page genuinely worth indexing, or is it thin, duplicative, or redundant with your own content? Improve it or consolidate it.
• Internal links: Do meaningful pages link to it with clear anchor text, or is it an orphan? Wire it into the site.

Fix those two levers and you address the actual cause of the "Discovered" and "Crawled, currently not indexed" statuses that crawl budget gets wrongly blamed for.

The trap that outlasts the fix

Here is the subtle part, and the reason indexing is not a one-time job. A page can get indexed, sit happily in results for weeks, and then silently fall out. Google reassesses quality over time. A wave of new similar pages can dilute it. Internal links that once pointed to it can disappear in a redesign. When it drops, nothing announces it. Your rankings for that page just quietly go to zero.

You will not catch that by checking once and moving on. You catch it by re-checking index status on a schedule and watching the trend, so a page that was indexed in March and vanished in May shows up as a change you can see rather than a mystery traffic dip. A running index history timeline turns "is this page still indexed?" from a manual chore into something you glance at.

The reframe

So the next time you feel the pull to optimize crawl budget, ask the honest question first. Do you have more than 10,000 URLs, and are your pages actually not being crawled? If not, close the crawl budget guide. Your pages are being seen. They are being judged and, for now, found wanting.

That is better news than a crawl budget problem, because the fix is in your hands. Make the page worth indexing. Link to it like it matters. Then keep an eye on whether it stays indexed, because getting in is only half the job. Staying in is the part almost nobody watches.