Can we talk about how many people still don't check their crawl budget?

I've been auditing sites for about a year now, and I'd say 7 out of 10 have major crawl budget waste. I saw a site last week with over 2,000 low-value pagination pages being indexed. The owner had no idea Googlebot was spending time there instead of on their new product pages. It's a huge deal because it slows down how fast your good content gets found. How do you guys usually spot and fix this kind of thing first?

4 comments

4 Comments

gray31423d ago

Check the indexed pages report in Search Console first. It shows you exactly what Google is wasting time on. Then you can noindex those pagination pages fast.

kevin_lane23d ago

Start with a site audit tool to find low value pages. Then use robots.txt or the noindex tag to block them from search bots.

skyler2175d ago

Kevin's robots.txt advice for indexing is like using a stop sign to hide a secret.

finleym3723d ago

Robots.txt blocks crawling, not indexing.