🔍 SEO / GEO

Crawling

crawling · web crawling · site crawl · crawler · spidering · search bot crawl

Definition

Crawling is the automated process in which a search robot walks the pages of a site through hyperlinks in order to collect content for subsequent indexation.

How crawling works

A search bot — Googlebot, Bingbot or another crawler — starts from a set of known URLs, a kind of
seed list. Moving from page to page through links, it downloads the HTML, records status codes
(200, 301, 404, 500) and hands the content to the indexing system. That whole process is crawling.

Crawl rate is self-regulating: the bot watches how fast the server responds and lowers its request
frequency when the site is slow. Google does this automatically. Yandex, the dominant search engine
in Russia and several CIS markets, additionally allows the crawl rate to be set by hand in its
webmaster console.

What drives crawl efficiency in e-commerce

A large online store has potentially millions of URLs — products, filters, sort orders, pagination,
parameter combinations. Left unmanaged, the bot scatters its budget across low-value pages and
never gets round to new products or important categories.

The main factors:

Factor	Effect on crawling
Server response time	A poor TTFB slows the walk and shrinks the budget
Redirect chains	Every redirect costs budget; three or more in a row may be ignored
Internal linking	Pages with no inbound links are effectively invisible to the bot
XML sitemap	Helps bots find new and updated URLs faster
Parameter URLs	Thousands of ?sort=price&color=red create duplicates — close them with robots.txt or canonical

Managing the crawl zone

Not every page needs to be scanned. Keep these out of the crawl:

Filter and sort pages (?sort=, ?page=, ?color=)
Basket, account area, checkout
Technical endpoints (/api/, /admin/)
Duplicate versions of content (print views, AMP twins)

The instruments: robots.txt (a ban at the URL-pattern level), the noindex meta tag (allow the
scan, forbid the index entry) and canonical (a signal about the preferred URL when duplicates
exist).

Common problems

The bot falls into a trap. Dynamically generated pages — on-site search results, infinite
pagination — can spawn hundreds of thousands of URLs. Close them in robots.txt.

JavaScript content is invisible. Recommendation widgets, prices and descriptions that load
through JavaScript after page initialisation reach the bot late, or never. Business-critical
content belongs in the initial HTML response.

New product pages are not crawled in time. The fix: add the <url> entry to the sitemap with a
<lastmod> value when a product is published — that is a priority signal for the bot.

Important: crawling is not indexation. A crawled page may still fail to enter the index if its
content is judged duplicate or low quality. Check the status of specific URLs in Google Search
Console, and in Yandex Webmaster if the Russian-language market matters to you.

Frequently asked questions

How does crawling differ from indexation?

Crawling is the bot collecting content. Indexation is the processing of that content and the addition of pages to the search database. Crawling comes first but guarantees nothing — a page can be crawled and still never be indexed, because of a noindex tag or low content quality.

What is crawl budget and why does it matter in e-commerce?

Crawl budget is the number of pages a bot is willing to walk per unit of time. For a large online store with millions of SKUs it is critical: if the bot spends its budget on filters, parameter URLs and duplicates, it may never reach new products. Manage the budget through robots.txt, canonical tags and sensible internal linking.

How do you speed up crawling?

A fast server response (TTFB under 200 ms), a simple link structure without deep nesting, a minimum of redirect chains and a correct XML sitemap all help. Remove parameter and duplicate URLs from the crawl zone with robots.txt or noindex tags.

Can JavaScript rendering and SPAs hurt crawling?

Yes. If content loads through JavaScript after the initial paint, the bot may not wait for it and may record an empty page. Googlebot can render JavaScript, but it does so in a queue, with delays of days or weeks. Business-critical content belongs in the initial HTML.