Technical foundations: crawling & Core Web Vitals

Lesson 2 of 8 · 30 min · part of the AI SEO & GEO Specialist certification

robots.txt tells crawlers which paths they may fetch. It is a request, not access control: well-behaved bots obey it, but it does not hide content, and a URL blocked in robots.txt can still appear in results if other sites link to it. To keep a page out of the index, allow crawling and use a noindex meta tag — the crawler must be able to fetch the page to see that tag.

A minimal, healthy robots.txt

User-agent: *
Disallow: /cart/
Disallow: /account/

Sitemap: https://www.example.com/sitemap.xml

Interactive exercise: Audit this robots.txt

Core Web Vitals are Google's user-experience metrics, measured on real visitors (field data). They are a ranking signal, though a modest one compared with relevance — but slow pages also lose visitors and conversions.

Interactive exercise: Match each Core Web Vital to what it measures