The Three Pillars of Search
Every search engine operates on three fundamental, sequential processes. Mastering these is the non-negotiable foundation of all SEO work. The pipeline is dependent: fail at crawling and you never reach indexing. Fail at indexing and ranking is impossible.
| Stage | Agent | What Happens | Key Signal |
|---|---|---|---|
| Crawling | Googlebot | Discovers & downloads pages | Crawl budget, robots.txt |
| Indexing | Google Indexer | Renders & stores content | Canonical, noindex tags |
| Ranking | RankBrain / MUM | Positions pages for queries | Relevance + E-E-A-T + UX |
Crawling: How Googlebot Works
Googlebot is a web spider that continuously follows links across the internet. It has a crawl budget — a limited number of pages crawled per site per day. Large sites with poor internal linking waste this budget on low-value pages.
- Sitemap.xml — tells Googlebot which URLs exist and when they were last updated
- Internal links — how Googlebot navigates between pages; orphan pages don't get crawled
- Robots.txt — instructs crawlers which paths to skip (does NOT prevent indexing if discovered another way)
Indexing: The Searchable Database
After crawling, Google renders your page (executing JavaScript) and stores the processed content in its index. Verify indexing status with site:yourdomain.com in Google Search, or the URL Inspection tool in Search Console.
noindex meta tag, password protection, thin/duplicate content, soft 404s, and canonical tags pointing elsewhere.Ranking: 200+ Signals Explained
Google evaluates pages against hundreds of ranking factors whenever a relevant query is made. In 2026, the most impactful signal clusters are:
- Relevance — content matches query intent (analyzed by MUM/Gemini models)
- E-E-A-T — Experience, Expertise, Authoritativeness, Trust (critical post-Feb 2026 Core Update)
- Page Experience — Core Web Vitals (LCP, INP, CLS) and mobile usability
- Backlink authority — quality and quantity of external links pointing to you
- User signals — click-through rate, dwell time, task completion
User Intent & the 4 Query Types
Understanding why someone searches is more important than the exact keyword. Google classifies every query by intent — pages that match the dominant intent outrank those that don't, regardless of other optimization.
| Intent Type | Example Query | Best Content Format |
|---|---|---|
| Informational | "how to do keyword research" | In-depth guide, tutorial |
| Navigational | "Ahrefs login" | Homepage / landing page |
| Transactional | "buy SEO tool subscription" | Product / pricing page |
| Commercial | "best SEO tools 2026" | Comparison, review article |
Building a Keyword Strategy
A keyword strategy maps topics to pages, ensuring each URL targets one primary intent. Start with seed keywords and expand to long-tail variations using Google Search Console, Ahrefs, or Semrush.
- Search Volume — monthly searches (higher = more potential, more competition)
- Keyword Difficulty (KD) — competition score 0–100; new sites should target KD <30
- CPC — higher cost-per-click = higher commercial value