How Search Engines Work

The website that ranked #1 for five years — then disappeared overnight

(The following scenario is illustrative — specific traffic, ranking, and revenue figures are reconstructed for teaching purposes and have not been independently verified.)

In 2018, a travel blog called Wandering Earl was ranking #1 for "budget travel tips" — a phrase searched approximately 40,000 times per month. The site had been there for years. Traffic was the founder's livelihood.

Then in March 2019, Google rolled out a core algorithm update. Within two weeks, the site dropped to page 4. Traffic fell an estimated 65%. Revenue roughly halved.

Earl had done nothing wrong. He hadn't cheated or spammed. But he'd built his business on a platform he didn't control, and he didn't fully understand the rules of that platform. He couldn't diagnose what happened because he didn't know how the system worked.

Understanding how search engines work isn't just background knowledge. It's the difference between building an SEO strategy that lasts and building one that collapses with the next algorithm update.

What a search engine is actually doing

Google processes over 8.5 billion searches per day (estimated — Google does not publish official daily search counts; Internet Live Stats, 2024–2025). For each one, it returns a ranked list of results in under a second. To do this at scale, Google has built three interconnected systems:

Crawling: Google's bots (called Googlebots or spiders) follow links across the internet. They visit pages, read their content, and follow the links on those pages to discover more pages. If no other page links to your page — Google may never find it.

Indexing: When a Googlebot visits your page, it tries to understand what the page is about. It reads the text, analyses the HTML structure, looks at the images, and evaluates the page's signals. If the page is deemed indexable (not blocked, not duplicate, not thin content), it gets added to Google's index.

Ranking: When you type a query, Google doesn't search the live web — it searches its index. From the hundreds of millions of potentially relevant results, it ranks them based on hundreds of signals to return the most useful answer for your specific query.

The three ranking pillars

Google's ranking algorithm considers hundreds of signals, but they cluster into three core pillars:

Pillar 1: Relevance Does this page actually answer the query? Google's job is to match intent — the underlying goal of what someone searched for — with content that satisfies it. A page about "how to tie a bowline knot" should rank for that query; a page about knot-related gifts shouldn't.

Google assesses relevance through:

• The words on the page (including headers, body text, alt text)
• The page's title and meta description
• How the content is structured (headers signal the organisation of information)
• Semantic understanding — Google understands meaning, not just keywords

Pillar 2: Authority Is this a trustworthy source? Google uses links from other websites as signals of credibility. When a reputable site links to your page, it's a vote of confidence — a signal that your content is trustworthy enough to reference.

This is called PageRank — the foundational concept behind Google's original algorithm. A page linked to by The New York Times and Harvard has more authority than a page with no links.

Pillar 3: Experience Is this page good to use? Google has expanded its quality signals in recent years to include:

• Page load speed
• Mobile-friendliness
• Core Web Vitals (measuring perceived page responsiveness)
• HTTPS security
• Absence of intrusive interstitials (pop-ups that block content)

A relevant, authoritative page that loads slowly on mobile will be outranked by a slightly less authoritative page that offers a great user experience.

There Are No Dumb Questions

"Does Google rank every page on the internet?"

No — Google's index doesn't contain the entire web. Pages can be excluded from the index intentionally (via a robots.txt file or a "noindex" tag) or unintentionally (if Google hasn't found them yet, or if they have quality problems that make Google decide not to index them). Roughly 5–10% of the pages Google crawls are not indexed — and for many new websites, that number is higher.

"How often does Google update its rankings?"

Rankings change constantly — Google makes thousands of small adjustments to its algorithm each year. Major "core updates" happen a few times per year and can cause significant ranking shifts. The goal is to understand the principles behind the algorithm, not to chase the latest update. Principles persist; tactics shift.

Search intent: the concept that changes everything

The most important insight in all of SEO is not about keywords or links. It's about search intent — understanding why someone types a query, not just what they type.

Google classifies searches into four intent categories:

Why this matters for SEO: If someone searches "best running shoes for flat feet," they want a curated list with recommendations — not a technical article about shoe anatomy. If you publish the technical article for that keyword, Google won't rank it because it doesn't match what the searcher actually wants.

Before writing any content optimised for a keyword, Google the keyword and study the top results: what format do they use? What does the content structure look like? What is the user actually trying to accomplish? That's your intent signal — and your content must satisfy it.

E-E-A-T: what Google looks for in quality content

Google evaluates content quality through a framework it calls E-E-A-T:

• Experience: Has the author actually done or experienced the thing they're writing about?
• Expertise: Does the author have deep knowledge of the topic?
• Authoritativeness: Is the author or site a recognised authority in the field?
• Trustworthiness: Is the information accurate, honest, and safe?

E-E-A-T is especially important for YMYL (Your Money or Your Life) topics — health, finance, legal, and safety content where poor information can cause real harm. Google holds these pages to a higher standard.

What E-E-A-T means practically:

• Author bios matter (and should mention relevant credentials)
• First-hand experience in content matters ("I tried this" beats "experts recommend")
• Citations and sources matter
• Site reputation (other sites linking to you, mentions in reputable publications) matters
• Accuracy and up-to-date information matters

AI-generated content without human expertise added fails E-E-A-T. Google is good at detecting generic, thin content — and it deprioritises it.

Using AI for SEO research: Before writing any page, prompt Claude: "I'm writing a page targeting the search query '[query]'. What is the search intent? Who is searching for this and what specific problem are they trying to solve? What would the ideal page look like to fully satisfy this intent? What expertise or experience signals should I demonstrate to rank well for this query?"

How Google discovers new content

If you publish a page and wait for Google to find it — you may be waiting months. Understanding how Google discovers content helps you accelerate it.

The discovery pathways:

Internal links: The most reliable way to ensure Google finds a new page is to link to it from existing pages that Google already crawls. A new blog post linked from your homepage or an existing popular post will be found within days.

Sitemaps: An XML sitemap is a file that lists all your important pages. Submitting it to Google Search Console tells Google "these are the pages I want you to crawl." Most CMS platforms generate sitemaps automatically.

Google Search Console: Google's free tool for site owners. Submit your sitemap here. Request indexing of individual URLs. See which of your pages are in Google's index, which have errors, and what queries they rank for. Every site should have Search Console set up from day one.

External links: When a well-crawled external site links to your new page, Googlebot follows that link and discovers your page.

Using AI to understand your indexing status: If you have access to Search Console data, paste it into Claude: "Here is my Google Search Console coverage report: [paste data]. Which pages are failing to index, and what are the most common reasons? What should I prioritise fixing first?"

Back to Wandering Earl

He didn't cheat. He didn't spam. But when Google's understanding of search intent for "budget travel tips" shifted — toward more recent content, toward different formats, toward sources with stronger authority signals — his old rankings couldn't hold.

He now understands why. The pages that replaced him weren't just more recent; they matched the evolved intent better and had stronger link authority in the current landscape. That's not a bug in the algorithm. That's the algorithm doing exactly what it's designed to do.

Understanding this doesn't bring the traffic back overnight. But it tells him exactly what to fix — and what to build differently the next time. The same knowledge you now have.

Key takeaways

• Search engines work in three stages: Crawling (discovering pages), indexing (understanding and storing them), ranking (retrieving and ordering them for queries). A page that isn't crawled and indexed cannot rank.
• Three ranking pillars: Relevance (does this match the query?), authority (is this a trustworthy source?), experience (is this page good to use?). All three matter.
• Search intent is the master concept. Google ranks the content that best satisfies the underlying goal of the search — not the content that uses keywords most frequently. Understand the intent before writing anything.
• E-E-A-T is Google's quality framework. Experience, expertise, authoritativeness, trustworthiness. Content that lacks genuine expertise or first-hand experience is at a structural disadvantage.
• Google Search Console is your most important free tool. Set it up from day one, submit your sitemap, and use it to understand what Google sees when it looks at your site.