What is a User-Agent String? Definition & SEO Use
Definition
The User-Agent is an HTTP request header that identifies the client making a request — typically a browser, a bot, or a custom HTTP client. Servers can read this header to serve different content based on the visitor: a desktop layout vs mobile, a browser vs a crawler, a chatbot vs a real user.
Common user agents
- Googlebot — Google's web crawler (smartphone variant by default since mobile-first indexing).
- Bingbot — Microsoft's crawler.
- GPTBot, CCBot, ClaudeBot — AI training and retrieval crawlers.
- Browsers — Chrome, Firefox, Safari, etc., each with distinctive UA strings.
- Custom bots — uptime monitors, security scanners, link checkers.
SEO and security considerations
- Cloaking — serving different content to Googlebot than to users is against guidelines and can trigger manual penalties.
- Bot blocking — blocking Googlebot in robots.txt or via firewall rules will deindex your site.
- Spoofing — bots can claim any user-agent string. Verify Googlebot via reverse DNS lookup, not the UA alone.
- Adaptive serving — serving distinct HTML to mobile vs desktop is allowed but requires correct canonicalization.
Treat user-agent as a hint, not a source of truth. For SEO, ensure your important content is served identically to crawlers and users — and audit regularly with a free site scan.
Related terms
robots.txt
The file that tells crawlers which parts of your site they can access.
404 Error
What happens when a page can't be found — and how it affects SEO.
301 Redirect
The permanent redirect that preserves link equity and guides users.
Crawl Budget
How search engines allocate resources to crawl your site — and why broken links waste it.
Broken Link
A hyperlink that no longer leads to its intended destination.
Redirect Chain
Multiple redirects in sequence that slow down pages and dilute SEO value.