Inconsistent or outdated user-agent strings trigger bot detection and make CAPTCHAs harder. Here's how to manage them properly.
Why User-Agent Matters
| Issue | Result |
|---|---|
Default Python UA (python-requests/2.31) |
Instant bot detection |
| Outdated Chrome UA (v90) | Flagged as suspicious |
| UA changes between requests | Session invalidated |
| UA doesn't match browser capabilities | Detection algorithms triggered |
Modern User-Agent Pool
import random
# Updated Chrome UAs (keep current)
CHROME_UAS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36",
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36",
]
FIREFOX_UAS = [
"Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:126.0) Gecko/20100101 Firefox/126.0",
"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:126.0) Gecko/20100101 Firefox/126.0",
"Mozilla/5.0 (X11; Linux x86_64; rv:126.0) Gecko/20100101 Firefox/126.0",
]
def get_random_ua(browser="chrome"):
"""Get a random modern user-agent."""
if browser == "firefox":
return random.choice(FIREFOX_UAS)
return random.choice(CHROME_UAS)
Session-Consistent UA
Use the same UA throughout a session:
import requests
class ConsistentSession:
"""Session that maintains consistent fingerprint."""
def __init__(self, user_agent=None):
self.session = requests.Session()
self.user_agent = user_agent or get_random_ua()
self.session.headers.update({
"User-Agent": self.user_agent,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
})
def get(self, url, **kwargs):
return self.session.get(url, **kwargs)
def post(self, url, **kwargs):
return self.session.post(url, **kwargs)
# Same UA for all requests in this session
session = ConsistentSession()
session.get("https://example.com/page1")
session.post("https://example.com/form")
UA Matching with CaptchaAI
Send the same UA to CaptchaAI that you use for browsing:
def solve_with_matching_ua(api_key, sitekey, pageurl, user_agent):
"""Send matching user-agent to CaptchaAI."""
resp = requests.post("https://ocr.captchaai.com/in.php", data={
"key": api_key,
"method": "userrecaptcha",
"googlekey": sitekey,
"pageurl": pageurl,
"userAgent": user_agent, # Match your browser's UA
"json": 1,
}, timeout=30)
return resp.json()
# Use the same UA everywhere
ua = get_random_ua()
session = ConsistentSession(user_agent=ua)
# Browse with this UA
session.get("https://example.com")
# Solve CAPTCHA with same UA
solve_with_matching_ua("YOUR_API_KEY", "SITEKEY", "https://example.com", ua)
Selenium UA Management
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
def create_driver_with_ua(user_agent):
"""Create Selenium driver with specific user-agent."""
options = Options()
options.add_argument(f"user-agent={user_agent}")
options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(options=options)
# Verify UA is set correctly
actual_ua = driver.execute_script("return navigator.userAgent")
assert user_agent in actual_ua, f"UA mismatch: {actual_ua}"
return driver
# Use same UA for browser and API
ua = get_random_ua()
driver = create_driver_with_ua(ua)
# ... solve with captchaai using same ua ...
Complete Headers for Realistic Requests
def get_realistic_headers(user_agent=None):
"""Build headers that match a real browser."""
ua = user_agent or get_random_ua()
headers = {
"User-Agent": ua,
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8",
"Accept-Language": "en-US,en;q=0.9",
"Accept-Encoding": "gzip, deflate, br",
"Connection": "keep-alive",
"Upgrade-Insecure-Requests": "1",
"Sec-Fetch-Dest": "document",
"Sec-Fetch-Mode": "navigate",
"Sec-Fetch-Site": "none",
"Sec-Fetch-User": "?1",
"Cache-Control": "max-age=0",
}
# Add Chrome-specific headers
if "Chrome" in ua:
headers["sec-ch-ua"] = '"Chromium";v="125", "Google Chrome";v="125", "Not.A/Brand";v="24"'
headers["sec-ch-ua-mobile"] = "?0"
headers["sec-ch-ua-platform"] = '"Windows"'
return headers
FAQ
Does user-agent affect CAPTCHA difficulty?
Yes. reCAPTCHA and Cloudflare Turnstile analyze user-agent strings. Outdated, missing, or bot-like UAs trigger harder challenges or higher block rates.
How often should I update my UA list?
Update every 1-2 months when new browser versions release. Outdated UAs stand out in detection systems.
Should I use the same UA for all requests?
Within a session, yes. Across different sessions or accounts, rotate UAs to avoid creating a trackable pattern.
Related Guides
Build smarter automation — start with CaptchaAI.
Discussions (0)
Join the conversation
Sign in to share your opinion.
Sign InNo comments yet.