Sending too many requests too fast triggers blocks, bans, and wasted CAPTCHA solves. Smart rate limiting keeps your automation running long-term.
Why Rate Limiting Matters
| Problem | Cause | Impact |
|---|---|---|
| IP blocked | Too many requests/minute | Automation stops |
| CAPTCHA difficulty increases | High-frequency patterns detected | Slower solves |
| Token rejected after solve | Site flagged the session | Wasted money |
| Account banned | Abnormal behavior patterns | Permanent block |
Token Bucket Rate Limiter
import time
import threading
class TokenBucketLimiter:
"""Rate limiter using token bucket algorithm."""
def __init__(self, rate, burst=1):
"""
Args:
rate: Requests per second allowed.
burst: Maximum burst size.
"""
self.rate = rate
self.burst = burst
self.tokens = burst
self.last_refill = time.monotonic()
self._lock = threading.Lock()
def acquire(self, timeout=None):
"""Wait until a token is available."""
deadline = time.monotonic() + timeout if timeout else None
while True:
with self._lock:
self._refill()
if self.tokens >= 1:
self.tokens -= 1
return True
if deadline and time.monotonic() >= deadline:
return False
# Wait for next token
with self._lock:
wait_time = (1 - self.tokens) / self.rate
time.sleep(min(wait_time, 0.1))
def _refill(self):
"""Add tokens based on elapsed time."""
now = time.monotonic()
elapsed = now - self.last_refill
self.last_refill = now
self.tokens = min(self.burst, self.tokens + elapsed * self.rate)
# 2 requests per second, burst of 5
limiter = TokenBucketLimiter(rate=2, burst=5)
def rate_limited_solve(solver, params):
"""Solve CAPTCHA with rate limiting."""
limiter.acquire()
return solver.solve(params)
Per-Domain Rate Limiter
Different sites need different rates:
from collections import defaultdict
from urllib.parse import urlparse
class DomainRateLimiter:
"""Apply different rate limits per target domain."""
DEFAULT_DELAY = 5.0 # seconds between requests to same domain
DOMAIN_DELAYS = {
"example.com": 3.0,
"api.example.com": 1.0,
}
def __init__(self):
self._last_request = defaultdict(float)
self._lock = threading.Lock()
def wait(self, url):
"""Wait appropriate time before requesting this domain."""
domain = urlparse(url).netloc
delay = self.DOMAIN_DELAYS.get(domain, self.DEFAULT_DELAY)
with self._lock:
elapsed = time.time() - self._last_request[domain]
if elapsed < delay:
sleep_time = delay - elapsed
time.sleep(sleep_time)
self._last_request[domain] = time.time()
domain_limiter = DomainRateLimiter()
def safe_request(url, **kwargs):
"""Make rate-limited request."""
domain_limiter.wait(url)
return requests.get(url, **kwargs)
Jittered Delays
Fixed delays create detectable patterns. Add randomness:
import random
import time
def human_delay(base_seconds=3.0, jitter=0.5):
"""Sleep with human-like randomness."""
delay = base_seconds + random.uniform(-jitter * base_seconds, jitter * base_seconds)
time.sleep(max(0.5, delay))
def exponential_backoff_delay(attempt, base=2.0, max_delay=60.0):
"""Exponential backoff with jitter for retries."""
delay = min(base * (2 ** attempt), max_delay)
jittered = delay * random.uniform(0.5, 1.5)
time.sleep(jittered)
Sliding Window Counter
Track request counts over time:
import time
from collections import deque
import threading
class SlidingWindowLimiter:
"""Count requests in a rolling time window."""
def __init__(self, max_requests, window_seconds):
self.max_requests = max_requests
self.window = window_seconds
self.timestamps = deque()
self._lock = threading.Lock()
def can_proceed(self):
"""Check if a new request is allowed."""
with self._lock:
now = time.time()
cutoff = now - self.window
# Remove old timestamps
while self.timestamps and self.timestamps[0] < cutoff:
self.timestamps.popleft()
if len(self.timestamps) < self.max_requests:
self.timestamps.append(now)
return True
return False
def wait_and_proceed(self):
"""Block until request is allowed."""
while not self.can_proceed():
time.sleep(0.5)
# Max 10 CAPTCHA solves per minute
solve_limiter = SlidingWindowLimiter(max_requests=10, window_seconds=60)
def limited_solve(solver, params):
"""Solve with sliding window rate limit."""
solve_limiter.wait_and_proceed()
return solver.solve(params)
Adaptive Rate Limiting
Automatically slow down when detecting problems:
class AdaptiveRateLimiter:
"""Adjust rate based on response signals."""
def __init__(self, initial_delay=3.0, min_delay=1.0, max_delay=30.0):
self.delay = initial_delay
self.min_delay = min_delay
self.max_delay = max_delay
self._lock = threading.Lock()
def report_success(self):
"""Speed up slightly after success."""
with self._lock:
self.delay = max(self.min_delay, self.delay * 0.95)
def report_block(self):
"""Slow down after being blocked."""
with self._lock:
self.delay = min(self.max_delay, self.delay * 2.0)
def report_captcha_harder(self):
"""Slow down when CAPTCHA difficulty increases."""
with self._lock:
self.delay = min(self.max_delay, self.delay * 1.5)
def wait(self):
"""Wait the current delay with jitter."""
with self._lock:
delay = self.delay
jittered = delay * random.uniform(0.8, 1.2)
time.sleep(jittered)
rate = AdaptiveRateLimiter(initial_delay=3.0)
# In your automation loop:
# rate.wait()
# result = make_request()
# if result.status_code == 429:
# rate.report_block()
# elif "captcha" in result.text and not expected:
# rate.report_captcha_harder()
# else:
# rate.report_success()
Recommended Rates by Use Case
| Use Case | Requests/min | Delay Between |
|---|---|---|
| Price monitoring | 5-10 | 6-12 seconds |
| Data collection | 10-20 | 3-6 seconds |
| QA testing | 2-5 | 12-30 seconds |
| Form testing | 1-3 | 20-60 seconds |
FAQ
Does rate limiting increase costs?
No. It actually reduces costs by preventing wasted CAPTCHA solves that would be rejected by the target site.
Should I rate limit CaptchaAI API calls too?
CaptchaAI handles high throughput, but for ERROR_NO_SLOT_AVAILABLE, implement backoff. Rate limiting applies mainly to the target site, not the CaptchaAI API.
How do I know if I'm being detected?
Signs include: more CAPTCHAs appearing, CAPTCHA difficulty increasing, HTTP 429 responses, session termination, or IP blocks.
Related Guides
Build sustainable automation — start with CaptchaAI.
Discussions (0)
Join the conversation
Sign in to share your opinion.
Sign InNo comments yet.