AI Crawler Checker: GPTBot, robots.txt & llms.txt Policy Audit
Check which AI crawlers each website blocks in robots.txt (GPTBot, ClaudeBot, Google-Extended, PerplexityBot), plus llms.txt and TDM opt-out signals.
10%
users
Apify builder
Ad Astra Connect Production-grade job market, e-commerce, and compliance data Actors with built-in change detection. Built by Ad Astra Connect (Toronto, Canada).
Leaderboard position
out of 10,354
New actors
#621
Total actors
#1,603
↑ +2,203
Monthly users
#2,155
↑ +2,005
Total users
#3,423
↑ +2,252
Monthly runs
#3,092
↑ +4,844
Total runs
#5,654
↑ +2,611
Portfolio stats
New actors
10
Total actors
10
↑ +8
Monthly users
10
↑ +8
Total users
20
↑ +16
Monthly runs
135
↑ +133
Total runs
205
↑ +189
New actors
Each bar shows one day in the last 30 days.
Portfolio concentration
8 of 10 actors cover 80%+ of monthly users.
Check which AI crawlers each website blocks in robots.txt (GPTBot, ClaudeBot, Google-Extended, PerplexityBot), plus llms.txt and TDM opt-out signals.
10%
users
A domain expiration monitor for domain expiry, registrar and DNSSEC data using RDAP, the official replacement for WHOIS domain lookup.
10%
users
Export all jobs from any company's Greenhouse, Lever, Ashby, Workable or Recruitee board: titles, locations, salary ranges, apply links. Get only new job postings on scheduled runs.
10%
users
Track new MCP servers and changed versions, packages and remote endpoints from the official public MCP registry API. Export an MCP server list or schedule change-only monitoring.
10%
users
Audit job postings for salary ranges and pay transparency law signals by location (US states, cities, Canadian provinces). Greenhouse, Lever, Ashby, Workable, Recruitee. Not legal advice.
10%
users
Monitor official consumer product recalls via CPSC, Health Canada and UK OPSS recall API feeds with brand, keyword, GTIN filters and new recall alerts.
10%
users
Scrape remote jobs from Jobicy and WeWorkRemotely (We Work Remotely) with salary data, source attribution, deduplication and new job alerts.
10%
users
Scrape any Shopify or WooCommerce store's products, variants, prices, discounts and stock. Monitor competitor price drops, restocks and new products on a schedule. No login or proxy.
10%
users
ActorHunt MCP
ActorHunt gives paid members MCP access to keyword data, actor evidence, and Store movement for deciding what to build next.