AI Crawler Policy Scanner - robots.txt, llms.txt and Bot Rules
Audit public robots.txt, llms.txt, sitemap, and policy pages for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and AI access rules.
5%
users
Apify builder
Sean J Public-data APIs for security risk, SEO/AI visibility, compliance evidence, and lead intelligence on Apify.
Leaderboard position
out of 10,354
New actors
#2,385
Total actors
#771
↑ +171
Monthly users
#1,454
↓ -106
Total users
#1,992
↑ +22
Monthly runs
#1,667
↓ -303
Total runs
#1,784
↓ -116
Portfolio stats
New actors
1
Total actors
25
↑ +12
Monthly users
20
↑ +5
Total users
69
↑ +21
Monthly runs
467
↑ +26
Total runs
2,723
↑ +508
New actors
Each bar shows one day in the last 30 days.
Portfolio concentration
16 of 25 actors cover 80%+ of monthly users.
Audit public robots.txt, llms.txt, sitemap, and policy pages for GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, and AI access rules.
5%
users
Discover public API documentation, OpenAPI or Swagger files, REST and GraphQL signals, auth schemes, SDKs, rate limits, and webhook evidence.
5%
users
Find competitor-only referring domains for backlink outreach and SEO authority gap analysis.
5%
users
Extract role-like job links and hiring signals from public company career pages. Return source URLs, engineering, sales, remote-work and ATS mentions for manual review.
5%
users
Export competitor Google ads creatives by advertiser domain from available Ads Transparency records. Review formats, shown dates and preview/source links. Up to 40 creatives per $0.04 snapshot; paid Apify plan required.
5%
users
Detect cookie consent platforms, marketing pixels, analytics tags, and cookie policy evidence from public website HTML and headers.
5%
users
Find CQC care home inspection reports and homecare report links in England. Look up known service IDs or filter a dated ratings snapshot; export provider-grouped CSV. $0.01/provider plus Apify usage. Independent metadata tool; no clinical findings.
5%
users
Extract legal company identity, VAT IDs, registry numbers, addresses, and public contacts from EU and UK website legal pages.
5%
users
Track Google AI Overview visibility, AI SERP citations, GEO visibility, top-10 rankings, and competitor share of voice for target domains using DataForSEO.
5%
users
Review Google Play keyword search results, competing apps, ratings, reviews and prices for ASO research. Export JSON/CSV with heuristic competition signals. Data access included; paid Apify plan required. Does not cover Apple App Store.
5%
users
Mine Google reviews for local-business complaints, objection themes, low-star risk signals, and sales or reputation opportunities.
5%
users
Extract public newsroom, press-release, media-kit, PR contact, and announcement signals from company websites.
5%
users
Scan public website privacy pages for GDPR, CCPA, DSAR, retention, transfer, DPO, and cookie policy evidence with source-linked snippets for manual review.
5%
users
Extract public SaaS pricing-page signals, price points, plan names, trial language, per-seat billing, usage pricing, and enterprise CTAs.
5%
users
Public data due diligence API for SEC filing red flags including restatements, late filings, restructuring items, executive changes, and clustered ownership filings.
5%
users
Check public security.txt files and vulnerability disclosure pages for contacts, expiry, policy, encryption, and bug bounty platform evidence.
5%
users
ActorHunt MCP
ActorHunt gives paid members MCP access to keyword data, actor evidence, and Store movement for deciding what to build next.