aicost crawler health

STALE
📈 Price changes (7d)
0
across all crawlers
🆕 New models (7d)
54
need review / catalog
🧹 Sanity catches (7d)
0
crawler self-heal working
❌ Failures (7d)
0
crawl or extraction errors

1. Token API Pricing Crawler

17 LLM/embedding/TTS API vendors · per-token rates · manually-curated source of truth at lib/pricing-data.js (crawler reads this for diff, does not auto-write)
AUTOMATED DAILY · 03:00 EDT
SourceLast modifiedVendors coveredStatus
lib/pricing-data.js 7d ago anthropic, openai, google, deepseek, xai, mistral, cohere, together, alibaba, minimax, moonshot, nvidia, stepfun, xiaomi, z-ai, tencent, meta ✓ clean

2. Product / Subscription Pricing Crawler

14 consumer/team plan vendors · seat tiers · slug stored as product:vendor
AUTOMATED DAILY · 03:00 EDT
VendorLast runModelsChangesNew🧹Status
anthropic3d ago1111⚠ review
openai3d ago33⚠ review
google141d ago44⚠ review
google-ai3d ago44⚠ review
google-workspace3d ago44⚠ review
github3d ago44⚠ review
cursor3d ago66⚠ review
windsurf3d ago⚠ partial
perplexity3d ago22⚠ review
midjourney3d ago55⚠ review
runway3d ago44⚠ review
elevenlabs3d ago77⚠ review
replit3d ago⚠ partial
vercel3d ago44⚠ review

3. Hybrid Plan Pricing Crawler

16 seat+overage vendors · slug stored as hybrid:vendor · CF /markdown + Gemini extract
AUTOMATED DAILY · 03:00 EDT
VendorLast runPlansChangesNew🧹Status
anthropic131d ago731⚠ partial
aws131d ago55⚠ partial
cursornever⏸ pending
github131d ago3⚠ partial
google131d ago733⚠ partial
hubspotnever⏸ pending
microsoft131d ago53⚠ partial
mistral131d ago22⚠ partial
notionnever⏸ pending
openai131d ago714⚠ review
salesforcenever⏸ pending
selfhostnever⏸ pending
servicenownever⏸ pending
slacknever⏸ pending
together131d ago44⚠ partial
windsurfnever⏸ pending

4. Manual Maintenance

1 vendor(s) blocked from auto-scrape
MANUAL QUARTERLY · OR ON ANNOUNCEMENTS
VendorLast verifiedDays sincePlansNext dueStatusProcedure
microsoft
Microsoft 365 / Copilot / Sales Copilot
2026-05-01133d72026-08-01! staledocs/MICROSOFT-MANUAL-UPDATES.md

5. Benchmark Crawler

10 mechanism topics · prompt-cache rates, batch discounts, latencies · writes aicost_benchmarks table
AUTOMATED WEEKLY · WED 03:00 EDT
TopicLast verifiedTopicsNeeds reviewStatus
anthropic-caching
Anthropic Prompt Caching
2d ago 6 0000000 ✓ clean
anthropic-batch
Anthropic Message Batches
2d ago 2 000 ✓ clean
openai-batch
OpenAI Batch API
2d ago 2 000 ✓ clean
openai-caching
OpenAI Prompt Caching
2d ago 2 000 ✓ clean
google-gemini
Google Gemini
2d ago 3 0000 ✓ clean
google-vertex
Google Vertex AI
2d ago 1 00 ✓ clean
aws-bedrock
AWS Bedrock
2d ago 1 00 ✓ clean
mistral
Mistral
2d ago 1 00 ✓ clean
pinecone
Pinecone
2d ago 2 000 ✓ clean
cohere
Cohere
2d ago 2 000 ✓ clean

📋 TLDR — what needs attention 🔧 Troubleshooting playbook →

🚨 Falling behind (9)

cursor · Hybrid Plan Pricing
Never run
hubspot · Hybrid Plan Pricing
Never run
notion · Hybrid Plan Pricing
Never run
salesforce · Hybrid Plan Pricing
Never run
selfhost · Hybrid Plan Pricing
Never run
servicenow · Hybrid Plan Pricing
Never run
slack · Hybrid Plan Pricing
Never run
windsurf · Hybrid Plan Pricing
Never run
microsoft · Manual Maintenance
Audit overdue by 43d

✅ Healthy (33)

pricing-data.js (17 vendors) · anthropic (product/subscription) · openai (product/subscription) · google (product/subscription) · google-ai (product/subscription) · google-workspace (product/subscription) · github (product/subscription) · cursor (product/subscription) · windsurf (product/subscription) · perplexity (product/subscription) · midjourney (product/subscription) · runway (product/subscription) · elevenlabs (product/subscription) · replit (product/subscription) · vercel (product/subscription) · anthropic (hybrid) · aws (hybrid) · github (hybrid) · google (hybrid) · microsoft (hybrid) · mistral (hybrid) · openai (hybrid) · together (hybrid) · anthropic-caching (benchmark) · anthropic-batch (benchmark) · openai-batch (benchmark) · openai-caching (benchmark) · google-gemini (benchmark) · google-vertex (benchmark) · aws-bedrock (benchmark) · mistral (benchmark) · pinecone (benchmark) · cohere (benchmark)

🛠 Manual run commands

Copy-paste reference for triggering crawlers manually. Use these when scheduled jobs fail, after vendor announcements, or for ad-hoc data refreshes. Replace <slug> with the vendor slug. Add --dry-run to preview without DB writes.

🔵 TOKEN API PRICING (detects vendor changes — pricing-data.js is manually-curated)
:: All vendors (full nightly run)
node scripts\aicost-pricing-crawler.js > D:\aicost\logs\pricing-now.log 2>&1

:: Single vendor
node scripts\aicost-pricing-crawler.js --vendor=anthropic

:: Dry run (preview, no DB writes)
node scripts\aicost-pricing-crawler.js --vendor=openai --dry-run
🔴 PRODUCT / SUBSCRIPTION PRICING (writes product:* rows)
:: All vendors (full weekly run)
node scripts\aicost-product-crawler.js > D:\aicost\logs\product-now.log 2>&1

:: Single vendor
node scripts\aicost-product-crawler.js --vendor=cursor

:: Dry run
node scripts\aicost-product-crawler.js --vendor=github --dry-run
🟡 HYBRID PLAN PRICING (writes hybrid:* rows)
:: All vendors (full weekly run)
node lib\aicost-hybrid-crawler.js > D:\aicost\logs\hybrid-now.log 2>&1

:: Single vendor
node lib\aicost-hybrid-crawler.js --vendor=anthropic

:: Dry run / Tier-A only / extract phase only
node lib\aicost-hybrid-crawler.js --vendor=openai --dry-run
node lib\aicost-hybrid-crawler.js --tier=A
node lib\aicost-hybrid-crawler.js --phase=extract
🟢 BENCHMARK CRAWLER (mechanism facts: cache rates, batch discounts, latencies)
:: All 10 topics (full weekly run)
node scripts\aicost-benchmark-crawler.js > D:\aicost\logs\benchmark-now.log 2>&1

:: Single topic
node scripts\aicost-benchmark-crawler.js --vendor=anthropic-caching
node scripts\aicost-benchmark-crawler.js --vendor=openai-batch

:: Dry run / phase-only
node scripts\aicost-benchmark-crawler.js --dry-run
node scripts\aicost-benchmark-crawler.js --phase=crawl
node scripts\aicost-benchmark-crawler.js --phase=extract

:: Available topic slugs:
::   anthropic-caching, anthropic-batch, openai-batch, openai-caching,
::   google-gemini, google-vertex, aws-bedrock, mistral, pinecone, cohere

:: Review flagged rows (Mistral couldn't verify claim from source text)
mysql -u root -p toolsinfo -e "SELECT topic_slug, claim, source_url FROM aicost_benchmarks WHERE needs_review = 1;"
🟣 V2 GUIDE REGENERATION (after pricing changes)
:: Single vendor
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --force

:: Multiple vendors
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --vendor=openai --force

:: All 16 V2 vendors (~$0.30, ~5-7 minutes)
node scripts\generate-vendor-pricing-guides-batch.js --all --force
⚙️ MANUAL MAINTENANCE (Microsoft quarterly audit)
:: Open the procedure doc
notepad D:\aicost\docs\MICROSOFT-MANUAL-UPDATES.md

:: Edit verified plan data with current Microsoft pricing
notepad D:\aicost\lib\hybrid-pricing-data-verified-2026q2.js

:: Update last_verified date in routes/crawler-health.js MANUAL_VENDORS
notepad D:\aicost\routes\crawler-health.js

:: Then regen Microsoft V2
node scripts\generate-vendor-pricing-guides-batch.js --vendor=microsoft --force
📅 SCHEDULED TASK MANAGEMENT
:: Register all scheduled tasks (run as admin)
powershell -ExecutionPolicy Bypass -File D:\aicost\powershell\Run-AicostCrawler.ps1

:: List all aicost scheduled tasks
schtasks /query /tn "aicost*"

:: Manually trigger a scheduled task
schtasks /run /tn "aicost-pricing-crawler"

:: View recent task history (last 24h)
powershell "Get-WinEvent -LogName 'Microsoft-Windows-TaskScheduler/Operational' -MaxEvents 50 | Where-Object {$_.Message -like '*aicost*'}"
🔍 DIAGNOSTIC QUERIES
:: Last run per vendor (all crawlers)
mysql -u root -p toolsinfo -e "SELECT vendor_slug, COUNT(*) AS runs, MAX(started_at) AS last_run FROM aicost_crawler_runs GROUP BY vendor_slug ORDER BY last_run DESC;"

:: Recent price changes (last 7 days)
mysql -u root -p toolsinfo -e "SELECT provider_slug, model_slug, field_name, old_value, new_value, change_pct, detected_at FROM aicost_price_changelog WHERE detected_at >= DATE_SUB(NOW(), INTERVAL 7 DAY) ORDER BY detected_at DESC LIMIT 20;"

:: V2 guide refresh status
mysql -u root -p toolsinfo -e "SELECT slug, schema_version, last_refreshed_at FROM aicost_guides WHERE type='vendor-pricing' AND schema_version='v2' ORDER BY last_refreshed_at DESC;"

:: pricing-data.js last modified
dir D:\aicost\lib\pricing-data.js
Auto-refreshes on page reload · 🔧 Troubleshooting playbook · JSON API · Updated 9/11/2026, 10:18:58 AM