aicost crawler health

STALE
๐Ÿ“ˆ Price changes (7d)
132
across all crawlers
๐Ÿ†• New models (7d)
205
need review / catalog
๐Ÿงน Sanity catches (7d)
40
crawler self-heal working
โŒ Failures (7d)
0
crawl or extraction errors

1. Token API Pricing Crawler

17 LLM/embedding/TTS API vendors ยท per-token rates ยท manually-curated source of truth at lib/pricing-data.js (crawler reads this for diff, does not auto-write)
AUTOMATED DAILY ยท 03:00 EDT
SourceLast modifiedVendors coveredStatus
lib/pricing-data.js 14h ago anthropic, openai, google, deepseek, xai, mistral, cohere, together, alibaba, minimax, moonshot, nvidia, stepfun, xiaomi, z-ai, tencent, meta โœ“ clean

2. Product / Subscription Pricing Crawler

14 consumer/team plan vendors ยท seat tiers ยท slug stored as product:vendor
AUTOMATED DAILY ยท 03:00 EDT
VendorLast runModelsChangesNew๐ŸงนStatus
anthropic38h ago14โ€”14โ€”โš  review
openai38h ago6โ€”6โ€”โš  review
google90d ago4โ€”4โ€”โš  review
google-ai38h ago8โ€”8โ€”โš  review
google-workspace38h ago4โ€”4โ€”โš  review
github38h ago4โ€”4โ€”โš  review
cursor38h ago6โ€”6โ€”โš  review
windsurf38h ago5โ€”5โ€”โš  review
perplexity38h ago3โ€”3โ€”โš  review
midjourney38h ago5โ€”5โ€”โš  review
runway38h ago5โ€”5โ€”โš  review
elevenlabs38h ago7โ€”7โ€”โš  review
replit38h ago5โ€”5โ€”โš  review
vercel38h ago4โ€”4โ€”โš  review

3. Hybrid Plan Pricing Crawler

16 seat+overage vendors ยท slug stored as hybrid:vendor ยท CF /markdown + Gemini extract
AUTOMATED DAILY ยท 03:00 EDT
VendorLast runPlansChangesNew๐ŸงนStatus
anthropic81d ago731โ€”โš  partial
aws81d ago5โ€”5โ€”โš  partial
cursorneverโ€”โ€”โ€”โ€”โธ pending
github81d ago3โ€”โ€”โ€”โš  partial
google81d ago733โ€”โš  partial
hubspotneverโ€”โ€”โ€”โ€”โธ pending
microsoft81d ago5โ€”3โ€”โš  partial
mistral81d ago2โ€”2โ€”โš  partial
notionneverโ€”โ€”โ€”โ€”โธ pending
openai81d ago714โ€”โš  review
salesforceneverโ€”โ€”โ€”โ€”โธ pending
selfhostneverโ€”โ€”โ€”โ€”โธ pending
servicenowneverโ€”โ€”โ€”โ€”โธ pending
slackneverโ€”โ€”โ€”โ€”โธ pending
together81d ago4โ€”4โ€”โš  partial
windsurfneverโ€”โ€”โ€”โ€”โธ pending

4. Manual Maintenance

1 vendor(s) blocked from auto-scrape
MANUAL QUARTERLY ยท OR ON ANNOUNCEMENTS
VendorLast verifiedDays sincePlansNext dueStatusProcedure
microsoft
Microsoft 365 / Copilot / Sales Copilot
2026-05-0182d72026-08-01โš  reviewdocs/MICROSOFT-MANUAL-UPDATES.md

5. Benchmark Crawler

10 mechanism topics ยท prompt-cache rates, batch discounts, latencies ยท writes aicost_benchmarks table
AUTOMATED WEEKLY ยท WED 03:00 EDT
TopicLast verifiedTopicsNeeds reviewStatus
anthropic-caching
Anthropic Prompt Caching
14h ago 4 00000 โœ“ clean
anthropic-batch
Anthropic Message Batches
14h ago 2 000 โœ“ clean
openai-batch
OpenAI Batch API
14h ago 2 000 โœ“ clean
openai-caching
OpenAI Prompt Caching
29d ago 2 000 ! stale
google-gemini
Google Gemini
14h ago 3 0000 โœ“ clean
google-vertex
Google Vertex AI
14h ago 1 00 โœ“ clean
aws-bedrock
AWS Bedrock
14h ago 1 00 โœ“ clean
mistral
Mistral
14h ago 1 00 โœ“ clean
pinecone
Pinecone
14h ago 2 000 โœ“ clean
cohere
Cohere
14h ago 2 000 โœ“ clean

๐Ÿ“Š Recent price changes (last 7 days ยท all crawlers)

TrackModelFieldOldNewฮ”%Detected
pricingmistral/mistral-large-3-25-12new_model_addedโ€”7/22/2026
pricingdeepgram/nova-3-multilingualcostPerMinute0.0058โ€”7/22/2026
pricingdeepgram/nova-3-monolingualcostPerMinute0.0048โ€”7/22/2026
pricingcohere/command-routput0.61.5150.0%7/22/2026
pricingcohere/command-rinput0.150.5233.3%7/22/2026
pricingmistral/mistral-large-3-25-12batchOutput3โ€”7/22/2026
pricingmistral/mistral-large-3-25-12batchInput1โ€”7/22/2026
pricinggoogle/gemini-3-1-flash-lite-imagecostPerImage_standard0.0336โ€”7/22/2026
pricinggoogle/gemini-3-1-pro-preview-customtoolscachedInput0.2โ€”7/22/2026
pricinggoogle/gemini-omni-flash-previewcostPerSecond0.1โ€”7/22/2026
pricinggoogle/gemini-3-5-live-translate-previewcostPerMinute0.0053โ€”7/22/2026
pricingopenai/gpt-image-1-5batchOutput1615-6.3%7/22/2026
pricingopenai/gpt-image-1-5output3230-6.3%7/22/2026
pricingopenai/gpt-realtime-minicostPerMinute0.090.017-81.1%7/22/2026
pricingopenai/gpt-realtime-1-5costPerMinute0.2880.034-88.2%7/22/2026
pricingopenai/gpt-5-4-probatchOutput90270200.0%7/22/2026
pricingopenai/gpt-5-4-probatchInput1560300.0%7/22/2026
pricingopenai/gpt-5-4batchOutput7.522.5200.0%7/22/2026
pricingopenai/gpt-5-4batchInput1.255300.0%7/22/2026
pricingopenai/gpt-5-5-probatchOutput90270200.0%7/22/2026
pricingopenai/gpt-5-5-probatchInput1560300.0%7/22/2026
pricingopenai/gpt-5-5batchOutput1545200.0%7/22/2026
pricingopenai/gpt-5-5batchInput2.510300.0%7/22/2026
pricingopenai/gpt-5-6-lunabatchOutput9โ€”7/22/2026
pricingopenai/gpt-5-6-lunabatchInput2โ€”7/22/2026
pricingopenai/gpt-5-6-terrabatchOutput22.5โ€”7/22/2026
pricingopenai/gpt-5-6-terrabatchInput5โ€”7/22/2026
pricingopenai/gpt-5-6-solbatchOutput45โ€”7/22/2026
pricingopenai/gpt-5-6-solbatchInput10โ€”7/22/2026
pricinganthropic/claude-opus-4-8batchOutput12.5โ€”7/22/2026

๐Ÿ“‹ TLDR โ€” what needs attention ๐Ÿ”ง Troubleshooting playbook โ†’

๐Ÿšจ Falling behind (10)

cursor ยท Hybrid Plan Pricing
Never run
hubspot ยท Hybrid Plan Pricing
Never run
notion ยท Hybrid Plan Pricing
Never run
salesforce ยท Hybrid Plan Pricing
Never run
selfhost ยท Hybrid Plan Pricing
Never run
servicenow ยท Hybrid Plan Pricing
Never run
slack ยท Hybrid Plan Pricing
Never run
windsurf ยท Hybrid Plan Pricing
Never run
openai-caching ยท Benchmark
Last run 686h ago (>14d)
microsoft ยท Manual Maintenance
Audit due in 8d

โœ… Healthy (32)

pricing-data.js (17 vendors) ยท anthropic (product/subscription) ยท openai (product/subscription) ยท google (product/subscription) ยท google-ai (product/subscription) ยท google-workspace (product/subscription) ยท github (product/subscription) ยท cursor (product/subscription) ยท windsurf (product/subscription) ยท perplexity (product/subscription) ยท midjourney (product/subscription) ยท runway (product/subscription) ยท elevenlabs (product/subscription) ยท replit (product/subscription) ยท vercel (product/subscription) ยท anthropic (hybrid) ยท aws (hybrid) ยท github (hybrid) ยท google (hybrid) ยท microsoft (hybrid) ยท mistral (hybrid) ยท openai (hybrid) ยท together (hybrid) ยท anthropic-caching (benchmark) ยท anthropic-batch (benchmark) ยท openai-batch (benchmark) ยท google-gemini (benchmark) ยท google-vertex (benchmark) ยท aws-bedrock (benchmark) ยท mistral (benchmark) ยท pinecone (benchmark) ยท cohere (benchmark)

๐Ÿ›  Manual run commands

Copy-paste reference for triggering crawlers manually. Use these when scheduled jobs fail, after vendor announcements, or for ad-hoc data refreshes. Replace <slug> with the vendor slug. Add --dry-run to preview without DB writes.

๐Ÿ”ต TOKEN API PRICING (detects vendor changes โ€” pricing-data.js is manually-curated)
:: All vendors (full nightly run)
node scripts\aicost-pricing-crawler.js > D:\aicost\logs\pricing-now.log 2>&1

:: Single vendor
node scripts\aicost-pricing-crawler.js --vendor=anthropic

:: Dry run (preview, no DB writes)
node scripts\aicost-pricing-crawler.js --vendor=openai --dry-run
๐Ÿ”ด PRODUCT / SUBSCRIPTION PRICING (writes product:* rows)
:: All vendors (full weekly run)
node scripts\aicost-product-crawler.js > D:\aicost\logs\product-now.log 2>&1

:: Single vendor
node scripts\aicost-product-crawler.js --vendor=cursor

:: Dry run
node scripts\aicost-product-crawler.js --vendor=github --dry-run
๐ŸŸก HYBRID PLAN PRICING (writes hybrid:* rows)
:: All vendors (full weekly run)
node lib\aicost-hybrid-crawler.js > D:\aicost\logs\hybrid-now.log 2>&1

:: Single vendor
node lib\aicost-hybrid-crawler.js --vendor=anthropic

:: Dry run / Tier-A only / extract phase only
node lib\aicost-hybrid-crawler.js --vendor=openai --dry-run
node lib\aicost-hybrid-crawler.js --tier=A
node lib\aicost-hybrid-crawler.js --phase=extract
๐ŸŸข BENCHMARK CRAWLER (mechanism facts: cache rates, batch discounts, latencies)
:: All 10 topics (full weekly run)
node scripts\aicost-benchmark-crawler.js > D:\aicost\logs\benchmark-now.log 2>&1

:: Single topic
node scripts\aicost-benchmark-crawler.js --vendor=anthropic-caching
node scripts\aicost-benchmark-crawler.js --vendor=openai-batch

:: Dry run / phase-only
node scripts\aicost-benchmark-crawler.js --dry-run
node scripts\aicost-benchmark-crawler.js --phase=crawl
node scripts\aicost-benchmark-crawler.js --phase=extract

:: Available topic slugs:
::   anthropic-caching, anthropic-batch, openai-batch, openai-caching,
::   google-gemini, google-vertex, aws-bedrock, mistral, pinecone, cohere

:: Review flagged rows (Mistral couldn't verify claim from source text)
mysql -u root -p toolsinfo -e "SELECT topic_slug, claim, source_url FROM aicost_benchmarks WHERE needs_review = 1;"
๐ŸŸฃ V2 GUIDE REGENERATION (after pricing changes)
:: Single vendor
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --force

:: Multiple vendors
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --vendor=openai --force

:: All 16 V2 vendors (~$0.30, ~5-7 minutes)
node scripts\generate-vendor-pricing-guides-batch.js --all --force
โš™๏ธ MANUAL MAINTENANCE (Microsoft quarterly audit)
:: Open the procedure doc
notepad D:\aicost\docs\MICROSOFT-MANUAL-UPDATES.md

:: Edit verified plan data with current Microsoft pricing
notepad D:\aicost\lib\hybrid-pricing-data-verified-2026q2.js

:: Update last_verified date in routes/crawler-health.js MANUAL_VENDORS
notepad D:\aicost\routes\crawler-health.js

:: Then regen Microsoft V2
node scripts\generate-vendor-pricing-guides-batch.js --vendor=microsoft --force
๐Ÿ“… SCHEDULED TASK MANAGEMENT
:: Register all scheduled tasks (run as admin)
powershell -ExecutionPolicy Bypass -File D:\aicost\powershell\Run-AicostCrawler.ps1

:: List all aicost scheduled tasks
schtasks /query /tn "aicost*"

:: Manually trigger a scheduled task
schtasks /run /tn "aicost-pricing-crawler"

:: View recent task history (last 24h)
powershell "Get-WinEvent -LogName 'Microsoft-Windows-TaskScheduler/Operational' -MaxEvents 50 | Where-Object {$_.Message -like '*aicost*'}"
๐Ÿ” DIAGNOSTIC QUERIES
:: Last run per vendor (all crawlers)
mysql -u root -p toolsinfo -e "SELECT vendor_slug, COUNT(*) AS runs, MAX(started_at) AS last_run FROM aicost_crawler_runs GROUP BY vendor_slug ORDER BY last_run DESC;"

:: Recent price changes (last 7 days)
mysql -u root -p toolsinfo -e "SELECT provider_slug, model_slug, field_name, old_value, new_value, change_pct, detected_at FROM aicost_price_changelog WHERE detected_at >= DATE_SUB(NOW(), INTERVAL 7 DAY) ORDER BY detected_at DESC LIMIT 20;"

:: V2 guide refresh status
mysql -u root -p toolsinfo -e "SELECT slug, schema_version, last_refreshed_at FROM aicost_guides WHERE type='vendor-pricing' AND schema_version='v2' ORDER BY last_refreshed_at DESC;"

:: pricing-data.js last modified
dir D:\aicost\lib\pricing-data.js
Auto-refreshes on page reload ยท ๐Ÿ”ง Troubleshooting playbook ยท JSON API ยท Updated 7/22/2026, 5:21:10 PM