aicost crawler health

STALE
๐Ÿ“ˆ Price changes (7d)
434
across all crawlers
๐Ÿ†• New models (7d)
175
need review / catalog
๐Ÿงน Sanity catches (7d)
41
crawler self-heal working
โŒ Failures (7d)
2
crawl or extraction errors

1. Token API Pricing Crawler

17 LLM/embedding/TTS API vendors ยท per-token rates ยท manually-curated source of truth at lib/pricing-data.js (crawler reads this for diff, does not auto-write)
AUTOMATED DAILY ยท 03:00 EDT
SourceLast modifiedVendors coveredStatus
lib/pricing-data.js 32h ago anthropic, openai, google, deepseek, xai, mistral, cohere, together, alibaba, minimax, moonshot, nvidia, stepfun, xiaomi, z-ai, tencent, meta โœ“ clean

2. Product / Subscription Pricing Crawler

14 consumer/team plan vendors ยท seat tiers ยท slug stored as product:vendor
AUTOMATED DAILY ยท 03:00 EDT
VendorLast runModelsChangesNew๐ŸงนStatus
anthropic6d ago14โ€”14โ€”โš  review
openai6d ago3โ€”3โ€”โš  review
google186d ago4โ€”4โ€”โš  review
google-ai6d ago7โ€”7โ€”โš  review
google-workspace6d ago4โ€”4โ€”โš  review
github6d ago4โ€”4โ€”โš  review
cursor6d ago7โ€”7โ€”โš  review
windsurf6d ago5โ€”5โ€”โš  review
perplexity6d ago3โ€”3โ€”โš  review
midjourney6d ago5โ€”5โ€”โš  review
runway6d ago4โ€”4โ€”โš  review
elevenlabs6d ago7โ€”7โ€”โš  review
replit6d ago4โ€”4โ€”โš  review
vercel6d ago4โ€”4โ€”โš  review

3. Hybrid Plan Pricing Crawler

16 seat+overage vendors ยท slug stored as hybrid:vendor ยท CF /markdown + Gemini extract
AUTOMATED DAILY ยท 03:00 EDT
VendorLast runPlansChangesNew๐ŸงนStatus
anthropic176d ago731โ€”โš  partial
aws176d ago5โ€”5โ€”โš  partial
cursorneverโ€”โ€”โ€”โ€”โธ pending
github176d ago3โ€”โ€”โ€”โš  partial
google176d ago733โ€”โš  partial
hubspotneverโ€”โ€”โ€”โ€”โธ pending
microsoft176d ago5โ€”3โ€”โš  partial
mistral176d ago2โ€”2โ€”โš  partial
notionneverโ€”โ€”โ€”โ€”โธ pending
openai176d ago714โ€”โš  review
salesforceneverโ€”โ€”โ€”โ€”โธ pending
selfhostneverโ€”โ€”โ€”โ€”โธ pending
servicenowneverโ€”โ€”โ€”โ€”โธ pending
slackneverโ€”โ€”โ€”โ€”โธ pending
together176d ago4โ€”4โ€”โš  partial
windsurfneverโ€”โ€”โ€”โ€”โธ pending

4. Manual Maintenance

1 vendor(s) blocked from auto-scrape
MANUAL QUARTERLY ยท OR ON ANNOUNCEMENTS
VendorLast verifiedDays sincePlansNext dueStatusProcedure
microsoft
Microsoft 365 / Copilot / Sales Copilot
2026-05-01178d72026-08-01! staledocs/MICROSOFT-MANUAL-UPDATES.md

5. Benchmark Crawler

10 mechanism topics ยท prompt-cache rates, batch discounts, latencies ยท writes aicost_benchmarks table
AUTOMATED WEEKLY ยท WED 03:00 EDT
TopicLast verifiedTopicsNeeds reviewStatus
anthropic-caching
Anthropic Prompt Caching
5d ago 8 000000000 โœ“ clean
anthropic-batch
Anthropic Message Batches
5d ago 2 000 โœ“ clean
openai-batch
OpenAI Batch API
5d ago 2 000 โœ“ clean
openai-caching
OpenAI Prompt Caching
19d ago 2 000 ! stale
google-gemini
Google Gemini
5d ago 3 0000 โœ“ clean
google-vertex
Google Vertex AI
5d ago 1 00 โœ“ clean
aws-bedrock
AWS Bedrock
5d ago 1 00 โœ“ clean
mistral
Mistral
5d ago 1 00 โœ“ clean
pinecone
Pinecone
5d ago 2 000 โœ“ clean
cohere
Cohere
12d ago 2 000 โœ“ clean

๐Ÿ“Š Recent price changes (last 7 days ยท all crawlers)

TrackModelFieldOldNewฮ”%Detected
pricingtogether/llama-3-3-70boutput1.045.08388.5%10/25/2026
pricingtogether/llama-3-3-70binput1.042.0395.2%10/25/2026
pricingdeepgram/nova-3-multilingualbatchInput0.00920.0052-43.5%10/25/2026
pricingdeepgram/nova-3-monolingualbatchInput0.00770.0043-44.2%10/25/2026
pricingmistral/mistral-small-4-0-26-03batchOutput0.2250.075-66.7%10/25/2026
pricingmistral/mistral-small-4-0-26-03batchInput0.0750.025-66.7%10/25/2026
pricingmistral/mistral-small-4-0-26-03output0.450.15-66.7%10/25/2026
pricingmistral/mistral-small-4-0-26-03input0.150.05-66.7%10/25/2026
pricingmistral/mistral-large-3-25-12batchOutput0.75โ€”10/25/2026
pricingmistral/mistral-large-3-25-12batchInput0.25โ€”10/25/2026
pricingxai/grok-imagine-imagecostPerImage_standard0.02โ€”10/25/2026
pricingxai/grok-imagine-image-qualitycostPerImage_standard0.05โ€”10/25/2026
pricingxai/grok-imagine-image-2-0costPerImage_standard0.04โ€”10/25/2026
pricingdeepseek/deepseek-v4-prooutput3.963.3-16.7%10/25/2026
pricingdeepseek/deepseek-v4-proinput1.321.1-16.7%10/25/2026
pricingdeepseek/deepseek-flashoutput1.20.28-76.7%10/25/2026
pricingdeepseek/deepseek-flashinput0.30.14-53.3%10/25/2026
pricinggoogle/gemini-3-8-flash-lite-ttscostPerMinute0.0015โ€”10/25/2026
pricinggoogle/gemini-3-8-flash-ttscostPerMinute0.00225โ€”10/25/2026
pricinggoogle/gemini-3-1-flash-lite-imagecostPerImage_standard0.0336โ€”10/25/2026
pricinggoogle/gemini-3-1-flash-lite-imagebatchOutput15โ€”10/25/2026
pricinggoogle/gemini-3-1-flash-lite-imagebatchInput0.125โ€”10/25/2026
pricinggoogle/gemini-3-1-flash-lite-imageoutput30โ€”10/25/2026
pricinggoogle/gemini-3-1-flash-lite-imageinput0.25โ€”10/25/2026
pricinggoogle/gemini-nano-banana-2-1costPerImage_standard0.0336โ€”10/25/2026
pricinggoogle/gemini-nano-banana-2-1batchOutput15โ€”10/25/2026
pricinggoogle/gemini-nano-banana-2-1batchInput0.75โ€”10/25/2026
pricinggoogle/gemini-nano-banana-2-1output30โ€”10/25/2026
pricinggoogle/gemini-nano-banana-2-1input1.5โ€”10/25/2026
pricinggoogle/gemini-3-1-pro-preview-customtoolscachedInput0.2โ€”10/25/2026

๐Ÿ“‹ TLDR โ€” what needs attention ๐Ÿ”ง Troubleshooting playbook โ†’

๐Ÿšจ Falling behind (10)

cursor ยท Hybrid Plan Pricing
Never run
hubspot ยท Hybrid Plan Pricing
Never run
notion ยท Hybrid Plan Pricing
Never run
salesforce ยท Hybrid Plan Pricing
Never run
selfhost ยท Hybrid Plan Pricing
Never run
servicenow ยท Hybrid Plan Pricing
Never run
slack ยท Hybrid Plan Pricing
Never run
windsurf ยท Hybrid Plan Pricing
Never run
openai-caching ยท Benchmark
Last run 461h ago (>14d)
microsoft ยท Manual Maintenance
Audit overdue by 88d

โœ… Healthy (32)

pricing-data.js (17 vendors) ยท anthropic (product/subscription) ยท openai (product/subscription) ยท google (product/subscription) ยท google-ai (product/subscription) ยท google-workspace (product/subscription) ยท github (product/subscription) ยท cursor (product/subscription) ยท windsurf (product/subscription) ยท perplexity (product/subscription) ยท midjourney (product/subscription) ยท runway (product/subscription) ยท elevenlabs (product/subscription) ยท replit (product/subscription) ยท vercel (product/subscription) ยท anthropic (hybrid) ยท aws (hybrid) ยท github (hybrid) ยท google (hybrid) ยท microsoft (hybrid) ยท mistral (hybrid) ยท openai (hybrid) ยท together (hybrid) ยท anthropic-caching (benchmark) ยท anthropic-batch (benchmark) ยท openai-batch (benchmark) ยท google-gemini (benchmark) ยท google-vertex (benchmark) ยท aws-bedrock (benchmark) ยท mistral (benchmark) ยท pinecone (benchmark) ยท cohere (benchmark)

๐Ÿ›  Manual run commands

Copy-paste reference for triggering crawlers manually. Use these when scheduled jobs fail, after vendor announcements, or for ad-hoc data refreshes. Replace <slug> with the vendor slug. Add --dry-run to preview without DB writes.

๐Ÿ”ต TOKEN API PRICING (detects vendor changes โ€” pricing-data.js is manually-curated)
:: All vendors (full nightly run)
node scripts\aicost-pricing-crawler.js > D:\aicost\logs\pricing-now.log 2>&1

:: Single vendor
node scripts\aicost-pricing-crawler.js --vendor=anthropic

:: Dry run (preview, no DB writes)
node scripts\aicost-pricing-crawler.js --vendor=openai --dry-run
๐Ÿ”ด PRODUCT / SUBSCRIPTION PRICING (writes product:* rows)
:: All vendors (full weekly run)
node scripts\aicost-product-crawler.js > D:\aicost\logs\product-now.log 2>&1

:: Single vendor
node scripts\aicost-product-crawler.js --vendor=cursor

:: Dry run
node scripts\aicost-product-crawler.js --vendor=github --dry-run
๐ŸŸก HYBRID PLAN PRICING (writes hybrid:* rows)
:: All vendors (full weekly run)
node lib\aicost-hybrid-crawler.js > D:\aicost\logs\hybrid-now.log 2>&1

:: Single vendor
node lib\aicost-hybrid-crawler.js --vendor=anthropic

:: Dry run / Tier-A only / extract phase only
node lib\aicost-hybrid-crawler.js --vendor=openai --dry-run
node lib\aicost-hybrid-crawler.js --tier=A
node lib\aicost-hybrid-crawler.js --phase=extract
๐ŸŸข BENCHMARK CRAWLER (mechanism facts: cache rates, batch discounts, latencies)
:: All 10 topics (full weekly run)
node scripts\aicost-benchmark-crawler.js > D:\aicost\logs\benchmark-now.log 2>&1

:: Single topic
node scripts\aicost-benchmark-crawler.js --vendor=anthropic-caching
node scripts\aicost-benchmark-crawler.js --vendor=openai-batch

:: Dry run / phase-only
node scripts\aicost-benchmark-crawler.js --dry-run
node scripts\aicost-benchmark-crawler.js --phase=crawl
node scripts\aicost-benchmark-crawler.js --phase=extract

:: Available topic slugs:
::   anthropic-caching, anthropic-batch, openai-batch, openai-caching,
::   google-gemini, google-vertex, aws-bedrock, mistral, pinecone, cohere

:: Review flagged rows (Mistral couldn't verify claim from source text)
mysql -u root -p toolsinfo -e "SELECT topic_slug, claim, source_url FROM aicost_benchmarks WHERE needs_review = 1;"
๐ŸŸฃ V2 GUIDE REGENERATION (after pricing changes)
:: Single vendor
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --force

:: Multiple vendors
node scripts\generate-vendor-pricing-guides-batch.js --vendor=anthropic --vendor=openai --force

:: All 16 V2 vendors (~$0.30, ~5-7 minutes)
node scripts\generate-vendor-pricing-guides-batch.js --all --force
โš™๏ธ MANUAL MAINTENANCE (Microsoft quarterly audit)
:: Open the procedure doc
notepad D:\aicost\docs\MICROSOFT-MANUAL-UPDATES.md

:: Edit verified plan data with current Microsoft pricing
notepad D:\aicost\lib\hybrid-pricing-data-verified-2026q2.js

:: Update last_verified date in routes/crawler-health.js MANUAL_VENDORS
notepad D:\aicost\routes\crawler-health.js

:: Then regen Microsoft V2
node scripts\generate-vendor-pricing-guides-batch.js --vendor=microsoft --force
๐Ÿ“… SCHEDULED TASK MANAGEMENT
:: Register all scheduled tasks (run as admin)
powershell -ExecutionPolicy Bypass -File D:\aicost\powershell\Run-AicostCrawler.ps1

:: List all aicost scheduled tasks
schtasks /query /tn "aicost*"

:: Manually trigger a scheduled task
schtasks /run /tn "aicost-pricing-crawler"

:: View recent task history (last 24h)
powershell "Get-WinEvent -LogName 'Microsoft-Windows-TaskScheduler/Operational' -MaxEvents 50 | Where-Object {$_.Message -like '*aicost*'}"
๐Ÿ” DIAGNOSTIC QUERIES
:: Last run per vendor (all crawlers)
mysql -u root -p toolsinfo -e "SELECT vendor_slug, COUNT(*) AS runs, MAX(started_at) AS last_run FROM aicost_crawler_runs GROUP BY vendor_slug ORDER BY last_run DESC;"

:: Recent price changes (last 7 days)
mysql -u root -p toolsinfo -e "SELECT provider_slug, model_slug, field_name, old_value, new_value, change_pct, detected_at FROM aicost_price_changelog WHERE detected_at >= DATE_SUB(NOW(), INTERVAL 7 DAY) ORDER BY detected_at DESC LIMIT 20;"

:: V2 guide refresh status
mysql -u root -p toolsinfo -e "SELECT slug, schema_version, last_refreshed_at FROM aicost_guides WHERE type='vendor-pricing' AND schema_version='v2' ORDER BY last_refreshed_at DESC;"

:: pricing-data.js last modified
dir D:\aicost\lib\pricing-data.js
Auto-refreshes on page reload ยท ๐Ÿ”ง Troubleshooting playbook ยท JSON API ยท Updated 10/26/2026, 11:45:52 AM