Architecture: How should Vitto handle web scraping — Browserless paid tier...

Documentation for Sessions

🏗️ Architecture: How should Vitto handle web scraping — Browserless paid tier...

Date: 2026-02-20
Time: 00:38:26 UTC
Session: gspace1-125.131
Agent: backend
Confidence: 95%
Status: ✅ Documented

📋 Decision Context

Question:

How should Vitto handle web scraping — Browserless paid tier vs HTTP+cheerio hybrid?

Task: Session 125: Web scraping architecture decision

🎯 Decision Made

Chosen:

HTTP fetch + cheerio as primary, Browserless free tier as fallback

Options Considered:

Browserless Starter at $140/mo (40 concurrent browsers)
Self-hosted Playwright on dedicated Railway service
HTTP fetch + cheerio as primary, Browserless free tier as fallback

💭 Reasoning

Browserless at $140/mo limits to 40 concurrent browsers — at scale this means only 40 simultaneous users can trigger browser tasks. HTTP+cheerio covers 80% of scraping (static/SSR pages) at $0 with no concurrent limits. Browserless free tier (1k units/mo) covers the remaining 20% (SPAs needing JavaScript rendering). At scale, self-hosted Playwright on a dedicated Railway service (~$10-20/mo) replaces Browserless entirely. This architecture costs $0 now and scales to unlimited.

📁 Files Involved

apis/backend-vitto-api/src/app/engine/browser-agent.service.ts

🔗 Related

Decision ID: dev_1771547906844_v853vvw
Session: 125
Workspace: gspace1

Auto-generated by MCP Smart Consolidation