Prometheus by Firecrawl

Prometheus generating the collector code is cool, but the part I’d love to see is whether it also generates validation checks with it. For example, after it samples the data, can it create simple assertions like required fields, expected value ranges, stable selectors, or row counts so teams can catch bad runs before the data reaches downstream systems?

Stumbled across this morning and let me tell you in just an hour of using it I have extracted immense value when it comes to building websites for local businesses and extracting specific components. I have been using the Firecrawl plugin in Claude Code for about 3 months now and to have a streamlined way to use it separately has been very helpful so far, cheers lads.

Web scraping for agents is one of those problems that sounds simple until you actually try it. Curious how it handles sites with heavy JS rendering or login walls — that's usually where these tools fall apart. Will test it on a few of my usual sources.

If you host it, can you see exactly what code it generated and edit it when you need to?

the run it yourself or let us host split is interesting because those are pretty different value propositions. running it yourself means you're still responsible for maintenance even if the agent writes the code. hosting means you're trusting Firecrawl to maintain the extraction logic as sites change, which is a significant operational commitment to make on someone else's behalf. curious what the SLA looks like for hosted maintenance and what happens when a site change breaks extraction and you need the data today

The automatic maintenance angle is clever. Having the agent detect and regenerate scraping code as page structures shift solves one of the biggest pain points in web data pipelines. We've spent significant time dealing with brittle scrapers that break silently and cause data quality issues downstream. How does Prometheus decide when a structural change warrants code regeneration vs. treating it as a transient anomaly?

This is neat. When page structures change, does Prometheus auto-detect the break or wait for you to flag it?

The maintenance part is what makes this useful. Writing a scraper once is easy enough, but keeping it working as pages change is where most web data projects quietly become a time sink.

Hey Product Hunt 👋 Eric, Caleb, and Nick from Firecrawl here. Today we're launching Prometheus, a Forward Deployed Agent for web data.

Our customers often know exactly what data they need, but not how to collect it. Typically, this is where our engineering team would come in: scoping the request, testing approaches against the site, and building the collector with Firecrawl.

We wanted that engineer on call for everyone, so we turned them into an agent.

Simply describe the data you want in plain English. Prometheus experiments against the live site, writes a genuine Firecrawl SDK collector in TypeScript, and runs it before handing it back, so the code is verified working. You get the script plus the sample data it produced.

From there, you have two options. Keep the code, which is reproducible, versionable, and entirely yours to embed wherever you like. Or leave it with us, and Prometheus runs it on a schedule, heals it when the page changes, and delivers the data wherever you need.

Connecting your Firecrawl account is a single OAuth grant, scoped to the team you pick and revocable anytime from your dashboard. It's also available over HTTP, CLI, and MCP, so your coding agent can reach for it too.

It's experimental, but it's already saving our engineering team hours every week. You can try it with Claude Fable 5 for free this weekend (Ending Sunday night).

Try it here: https://www.firecrawl.dev/promet...

We’re excited to see what you build with it.