Firecrawl Review: Search, Scrape, Crawl and Monitor the Web

Firecrawl has expanded from general web retrieval into a larger piece of scientific-research infrastructure. On August 13, 2026, the company added a Life Sciences category to its Research Index, announcing coverage of more than 41 million papers across drug discovery, clinical trials, biology, and related biomedical literature. Firecrawl also made all current Research Index categories free to query, including the AI and machine-learning collection launched in June.

The current documentation describes the combined index as roughly 43 million paper abstracts. Most of the corpus comes from PubMed, bioRxiv, and medRxiv, with arXiv adding physics, mathematics, computer-science, and other technical research. Agents can search abstracts in natural language, inspect canonical metadata and source identifiers, retrieve passages relevant to a question, and expand from a seed paper to similar papers, citations, or references.

That is a meaningful addition for builders who would otherwise combine several source APIs, parsers, identifier systems, ranking layers, and full-text tools. It is also easy to overstate. A large index is not a completed review, a relevant passage is not an appraisal, and a high recall score from the vendor is not independent evidence that a query found everything that matters. The feature is valuable precisely when it is used as a retrieval layer with documented checks around it.

### A paper index is different from web search

Firecrawl now has two features that can sound like the same thing. The dedicated Research Index searches a corpus of paper abstracts and returns paper records with identifiers, titles, abstracts, scores, and optional ranking signals. Ordinary Firecrawl Search can also be restricted to a research category, but that searches the open web across a limited set of academic websites and returns standard web results.

The distinction matters in an agent workflow. A paper record can be addressed through identifiers such as PMID, PMCID, DOI, or arXiv ID. The agent can pass that identifier into a second request to inspect the paper or retrieve question-relevant passages. It can then expand from strong seeds through similar papers, citers, or references. Ordinary web search is better suited to pages on journal, preprint, or institutional sites, but it does not provide the same paper-specific graph and passage tools.

The safer implementation makes this distinction explicit in code and in the audit log. A user asking for peer-reviewed trials should not silently receive a mixture of ordinary web pages, PubMed records, bioRxiv preprints, and medRxiv preprints under one label. Each result should retain its source ID, publication type, publication or update date, abstract, retrieved passages, and current status. Preprints should be marked as preprints until their publication status is verified.

### The retrieval claim needs context

Firecrawl reports that the Life Sciences category achieved 90 percent recall at 10 on its paper-retrieval evaluation. In plain terms, the company says the system found relevant papers within the first ten results for nine out of ten benchmark cases. That would be useful for agents because a strong first page reduces the number of papers and tokens needed before a relevant source appears.

The figure remains a vendor-run benchmark. The announcement does not establish that the same performance holds across every clinical specialty, rare disease, molecular target, language, study design, publication date, or ambiguous real-world question. Recall at 10 is also different from comprehensive recall. A systematic review may need nearly every eligible study, including older terminology, negative results, regional journals, registry records, corrections, and grey literature.

Teams should reproduce the evaluation on their own question set. Build a gold set of known relevant papers, include easy and difficult queries, test synonyms and identifiers, and measure recall separately for recent work, older work, preprints, reviews, trials, and niche topics. Compare the index with discipline databases and direct source searches. Record both missed papers and irrelevant papers so ranking quality and total retrieval quality are visible.

Firecrawl's June announcement also cited strong performance on arXivQA and an internal benchmark from a customer. Those results are useful product signals, not universal guarantees. The WhatAI view is straightforward: benchmark claims should guide a pilot, not replace one.

### Abstract search and passage reading solve different problems

The Research Index searches roughly 43 million abstracts. Abstracts are efficient for broad discovery, but they compress methods, eligibility rules, subgroup results, uncertainty, adverse events, and limitations. They can emphasize a primary result while omitting details that change whether the finding applies to a specific population or decision.

Firecrawl's passage endpoint helps by retrieving full-text sections relevant to a natural-language question. An agent can ask whether a paper used a particular assay, population, comparator, benchmark, or evaluation method before including it. This is much better than treating an abstract as the complete paper. It still requires verification. The passage may omit surrounding qualifiers, come from a cited discussion of another paper, or fail to expose a table, figure, correction, or supplementary method.

A robust pipeline stores the paper identifier, passage, section or location when available, retrieval query, retrieval date, and source link. A human or a separate verification stage checks the original document around that passage. If the full paper cannot be inspected, the system should say so rather than convert an abstract or isolated excerpt into a strong claim.

The same rule applies to citation expansion. Similarity, co-citation, bibliographic coupling, references, and citers are discovery signals. They can surface a research neighborhood and help an agent find terminology that its first query missed. Citation count and network position are not measures of methodological quality, truth, or applicability. A widely cited paper can be flawed, outdated, or cited because it is controversial.

### Life sciences raises the stakes

Research retrieval in AI and software can lead to a poor implementation or wasted engineering time. Life-science retrieval can influence clinical information, drug discovery, biological models, and public-health decisions. That does not make automated retrieval inappropriate, but it raises the standard for how the results are labelled and reviewed.

The new corpus includes PubMed alongside bioRxiv and medRxiv. Those sources serve different roles. PubMed indexes biomedical literature and includes records with different publication types and statuses. bioRxiv and medRxiv distribute preprints before peer review. An agent should never describe every result as a peer-reviewed paper merely because it came from a scientific index.

Clinical work also needs more than papers. Current guidelines, regulatory notices, trial registries, safety communications, retractions, corrections, and local standards may matter more than a highly ranked article. A high-stakes workflow should search the appropriate primary systems, verify dates and jurisdictions, and involve qualified domain judgment. Firecrawl can improve discovery and evidence routing, but it does not provide medical advice or a complete clinical decision process.

Developers should design the output schema around uncertainty. Useful fields include source namespace, identifier, publication status, peer-review status when verified, date, study design, population, intervention or exposure, comparator, outcome, passage, limitations, correction or retraction status, full-text availability, and verification state. Missing values should remain null. A language model should not infer peer review, trial design, sample size, or effect direction from a title.

### Free endpoints lower the barrier, not the operating cost

Firecrawl says the Research Index paper endpoints are now free across categories. That lowers the direct cost of paper search, inspection, passage retrieval, and related-paper expansion. Unauthenticated requests can get started without an API key, while an account and key provide higher limits. The free status is a current product term and can change, so production systems should monitor the official pricing and documentation.

The complete research system still has costs. An agent may use paid web Search to find guidelines or current announcements, Scrape to collect source pages, Parse to process files, Monitor to track updates, and language models to classify, summarize, compare, and verify evidence. Storage, deduplication, evaluation, human review, and additional database access also remain. Free retrieval should not be confused with a free end-to-end review.

Firecrawl's broader pricing now starts with 1,000 monthly credits at no cost. Standard Scrape, Crawl, Map, and Monitor requests generally cost one credit per page, Search costs two credits per ten results, and browser Interact costs two credits per minute. Advanced formats and modes can add cost. Teams that mix the free paper index with the paid context API should record which endpoint generated each charge and cap retries and recursive exploration.

### A defensible research-agent workflow

Start with a written question and intended use. Define the population, concept, intervention or exposure, comparator, outcome, timeframe, publication types, languages, and sources that matter. Run several Research Index queries using synonyms, technical terms, and known identifiers. Keep the exact queries and dates.

Next, inspect the returned records and verify publication status. Build an evidence table rather than asking the model for a conclusion immediately. Use passage retrieval to test whether each candidate actually contains the method, dataset, population, or result relevant to the question. Expand from the best seeds through references and citers, but preserve why each paper entered the set.

Then search outside the index. Use appropriate bibliographic databases, clinical-trial registries, guideline repositories, regulatory sources, and direct publisher records. Check corrections and retractions. Deduplicate by DOI, PMID, PMCID, arXiv ID, title, and authors rather than relying on a single field. For a formal review, apply a documented protocol, independent screening, critical appraisal, and transparent inclusion and exclusion decisions.

Finally, separate retrieval quality from answer quality. Measure whether the right papers were found, whether the right passages were returned, whether extracted fields match the paper, and whether the final synthesis represents uncertainty and disagreement. A fluent answer can conceal failure at any earlier stage.

Firecrawl's Life Sciences expansion gives developers a useful paper-specific retrieval layer with unusually broad biomedical coverage, source identifiers, passage access, and graph expansion. Making it free increases its practical value for experimentation and open research tooling. The responsible conclusion is narrower than the launch headline: it can help agents find and inspect evidence, but it does not prove completeness, peer review, methodological quality, or clinical applicability. Those judgments still belong in the workflow.

WhatAI Decision Box

Firecrawl provides one developer platform for live web search, page scraping, site crawling, URL mapping, structured extraction, document parsing, browser interaction, monitoring, and agent-driven research. Its central value is converting complex web and file content into predictable Markdown or JSON that an application or language model can use.

Use Search when the destination URLs are unknown, Scrape for a known page, Map to inventory a domain, Crawl for multi-page extraction, Parse for uploaded documents, Interact for browser actions, Monitor for recurring change detection, and Agent for preview-stage autonomous gathering. The Research Index provides a separate paper-specific API for abstracts, metadata, passages, and related work.

Firecrawl suits developers, AI teams, researchers, and data engineers that want managed extraction instead of maintaining browsers, proxies, parsers, queues, and document cleaners. It is not a permission layer or a guarantee of accurate data. Teams must define lawful sources, respect target controls, defend against prompt injection, validate schemas, monitor credit use, retain source URLs, and review consequential outputs.

Use Cases

Pricing varies by plan and region — see current pricing.

Plan features change — last updated: 2026-08-16.

Firecrawl Community Discussions

Explore community discussions. Ask and answer questions on Firecrawl to grow and learn together.

Firecrawl Pros & Cons

How to Get Results with Firecrawl: Step-by-Step Workflow

1. Define the lawful source set

   List approved domains, paths, document types, intended use, target terms, robots rules, attribution, retention, personal-data exclusions, and who may access the collected material.

2. Choose the narrow endpoint

   Use Search for unknown sources, Scrape for one known page, Map for site inventory, Crawl for multiple linked pages, Parse for files, Monitor for changes, and Interact only when retrieval is insufficient.

3. Estimate credits and scale

   Calculate pages, search results, advanced formats, browser minutes, monitor frequency, concurrency, retries, and growth. Set request, daily, and monthly limits before using a production key.

4. Map before crawling

   Inventory the site, remove irrelevant paths, duplicates, language mirrors, calendars, faceted navigation, authentication routes, private areas, and external domains, then approve the final URL scope.

5. Define the output contract

   Choose Markdown or a strict JSON Schema, require canonical URL and observed timestamp, preserve supporting text, use null for missing data, and define validation and rejection rules.

6. Secure the request path

   Keep API keys server-side, allowlist targets, block private and metadata networks, inspect redirects, limit headers and actions, separate scraped instructions from system prompts, and verify webhook signatures.

7. Run a representative pilot

   Test static, JavaScript, PDF, image-heavy, paginated, redirected, blocked, missing, changed, duplicate, and malicious pages. Record completeness, failures, latency, credit use, and parsing accuracy.

8. Validate and deduplicate

   Check schema types, required fields, source support, canonical URLs, content hashes, timestamps, publication status, and extraction confidence before storage or model use.

9. Add monitored refresh

   Monitor only sources that need freshness, set a justified cadence, filter cosmetic noise, deliver signed idempotent events, and update only changed documents instead of recrawling everything.

10. Operate and audit

    Track success rate, missing fields, drift, duplicates, target complaints, robots changes, injection attempts, webhook failures, storage growth, credit use, source deletions, and access. Reapprove scope regularly.

Firecrawl Gotchas and Limits to Know Before You Start

• Firecrawl is retrieval infrastructure, not legal permission to collect, store, republish, train on, or commercialize target content.
• Review target terms, robots directives, copyright, database rights, privacy laws, contractual restrictions, and rate limits before collection.
• The Crawl endpoint respects FirecrawlAgent robots rules, but users should not assume every endpoint or target policy is equivalent.
• Never let untrusted users provide arbitrary URLs without blocking private networks, metadata endpoints, internal hosts, credential-bearing URLs, and unsafe redirects.
• Scraped pages can contain prompt injection intended to manipulate an agent or expose secrets through tools and outbound requests.
• Treat retrieved content as untrusted data, separate it from system instructions, and restrict downstream tools and credentials.
• Markdown or JSON that parses successfully can still omit hidden tabs, pagination, lazy-loaded content, images, tables, or gated sections.
• JavaScript rendering improves coverage but does not guarantee success against authentication, CAPTCHAs, anti-bot controls, or unusual single-page applications.
• Schema extraction can hallucinate, normalize incorrectly, or fill a field from unrelated page content.
• Preserve source URLs, observed timestamps, supporting passages, and null values so every extracted record can be audited.
• Map results should be reviewed before Crawl because an unexpected site graph can consume far more pages than planned.
• Set page limits, depth, path rules, domain rules, concurrency, timeout, and budget before launching a crawl.
• Crawl results can contain duplicates, canonical variants, faceted pages, calendar traps, language mirrors, and session-specific URLs.
• A successful request is billed according to the endpoint and enabled features; advanced JSON, Enhanced Mode, and other options can cost extra credits.
• Search costs 2 credits per 10 results before optional scraping and advanced output costs are considered.
• Interact costs 2 credits per browser minute, so stuck navigation and open sessions need strict time limits.
• Monitor costs 1 credit per page per check, making short intervals across large sites expensive.
• Self-serve credits do not roll over; Scale and Enterprise rollover is subject to their terms.
• The public pricing page shows Hobby, Standard, Growth, and Scale monthly equivalents with annual billing.
• Agent is in preview with 5 free daily runs and dynamic pricing, so limits and behavior can change.
• Research Index paper endpoints are currently free, but unauthenticated and authenticated rate limits can differ and the offer can change.
• The Research Index searches roughly 43 million abstracts, not every scholarly paper or every discipline equally.
• The research corpus includes PubMed, bioRxiv, medRxiv, and arXiv, so results can include preprints that are not peer reviewed.
• Firecrawl's 90% recall at 10 claim for life-science retrieval is a vendor evaluation and should not be treated as independent proof of comprehensive recall.
• Passage retrieval helps verify a paper, but it does not replace reading methods, results, limitations, corrections, and publication status.
• The Research Index is different from ordinary web Search with the research category; the latter searches a small set of academic websites.
• Lockdown Mode works only with Scrape and returns a cache-miss error rather than making a live request.
• Lockdown content can be up to two years old and is unsuitable for live prices, breaking news, current availability, or other freshness-sensitive facts.
• Lockdown ignores conflicting live options such as actions, wait settings, custom headers, proxy settings, and change tracking.
• Monitoring can report cosmetic or noisy changes unless goals, filters, canonicalization, and diff rules are tested.
• Webhook consumers must verify signatures, deduplicate events, handle retries, and make downstream actions idempotent.
• Standard cloud handling and Enterprise zero-data retention are different service configurations and should not be conflated.
• The Enterprise page states that scraped content is processed and immediately deleted under its zero-retention configuration.
• Self-hosting does not make scraping offline; target requests and optional proxy, AI, or parser providers create outbound data flows.
• The self-host quickstart disables API authentication and lacks durable storage, TLS, high availability, and every cloud feature.
• Self-hosted operators own secrets, patching, security, storage, backups, queues, monitoring, recovery, and incident response.
• Firecrawl's terms prohibit several uses involving personal information, intelligence, law-enforcement evidence, hard background checks, and Fair Credit Reporting Act purposes.
• The affiliate program requires a unique approved referral link; the official commission page is not itself a WhatAI tracking URL.

Which Firecrawl Feature Fits Your Use Case

Starter Prompts for Firecrawl

What is Firecrawl?

Firecrawl is a developer platform that searches, scrapes, crawls, parses, monitors, and interacts with web content, returning formats such as Markdown, HTML, screenshots, links, summaries, or schema-constrained JSON.

How much does Firecrawl cost?

As verified on August 16, 2026, Free is $0, Hobby is $16 monthly billed yearly, Standard is $83, Growth is $333, Scale is $599, and Enterprise is custom. Self-serve plan prices shown here use annual billing.

What does the Firecrawl Free plan include?

The Free plan includes 1,000 credits per month, 2 concurrent requests, and low rate limits. A standard Scrape, Crawl, Map, or Monitor operation generally costs 1 credit per page.

How do Firecrawl credits work?

Standard Scrape, Crawl, Map, and Monitor requests cost 1 credit per page or check. Search costs 2 credits per 10 results, and Interact costs 2 credits per browser minute. Advanced formats and modes can cost more.

Do unused Firecrawl credits roll over?

Unused credits do not roll over on self-serve plans. Firecrawl says rollover is available on Scale and Enterprise subject to their plan or contract terms.

What is the difference between Scrape and Crawl?

Scrape processes a known URL. Crawl begins at a URL, discovers linked pages through sitemaps and traversal, then scrapes the permitted pages according to depth, path, domain, and page-limit settings.

What is the difference between Search and Map?

Search finds ranked pages across the web for a query and can scrape their contents. Map inventories URLs on a particular website, which is useful for choosing scope before a more expensive crawl.

Can Firecrawl extract structured JSON?

Yes. The JSON output format accepts a natural-language extraction prompt, a JSON Schema, or both. The result still needs type, completeness, source, and business-rule validation before use.

Does Firecrawl handle JavaScript-heavy websites?

Yes. The managed service renders dynamic sites and handles proxies, caching, rate limits, and browser behavior. Difficult authentication, anti-bot controls, unusual applications, and target changes can still cause missing or failed extraction.

Can Firecrawl parse PDFs and office documents?

Yes. Parse supports PDF, Word, Excel, PowerPoint, OpenDocument, EPUB, CSV, HTML, and additional file types, with files up to 50 MB per request. PDF parsing offers automatic, fast text, and OCR modes.

What does Firecrawl Monitor do?

Monitor runs scheduled checks against pages, sites, or recurring web searches, records structured changes, and can notify a webhook, email address, or Slack channel when a relevant change or error occurs.

What is Firecrawl Lockdown Mode?

Lockdown Mode serves Scrape results only from Firecrawl's existing cache and prevents an outbound request. A cache miss returns an error. It can return stale content and currently does not cover Crawl, Map, or Search.

What is the Firecrawl Research Index?

It is a paper-specific index covering roughly 43 million abstracts from sources including PubMed, bioRxiv, medRxiv, and arXiv. Agents can search papers, inspect metadata, retrieve relevant passages, and expand to related papers.

Is the Firecrawl Research Index free?

Firecrawl currently states that its Research Index paper endpoints are free across all available categories, including AI and machine learning plus life sciences. Higher unauthenticated and authenticated limits can differ.

Can Firecrawl be self-hosted?

Yes. The open-source stack can be run with Docker Compose. The quickstart is not a production architecture and does not include every cloud capability. Operators own authentication, TLS, storage, scaling, monitoring, recovery, upgrades, and providers.

Does Firecrawl respect robots.txt?

Firecrawl says Crawl respects rules set for the FirecrawlAgent directive. Endpoint behavior is not a substitute for authorization. Users remain responsible for target terms, privacy, copyright, robots rules, rate limits, and applicable law.

Does Firecrawl have an affiliate program?

Yes. The official program currently lists 25% of subscription revenue for the first 12 months and 15% ongoing for active referred subscriptions. Applications and tracking use Dub.

Related AI, Coding and Development Tools

Explore the Network