Crawlee is a powerful Python library for web scraping and browser automation , designed to create scrapers that look almost like real users .

It offers rich configuration options , allowing users to customize the tool to suit their specific needs .

The library offers two main types of crawlers : BeautifulSoupCrawler and PlaywrightCrawler . BeautifulSoupCrawler is ideal for projects that require efficient data extraction from HTML content without the need to execute client-side JavaScript . While PlaywrightCrawler uses a headless browser to handle web pages that rely on client-side JavaScript or require interaction with JavaScript content .

Both crawlers support features such as automatic retries , proxy rotation , session management , custom request routing, and robust error handling .

https://github.com/apify/crawlee-python

Site:

Crawlee is a powerful Python library for web scraping and browser automation , designed to create scrapers that look almost like real users .

It offers rich configuration options , allowing users to customize the tool to suit their specific needs .

The library offers two main types of crawlers : BeautifulSoupCrawler and PlaywrightCrawler . BeautifulSoupCrawler is ideal for projects that require efficient data extraction from HTML content without the need to execute client-side JavaScript . While PlaywrightCrawler uses a headless browser to handle web pages that rely on client-side JavaScript or require interaction with JavaScript content .

Both crawlers support features such as automatic retries , proxy rotation , session management , custom request routing, and robust error handling .

https://github.com/apify/crawlee-python

Site: https://crawlee.dev/python/