GitHub - unclecode/crawl4ai: 🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyNCrawl4AI turns the web into clean, LLM ready Markdown for RAG, agents, and data pipelines. Fast, controllable, battle tested by a 50k+ star community. ✨ Check out latest update v0.8.6 ✨ New in v0.8.6: Security hotfix — replaced litellm with unclecode-litellm due to a PyPI supply chain compromise. If you're on v0.8.5, please upgrade immediately. ✨ Recent v0.8.5: Anti-Bot Detection, Shadow DOM & 60+
GitHub - ghostlulzhacks/commoncrawl: Gathers urls from common crawl
GitHub - scrapinghub/spidermon: Scrapy Extension for monitoring spiders execution.
