This document discusses building a single database containing all web data by creating a scalable web crawler, data store, and data retrieval system. It describes the challenges of collecting and structuring data from millions of websites, building a NoSQL data store using Cassandra to handle terabytes of data, and providing an intuitive RESTful API for querying the unified database. The project a
![The Internet as a Single Database](https://cdn-ak-scissors.b.st-hatena.com/image/square/7d22411d3ace575a883704251541b8d61d4d88b7/height=288;version=1;width=512/https%3A%2F%2Fcdn.slidesharecdn.com%2Fss_thumbnails%2Fhoustoncodecamp-110901105543-phpapp02-thumbnail.jpg%3Fwidth%3D640%26height%3D640%26fit%3Dbounds)