Distributed RSS/Atom reader

Distributed system for reading RSS/Atom feeds. The system reads feeds, parses them and saves new entries into database. It also pulls the full content of the entry into database. The system is horizontally scalable (workers and multiple threads per worker) and resilient to partial outages (using message broker). The system uses MongoDB for persistent storage and Apache ActiveMQ for distributing workload. This project was done for a challenge which was organised by Zemanta and Faculty of Computer and Information Science, University of Ljubljana. The project was selected as the best solution in the category of distributed web page aggregation and has consequently won the challenge in that category (news in English, news in Slovenian).

Launch Project