As 2016 draws to a close, I look forward to a new year filled with contentment and opportunities. Since I am not going back to New York this year, my journey to an abundant 2017 might just start a bit earlier. To do so, I think a full-function scraper for Mac OS should be a great start and an awesome holiday project for myself as well!
Goal:
A web scraper crawls and stores a given webpage for targeted elements with proxy masking and user agent rewrite support. The scraper can also be used scrap public proxy sites to get a list of working proxies for more intensive web scraping.
Features:
- Web crawling/scraping
- Public proxy scraping
- Private proxy support
- User agent overwrite and randomization
Solutions:
A python script will take care of all the major functionalities and a native OS app in Swift will take over for the interface (input/output) and UX.
Note:
Comparability will be the first priority for this project. The final product should be able to be used by anyone on any Mac OS machine regardless its settings and environments. Performance will also be closely watched and tested. Estimated turnaround time: 2 weeks.