Stealth Web-Scraping Framework for Algorithmic Auditing
Reproducible scraping infrastructure for longitudinal search-engine audits
Research infrastructure built during a short-term expert engagement with the Weizenbaum Institute’s Platform Algorithms and Digital Propaganda group and the University of Bern (December 2025).
The framework supports longitudinal algorithmic auditing of Yandex and Google in the context of Kremlin-disinformation research. It is built on Botasaurus with WebGL/Canvas fingerprint evasion, proxy support, parallelization, CAPTCHA handling, and structured logging. The guiding principle was research infrastructure first, code second: the system is reproducible, modular, and documented for hand-off to social-science researchers.
Stack: Python, Botasaurus, proxy orchestration, structured logging. Code: github.com/nimathing2052/webscraper-botasaurus (this is also private, reach out to me individually if you’d want to know more)