A Scalable Wikipedia Image Scraper: 15,339 Municipality Images Across 50 States

Jul 23

1 min read

Machine learning is only as good as the data behind it — and sometimes that data has to be built. This project is a scalable web crawler that scraped and organized 15,339 municipality images across all 50 U.S. states, creating a large, structured image dataset from scratch.

How it works

The crawler navigates Wikipedia pages for U.S. municipalities, drives the browser to reach content that isn't in the static HTML, parses each page to locate the relevant images, downloads them, and files everything in an organized, per-state structure — all while handling the scale and inconsistency that comes with thousands of pages.

Technologies used

Python: the crawler and data-organization logic.
Selenium: browser automation to load and navigate dynamic pages reliably.
BeautifulSoup: parsing HTML to extract image URLs and metadata.
File & directory organization: sorting 15,339 images into a clean, per-state structure for downstream use.

Impact

The result is a large, organized image dataset — 15,339 images spanning every U.S. state — assembled automatically and ready to feed downstream computer-vision or analytics work.

Tech stack: Python, Selenium, BeautifulSoup, web crawling, browser automation, HTML parsing, dataset engineering.