All projects
Research
ProtoData
Mining and Mapping the Protoweb studies structure and behaviour of the early web using restored sites via a Protoweb proxy. The crawler is class-based (initialization, collection, storage) with optional domain boundaries, then materializes data for SQL and graph analysis.
PythonBeautifulSoupSQLiteNeo4jpandas
Highlights
- Proxy-aware crawling with domain-boundary and open-boundary modes.
- Triple storage: raw CSV, relational SQLite (pages/links), Neo4j graph of URLs and edges.
- SQL/pandas analysis scripts over the structured crawl.
- Research write-up: “Mining and Mapping the Protoweb.”
Outcomes
Dataset on the order of ~3001 pages across ~40 domains.
Repeatable pipeline from crawl → SQLite → Neo4j for structural questions.
Why it matters
Shows data engineering and systems thinking: careful crawling, storage tradeoffs, and turning messy web archives into analyzable graph and relational models.