$ cat ./protodata.md

research

protodata

mining and mapping the protoweb studies structure and behaviour of the early web using restored sites via a protoweb proxy. the crawler is class-based (initialization, collection, storage) with optional domain boundaries, then materializes data for sql and graph analysis.

python · beautifulsoup · sqlite · neo4j · pandas

## highlights

• proxy-aware crawling with domain-boundary and open-boundary modes.
• triple storage: raw csv, relational sqlite (pages/links), neo4j graph of urls and edges.
• sql/pandas analysis scripts over the structured crawl.
• research write-up: “mining and mapping the protoweb.”

## outcomes

• - dataset on the order of ~3001 pages across ~40 domains.
• - repeatable pipeline from crawl → sqlite → neo4j for structural questions.