THE PROBLEM
Why the system needed to exist.
Useful public data was trapped behind inconsistent tables, downloads, page flows and formats. Manual retrieval made historical imports take days and was difficult to reproduce.
All projectsWeb scraping · Data engineering
Real project
20+ public data sources, collected automatically.
More than 20 source-specific collectors that transformed statistical and customs websites into repeatable, multi-year ingestion workflows.
20+sources automated
- The problem
- Customs and statistics websites each publish data differently. Historical imports took days of manual downloads and couldn't be repeated reliably.
- What I built
- 20+ source-specific scrapers feeding one standard schema, with checks that stop bad data instead of passing it on.
- My role
- Investigated each source, wrote the extraction logic and packaged runs with Docker. Junior Data Analyst.
- Result
- Up to 80% less processing time on customs and statistical data. Multi-year imports now repeatable.
- Why it was hard
- Four formats: HTML, Excel, CSV and PDF
- Websites that change structure without warning
- Large multi-year data pulls
- Tools
- Python · BeautifulSoup · Requests · Pandas · Docker
MY ROLE
Where I created leverage.
I investigated each source, designed resilient extraction logic, normalized the output and packaged the scripts for consistent local and cloud execution.
SANITIZED WORK SAMPLE
Source quirks stay outside the canonical dataset.
The pattern below captures how independent collectors feed one reliable analytical contract.
Each collector handles navigation, pagination and file-specific behavior.
Names, dates, units and identifiers are normalized before delivery.
Failures become explicit review states rather than silent partial exports.
Architecture-level evidence only. Selectors, access patterns, schedules and destinations are omitted.
Technical details
SYSTEM FLOW
From friction
to a repeatable flow.
- Source discovery
- Request strategy
- Extraction
- Schema detection
- Normalization
- Validation
- Incremental export
- Deployment
CONSTRAINTS
The difficult parts.
- HTML, Excel, CSV and PDF source formats
- Changing website structures
- Large multi-year pulls
- Local and cloud runtime consistency
DESIGN DECISIONS
How I approached them.
- Kept extraction adapters separate from the canonical output schema.
- Added schema checks and clear failures instead of silently producing partial data.
- Used streaming or incremental output for large files to control memory use.
- Standardized runtime behavior with Docker and development containers.
OUTCOMES
Project outcomes
automated scrapers
Across customs and statistical sources.
faster processing
Up to 80% less processing time in key workflows.
delivery
Consistent local and cloud execution.
TECHNICAL SURFACE
NEXT SYSTEM / 06
↗