Real project
Finding the same ship under different names. A production-grade quality layer combining deterministic cleanup, fuzzy matching and maritime timing heuristics to resolve noisy vessel records.
Auditable every merge reviewable
The problem The same vessel appeared under different spellings and slightly different dates. Simple duplicate checks missed them, which inflated the data.
What I built A matching engine that combines cleanup rules, fuzzy name matching and date checks, with a human review step and an audit trail.
My role Built it from start to finish: the matching rules, the grouping logic and how it runs. My team uses it today.
Result A reusable quality layer shared across country pipelines, plus a self-service tool for duplicate checks and clean exports. Why it was hard Vessel names changed between reports ETA, ETB and ETS dates were incomplete or repaired A false merge was as costly as a missed duplicate
Tools Python · Pandas · fuzzy matching · disjoint-set grouping · Streamlit
Illustrative photo. On this page Overview System flow Constraints and decisions Outcomes THE PROBLEM
Why the system needed to exist. The same vessel or trade event could appear with prefixes, spelling differences, repaired dates or slightly shifted arrival estimates. Simple row-level duplicate checks could not recognize the underlying entity.
MY ROLE
Where I created leverage. I translated operational knowledge into reusable matching rules, designed the grouping strategy and created reviewable outputs for both accepted and dropped records.
SANITIZED WORK SAMPLE
A match stays explainable from input to decision. This fictional example now leads into a working browser demo of the review pattern without exposing production names, thresholds or reference data.
01 / NOISY INPUT MV OCEAN STAR / M.V. OCEANSTAR Two spellings can describe the same underlying entity.
02 / DECISION PATH NORMALIZE → BLOCK → SCORE → DATE CHECK Cheap deterministic rules narrow the candidates before contextual checks.
03 / REVIEW OUTPUT SURVIVOR + DROPPED-RECORD TRAIL Analysts can inspect what remained, what was removed and why.
+ Illustrative records only. Production thresholds, entities and mapping tables remain private.
Records scanned 12 Duplicate groups 3 ETA > ETS issues 1 Negative quantity 1 Synthetic records and detection status Record Status Vessel ETA Cut-off Qty Product SG-001 DG-01 MV OCEAN STAR 03-14 03-05 25,400 Diesel SG-002 DG-01 M.V. OCEANSTAR 03-16 03-08 24,800 Diesel SG-003 DG-02 BLUE HORIZON 04-02 03-20 32,000 Crude Oil SG-004 DG-02 Blue-Horizon 04-04 03-24 33,500 Crude Oil SG-005 Clear NORDIC LIGHT 04-18 04-10 18,100 LNG SG-006 Clear NORDIC LIGHT 04-19 04-10 17,950 LNG SG-007 DG-03 PACIFIC DAWN 05-07 04-27 42,100 Gasoline SG-008 DG-03 PACIFC DAWN 05-09 05-01 40,500 Gasoline SG-009 ETA > ETS ALTAIR 06-14 06-02 12,600 LPG SG-010 Negative qty SIRIUS 06-21 06-10 -2,400 Diesel SG-011 Clear MERIDIAN 07-03 06-20 28,900 Crude Oil SG-012 Clear ORION BRIDGE 07-19 07-08 14,750 LNG
02 Human review
The detector proposes groups; the analyst still owns the decision.
DG-01 100% name score
SG-001 MV OCEAN STAR REMOVE IF ACCEPTED
SG-002 M.V. OCEANSTAR KEEP · LATEST CUT-OFF
ETA gap 2d
ETS gap 1d
Qty variance 2.4% ACCEPT REJECT
DG-02 100% name score
SG-003 BLUE HORIZON REMOVE IF ACCEPTED
SG-004 Blue-Horizon KEEP · LATEST CUT-OFF
ETA gap 2d
ETS gap 1d
Qty variance 4.5% ACCEPT REJECT
DG-03 91% name score
SG-007 PACIFIC DAWN REMOVE IF ACCEPTED
SG-008 PACIFC DAWN KEEP · LATEST CUT-OFF
ETA gap 2d
ETS gap 1d
Qty variance 3.8% ACCEPT REJECT
03 / SURVIVORSHIP OUTPUT 12 records retained Accept a proposed group to preview the cleaned output.
PUBLIC CODE BOUNDARY
Show the reasoning, not the private plumbing. This excerpt exposes the decision contract visitors need to understand. Authentication, private endpoints, source identifiers, production thresholds and company-specific mappings stay out of the portfolio.
const isMatch =
vesselScore >= threshold &&
etaGapDays <= etaTolerance &&
quantityGapPct <= quantityTolerance &&
(!checkETS || etsGapDays <= etsTolerance) &&
sameProduct && sameCountry;Synthetic capability demo · Fictional vessels, countries, ports, dates and quantities · No visitor data is stored
Technical details SYSTEM FLOW
From friction to a repeatable flow. Prefix cleanup Name normalization Candidate blocking Fuzzy scoring Date heuristics DSU grouping Survivorship QA export CONSTRAINTS
The difficult parts. Names changed between reports ETA, ETB and ETS values were incomplete or repaired False merges were as costly as missed duplicates The result needed to remain auditable by analysts DESIGN DECISIONS
How I approached them. Used deterministic cleanup before fuzzy logic to reduce ambiguous comparisons. Applied candidate blocking so large datasets stayed computationally practical. Combined disjoint-set grouping with ETA/ETB/ETS context rather than trusting one similarity score. Exported dropped and surviving records separately to keep decisions reviewable. OUTCOMES
Project outcomes Related work: 11,000+ shipper and receiver names standardized.
Audit-ready decision trail Clean and dropped outputs remain reviewable.
Reusable quality layer Shared logic across country-specific pipelines.
TECHNICAL SURFACE
Python Pandas DSU Fuzzy matching Entity resolution Data quality Streamlit
Client data, source names and internal screens are withheld. Demos use synthetic data. NEXT SYSTEM / 06
20+ public data sources, collected automatically. ↗