All projectsData quality · Entity resolution

Real project

Finding the same ship under different names.

A production-grade quality layer combining deterministic cleanup, fuzzy matching and maritime timing heuristics to resolve noisy vessel records.

Auditableevery merge reviewable
The problem
The same vessel appeared under different spellings and slightly different dates. Simple duplicate checks missed them, which inflated the data.
What I built
A matching engine that combines cleanup rules, fuzzy name matching and date checks, with a human review step and an audit trail.
My role
Built it from start to finish: the matching rules, the grouping logic and how it runs. My team uses it today.
Result
A reusable quality layer shared across country pipelines, plus a self-service tool for duplicate checks and clean exports.
Why it was hard
  • Vessel names changed between reports
  • ETA, ETB and ETS dates were incomplete or repaired
  • A false merge was as costly as a missed duplicate
Tools
Python · Pandas · fuzzy matching · disjoint-set grouping · Streamlit
Blue-lit server racks representing reliable data infrastructure
Illustrative photo.

THE PROBLEM

Why the system needed to exist.

The same vessel or trade event could appear with prefixes, spelling differences, repaired dates or slightly shifted arrival estimates. Simple row-level duplicate checks could not recognize the underlying entity.

MY ROLE

Where I created leverage.

I translated operational knowledge into reusable matching rules, designed the grouping strategy and created reviewable outputs for both accepted and dropped records.

SANITIZED WORK SAMPLE

A match stays explainable from input to decision.

This fictional example now leads into a working browser demo of the review pattern without exposing production names, thresholds or reference data.

01 / NOISY INPUTMV OCEAN STAR / M.V. OCEANSTAR

Two spellings can describe the same underlying entity.

02 / DECISION PATHNORMALIZE → BLOCK → SCORE → DATE CHECK

Cheap deterministic rules narrow the candidates before contextual checks.

03 / REVIEW OUTPUTSURVIVOR + DROPPED-RECORD TRAIL

Analysts can inspect what remained, what was removed and why.

Illustrative records only. Production thresholds, entities and mapping tables remain private.

INTERACTIVE, SYNTHETIC DEMO

Tune the rules. Review the evidence. Keep the right record.

Test a browser-safe reconstruction of the workflow using 12 fictional maritime records. Nothing is uploaded, stored or connected to a private system.

Records scanned12
Duplicate groups3
ETA > ETS issues1
Negative quantity1
Synthetic records and detection status
RecordStatusVesselETACut-offQtyProduct
SG-001DG-01MV OCEAN STAR03-1403-0525,400Diesel
SG-002DG-01M.V. OCEANSTAR03-1603-0824,800Diesel
SG-003DG-02BLUE HORIZON04-0203-2032,000Crude Oil
SG-004DG-02Blue-Horizon04-0403-2433,500Crude Oil
SG-005ClearNORDIC LIGHT04-1804-1018,100LNG
SG-006ClearNORDIC LIGHT04-1904-1017,950LNG
SG-007DG-03PACIFIC DAWN05-0704-2742,100Gasoline
SG-008DG-03PACIFC DAWN05-0905-0140,500Gasoline
SG-009ETA > ETSALTAIR06-1406-0212,600LPG
SG-010Negative qtySIRIUS06-2106-10-2,400Diesel
SG-011ClearMERIDIAN07-0306-2028,900Crude Oil
SG-012ClearORION BRIDGE07-1907-0814,750LNG
02Human review

The detector proposes groups; the analyst still owns the decision.

DG-01100% name score

SG-001MV OCEAN STARREMOVE IF ACCEPTED

SG-002M.V. OCEANSTARKEEP · LATEST CUT-OFF

ETA gap
2d
ETS gap
1d
Qty variance
2.4%
DG-02100% name score

SG-003BLUE HORIZONREMOVE IF ACCEPTED

SG-004Blue-HorizonKEEP · LATEST CUT-OFF

ETA gap
2d
ETS gap
1d
Qty variance
4.5%
DG-0391% name score

SG-007PACIFIC DAWNREMOVE IF ACCEPTED

SG-008PACIFC DAWNKEEP · LATEST CUT-OFF

ETA gap
2d
ETS gap
1d
Qty variance
3.8%
03 / SURVIVORSHIP OUTPUT12 records retained

Accept a proposed group to preview the cleaned output.

PUBLIC CODE BOUNDARY

Show the reasoning, not the private plumbing.

This excerpt exposes the decision contract visitors need to understand. Authentication, private endpoints, source identifiers, production thresholds and company-specific mappings stay out of the portfolio.

const isMatch =
  vesselScore >= threshold &&
  etaGapDays <= etaTolerance &&
  quantityGapPct <= quantityTolerance &&
  (!checkETS || etsGapDays <= etsTolerance) &&
  sameProduct && sameCountry;

Synthetic capability demo · Fictional vessels, countries, ports, dates and quantities · No visitor data is stored

Technical details

SYSTEM FLOW

From friction
to a repeatable flow.

  1. Prefix cleanup
  2. Name normalization
  3. Candidate blocking
  4. Fuzzy scoring
  5. Date heuristics
  6. DSU grouping
  7. Survivorship
  8. QA export

CONSTRAINTS

The difficult parts.

  • Names changed between reports
  • ETA, ETB and ETS values were incomplete or repaired
  • False merges were as costly as missed duplicates
  • The result needed to remain auditable by analysts

DESIGN DECISIONS

How I approached them.

  • Used deterministic cleanup before fuzzy logic to reduce ambiguous comparisons.
  • Applied candidate blocking so large datasets stayed computationally practical.
  • Combined disjoint-set grouping with ETA/ETB/ETS context rather than trusting one similarity score.
  • Exported dropped and surviving records separately to keep decisions reviewable.

OUTCOMES

Project outcomes

Related work:

11,000+ shipper and receiver names standardized.

Audit-ready

decision trail

Clean and dropped outputs remain reviewable.

Reusable

quality layer

Shared logic across country-specific pipelines.

TECHNICAL SURFACE

PythonPandasDSUFuzzy matchingEntity resolutionData qualityStreamlit
NEXT SYSTEM / 06

20+ public data sources, collected automatically.