Data Dedupe node

Transform

What it does

The Data Dedupe node finds records that describe the same real-world entity even when they are not identical. It scores how confident each match is, collects matches into groups, picks one record per group to survive, and sends each kind of duplicate out through its own port.

It holds every row until the last one arrives, because two records for the same entity can be thousands of rows apart.

When to use

• The key itself is unreliable: the same customer appears as jane.doe@acme.com and Jane.Doe@acme.com, as Jon Smith and John Smith, or as Acme Corp and ACME Corporation.
• You want certain matches handled automatically and doubtful ones sent to a person.

When a key such as customer_id is clean, use Dedupe by key in a Transform instead. It keeps the first or last row per key, with no scoring.

Configure

Select the node and press Configure to open Configure Data Dedupe. The steps are Matching rules, Clean and exclude, Scoring, Scenarios, Survivorship and Summary.

Save and Done stay unavailable while a required setting is missing. Cancel discards the changes made since the dialog opened or was last saved. A message confirms each save.

Matching rules

A rule identifies one entity. It holds one or more groups of fields, and each field is compared with a method. A fuzzy method also has a minimum similarity.

Each group combines its fields with ALL or ANY, and the rule combines its groups the same way. One rule can say:

(Name and Email) or (Name and Phone) or (Name and Address)

Rules run in priority order, and a pair of records is credited to the first rule it passes. Put the rule you trust most at the top.

Method	Kind	What it compares
Exact	exact	Identical, character for character.
Normalized exact	exact	Identical after trimming, casing and punctuation cleanup.
Email canonical	exact	Lower-cases, and strips dots and +tags for Gmail-style addresses.
Phone number	exact	Digits only, ignoring spaces, punctuation and a country code. Equal digits, or equal last ten digits, match.
Jaro-Winkler	fuzzy	Typos and transpositions, favouring matching starts. Suits names.
Levenshtein	fuzzy	Edit distance: inserts, deletes and swaps.
Phonetic	fuzzy	Sounds-alike, so Kathryn matches Catherine. Each word is coded separately.
Token set	fuzzy	Word overlap in any order, so "Acme Corp" matches "Corp Acme Inc".

Blocking: a blocking key, such as postcode, country or the first letter of the surname, compares only records that share it. The Blocking card is open by default. Records are compared in pairs, so without a key 48,000 records means 1.1 billion comparisons. The Summary step flags a missing key.

Clean and exclude

Values such as NA, TBD or unknown repeat across thousands of records. Values on the Placeholder values list are never used as evidence of a match. Under When a field is excluded, choose what happens when a field holds one:

• score the pair on its other fields (the usual choice);
• count the field as a failed check;
• leave the record out of dedupe.

Try a value shows whether a value would be excluded and what it normalises to.

Scoring

Every matched pair scores 0 to 100. Two thresholds split the range: below review is unique, between the two needs a person, and at or above auto-merge is certain. Review is always below auto-merge.

Scenarios

A scenario describes one kind of duplicate and sends it to an output. Define up to ten scenarios across up to ten named outputs; several scenarios can share an output.

Scenarios are checked from the top, and the first that fits wins. Every record leaves through exactly one port. Records that match no scenario leave on unique. Renaming an output keeps its edges.

Each output can send all records in the group, only survivors, or only duplicates. Choose which columns are added to each row: group_id, group_size, scenario, confidence, match_rule, is_survivor, matched_fields.

Survivorship

Matched records form groups, one per entity. One record per group is the survivor. Choose how:

• most recently updated (name the date column);
• most complete;
• oldest;
• source priority (name the source column and rank its values; an unranked source comes last).

Ties go to the first record read, so the same input keeps the same survivor every run.

Fill survivor gaps lets the survivor take values for its blank or placeholder fields from the rest of its group, building one complete record from several partial ones.

Ports

Port	Direction	Kind	Notes
in	in	data	Records to deduplicate.
out1 ... out10	out	data	Your named outputs. Only the ones you define are shown.
unique	out	data	Records that matched no scenario. Always present.
error	out	control	Receives the records when the node fails.

Examples

The same customer from two systems.

Start

CRM contacts

Web signups

Combine

Data Dedupe

Map columns

Load survivors

1. 1
   Write one rule per way two records can be the same

   Start with (Email) on its own, then add (Name and Phone) for records with no email.
2. 2
   Set a blocking key

   A key on postcode, country or the first letter of the surname compares only records that share it.
3. 3
   Name the outputs you act on

   For example exact for matches safe to merge and review for a person to check. Connect each to its own branch.
4. 4
   Pick the survivor and fill its gaps

   Most complete suits records from different systems. Turn on **Fill survivor gaps** so the survivor takes the phone number the other record had.

Three records in:

source	name	email	phone
crm	Jon Smith	Jon.Smith@acme.com	None
web	John Smith	jon.smith@acme.com	555 0101
crm	Ada Lovelace	ada@example.com	555 0199

The first two match on the email rule, because the email comparison ignores case and the dots before the @. They form one group; the more complete record survives, and Fill survivor gaps gives it the phone number. Ada matches nothing and leaves on unique.

Troubleshooting

Symptom	Cause	What to do
The run takes very long	No blocking key, so every record is compared with every other.	Add a blocking key.
Unrelated records are grouped	A placeholder value such as NA is not on the placeholder list, or a fuzzy threshold is low.	Add the value to the list, or raise the minimum similarity.
Phone numbers in different countries match	Matching compares digits, and two numbers share their last ten digits.	Add a blocking key on country or region.
Expected duplicates leave on unique	Their score is below review, or no scenario fits them.	Lower the threshold, or add a scenario.
The survivor changes between runs	The input order changed, and ties go to the first record read.	Use a survivorship strategy with a column, such as most recently updated.

Related nodes

Transform

Dedupe by key, when the key is already clean

Union

Combine the streams before deduplicating them

Data Profile

See how clean a column is before writing rules