Python Dedupe with Composite Keys

Reference entry · last updated September 21, 2026

Deduplicating a list in Python while preserving first-seen order keeps a result list and a seen set or dict of keys, and appends a record only when its key has not been seen. When records are distinct only by a combination of fields, the key is a tuple of those fields. A set is an unordered collection with no duplicate elements, used for membership testing.[1][2]

First principles and definitions

Hashability

An object is hashable if it has a hash value that never changes during its lifetime and can be compared to other objects, through __hash__ and __eq__. Objects that compare equal must have the same hash value. Hashability is what makes an object usable as a dictionary key and a set member.[3]

Most immutable built-ins are hashable. Mutable containers such as lists and dictionaries are not, and immutable containers such as tuples are hashable only if their elements are.[3] This is why a composite key is a tuple and not a list.

seen = {("call_ended", "c1")}       # tuple key, fine
seen = {["call_ended", "c1"]}       # TypeError: unhashable type: 'list'

Set and dict membership

A set stores distinct hashable members and answers membership by hash. A dict maps hashable keys to values and preserves insertion order, and updating a key does not change its position.[1][2][3] Either can serve as the seen structure.

First-seen order

A set alone cannot preserve order, and a set alone also discards the records. The pattern pairs the seen structure with a result list: iterate in order, compute the key, and append the record only when the key is new.[1]

def dedupe(records):
    seen = set()
    result = []
    for record in records:
        key = record["id"]
        if key not in seen:
            seen.add(key)
            result.append(record)
    return result

Because the loop visits the input in order and appends on first sight, the output preserves first-seen order. A bare set(records) is not a substitute. With the dict records shown here it raises TypeError: unhashable type: 'dict', and with hashable records it keeps complete elements, loses the order, and does not extract the keys. The seen set holds only the keys, while the result list retains the first-seen records.

Composite keys

When one field is not unique, the key is a tuple of the fields that together are. The tuple is hashable, so it works as a set member or dict key.[3]

key = (record["event"], record["call_id"])

A tuple is immutable, so it cannot be extended in place. Build it with all the fields at once, or concatenate a one-element tuple. Note the trailing comma: (x) is just x, while (x,) is a one-element tuple.

key = key + (record["start_timestamp"],)   # concatenate
key = (record["event"], record["call_id"], record["start_timestamp"])

Two records with the same event and call but different timestamps have different keys, so both survive. That is the point of adding the field to the key: it turns an otherwise duplicate pair into a distinct one.

Check before insert

The membership test must run before the key is recorded. Writing the insert first makes the test always false, because the key is already in the structure by the time it is checked, and every record is then either dropped or kept by accident.[1]

# wrong order: the test can never be true
seen.add(key)
if key not in seen:
    result.append(record)

# correct order
if key not in seen:
    result.append(record)
    seen.add(key)

The same order matters with a dict, where the equivalent is to check if key not in seen: before assigning seen[key] = True.

Pass-through exceptions

Some records must be processed every time even when their key repeats, such as a streaming transcript update. The condition becomes an OR: append when the record is a pass-through type, or when its key is new. The key is still recorded either way.[1]

if record["event"] == "transcript_updated" or key not in seen:
    result.append(record)
seen.add(key)

Choosing a structure

Only membership matters: a set of keys.
You need a value per key, such as the first or last record: a dict keyed by the composite key.
One field is not unique: a tuple of the distinguishing fields.
Order is part of the contract: a result list plus the seen structure, never a bare set.
Splitting the text first: see Python String Splitting.

References

^ Python Documentation, "Data Structures," Sets. Free full text: docs.python.org/3/tutorial/datastructures.html
^ Python Documentation, "Built-in Types," Set Types and Mapping Types. Free full text: docs.python.org/3/builtins/stdtypes.html
^ Python Documentation, "Glossary," hashable. Free full text: docs.python.org/3/glossary.html