How to Find Duplicate Customer Addresses

To find duplicate customer addresses, standardize every address first, then group rows that share the same standardized street line, unit number and 5-digit ZIP code. Comparing the text as typed misses most duplicates, and fuzzy text matching joins different suites in one building. In our 50-row test, grouping on sthan.io's standardized address and unit found 66 of the 84 duplicate pairs and made no false merges.

Why the text as typed is not enough

One office can be typed many ways: "233 South Wacker Drive, Suite 3300", "233 s wacker dr" with "ste 3300" on a second line, or "233 S. Wacker Dr. #3300" with no ZIP code. People read these as one place. A computer comparing text sees different strings.

The usual fixes work on the text. A normalizer lowercases everything and swaps a few words for their short forms. Fuzzy matching scores how alike two strings look and joins pairs above a cutoff. Both still compare text. Standardizing asks address data what the address is, so every spelling of one place comes back as the same line. Our address standardization guide covers what changes in that step, and the address parser guide covers splitting one address field into parts.

The test set

We built a small test file by hand: 50 rows for 13 places, all office towers, government buildings and landmarks. No private homes. We decided the true groups before running anything: 11 places appear more than once and 2 appear once, which gives 84 true duplicate pairs (two rows that are the same place). The file is constructed, small and only illustrative. A real customer list will have its own mix of mistakes.

In all, 9 rows have no ZIP code, 5 have a typo in the street name, 4 put the suite on its own line and 2 are exact copies of another row. The two places that appear once are there on purpose: "1 Market St" looks a lot like "1355 Market St", and "1 Wall St" looks a lot like "11 Wall St".

Four ways to match

Each method below turns every row into something to compare, then puts rows that match into one group.

• (a) Exact match. The full address as typed, joined into one line.
• (b) Simple normalizer. Lowercase, drop periods and commas, then swap 15 words and the # sign for short forms (street to st, avenue to ave, north to n, suite and # to ste, and so on).
• (c) Fuzzy match. token_sort_ratio from the Python library rapidfuzz 3.14.5 (Python 3.13.9), with processor=utils.default_process turned on, which lowercases the text and strips punctuation (it is off by default). It sorts the words in each string and scores how alike the two are from 0 to 100. Any two rows at or above the cutoff join one group.
• (d) Standardized address + unit. Each row sent once to the sthan.io US Address Verification API. The match key is the standardized street line without the unit, the unit number, and the 5-digit ZIP. Rows the API could not confirm are kept out of the groups and sent to a person (more on that below).

To score them we count pairs. Precision is the share of pairs a method called duplicates that really are duplicates: low precision means records of different customers get merged. Recall is the share of the 84 true pairs it found: low recall means duplicates stay in your list.

Results

Exact match and the normalizer never merged two different places, but they found only 2 and 18 of the 84 pairs. A missing ZIP or a state written out was enough to keep two rows apart. Fuzzy matching found far more pairs, but 42 of its 111 were wrong. Standardizing kept precision at 100% and found 66 pairs. Its 18 missed pairs all come from 6 rows it sent to a person instead of guessing.

No cutoff fixed fuzzy matching on this file. Lowering it found every true pair and kept every false merge; raising it lost true pairs faster than false ones:

The false merges

All 42 wrong pairs from fuzzy matching at cutoff 85 fall into four kinds:

The suite pairs are the dangerous kind. Different suites can belong to different customers, so a small difference in the text is no reason to merge them, and a string score sees only one changed character. Even at cutoff 95, fuzzy matching still joined 10 pairs of rows from different suites. The standardized key kept them apart, because the unit is its own part of the key.

What standardizing did to the rows

Some rows from the test, sent to the sthan.io verification API on October 16, 2026:

ConfirmedPrimaryOnly means the building was confirmed but the suite or apartment number was missing or not confirmed. In these two rows no suite was typed, for 200 N Spring St or 350 5th Ave.

Two details shaped the match key. First, Suite 300 and Suite 900 at 1355 Market St came back with the same ZIP+4, 94103-1337, so the ZIP+4 alone cannot tell those offices apart; the unit has to be in the key. Second, the same 350 5th Ave address came back with ZIP+4 0110 when the ZIP was typed and 0100 when it was not, so the key uses the 5-digit ZIP. USPS Publication 28 lists Suite as STE and uses # when the kind of unit is not known; the script keeps only the unit number, so "Ste 3300", "Suite 3300" and "#3300" give the same key.

Rows sent to a person

Six rows did not get an answer the script trusts, so it left them out of every group:

The last two rows show why the status matters. Chicago City Hall typed with "LaSalle" as one word came back in Wenona, a different Illinois town, with status Unknown. "121 N La Salle St, Chicago" came back correctly. If the script grouped every answer, the two Wenona rows would form their own group at the wrong address: on this file that scores 71 of 84 pairs with no false merges, but the group's address is wrong. So the script groups only rows with status Confirmed, ConfirmedPrimaryOnly or NotDeliverable. We set that rule after reading these answers, and we report both runs here.

The script

The script reads a CSV with the columns address1, address2, city, state and zip, sends each distinct address to the API once, one call at a time, and writes your rows back with six new columns: standardized, unit_used, status, match_tier, duplicate_group and group_size. It saves every raw answer in a responses folder, so running it again costs no calls. If the API returns no unit but the row has one typed, it uses the typed unit; in this test the API returned the unit for all 16 rows that had one.

pip install requests
export STHAN_API_KEY="your_key"
python dedupe_addresses.py customers.csv grouped.csv

"""Standardize every address in a CSV with the sthan.io verification API,
then group rows that share the same standardized address and unit.

Usage:
    set STHAN_API_KEY=your_key          (Windows)
    export STHAN_API_KEY=your_key       (macOS, Linux)
    python dedupe_addresses.py customers.csv grouped.csv

The input needs the columns address1, address2, city, state, zip
(address2 and zip may be empty). Every raw API answer is saved in
./responses so a second run does not call the API again.
"""
import csv
import hashlib
import json
import os
import re
import sys
import time
import urllib.parse

import requests

API = "https://api.sthan.io/v2/address-verification/usa/"
CACHE_DIR = "responses"
MAX_CALLS = 60          # stop after this many API calls in one run
PAUSE_SECONDS = 1.0     # wait between calls, one call at a time
# Only answers with one of these statuses are used for grouping. "Unknown"
# and no-match rows go to a person instead (column status says why).
TRUSTED = {"Confirmed", "ConfirmedPrimaryOnly", "NotDeliverable"}

UNIT_RE = re.compile(
    r"(?:#|\b(?:ste|suite|unit|apt|apartment|fl|floor|rm|room)\b\.?)\s*#?\s*([a-z0-9-]+)\s*$",
    re.IGNORECASE)


def one_line(row):
    """Join the address columns into one line the API can read."""
    street = " ".join(p.strip() for p in (row["address1"], row["address2"]) if p.strip())
    place = " ".join(p.strip() for p in (row["state"], row["zip"]) if p.strip())
    return ", ".join(p for p in (street, row["city"].strip(), place) if p)


def typed_unit(row):
    """The unit as the customer typed it, used only if the API returns none."""
    for part in (row["address2"], row["address1"]):
        m = UNIT_RE.search(part.strip())
        if m:
            return m.group(1).upper()
    return ""


def standardize(address, key, calls):
    """Return the API's Result for one address, from the cache when we have it."""
    os.makedirs(CACHE_DIR, exist_ok=True)
    name = hashlib.sha1(address.encode("utf-8")).hexdigest()[:16] + ".json"
    path = os.path.join(CACHE_DIR, name)
    if os.path.exists(path):
        with open(path, encoding="utf-8") as f:
            saved = json.load(f)
        return (saved["body"].get("Result") if saved["status"] == 200 else None), calls
    if calls >= MAX_CALLS:
        raise SystemExit(f"Stopped: reached {MAX_CALLS} API calls.")
    url = API + urllib.parse.quote(address, safe="")
    for attempt in range(3):
        time.sleep(PAUSE_SECONDS)
        resp = requests.get(url, headers={"Authorization": "Bearer " + key}, timeout=120)
        calls += 1
        if resp.status_code == 429 and attempt < 2 and calls < MAX_CALLS:
            wait = resp.headers.get("Retry-After", "60")
            time.sleep(int(wait) if wait.isdigit() else 60)
            continue
        break
    try:
        body = resp.json()
    except ValueError:
        body = {"text": resp.text}
    # 200 = an answer, 404 = "Not Found": both are final and are cached.
    final = resp.status_code in (200, 404) and isinstance(body, dict)
    if not final:
        # keep a failed call (bad key, rate limit, server error) for a look, not as cache
        path = path[:-5] + f".failed-{resp.status_code}-{int(time.time())}.json"
    with open(path, "w", encoding="utf-8") as f:
        json.dump({"input": address, "status": resp.status_code, "body": body}, f, indent=2)
    if resp.status_code != 200 or not isinstance(body, dict):
        return None, calls
    return body.get("Result"), calls


def match_key(result, row):
    """Standardized street line without the unit, the unit, and the 5-digit ZIP."""
    if not result or not result.get("fullAddress") or not result.get("addressLine1"):
        return None, ""
    if result.get("deliverableStatus") not in TRUSTED:
        return None, ""
    line = result["addressLine1"].upper()
    unit = (result.get("unitNumber") or "").upper()
    utype = (result.get("unitType") or "").upper()
    suffix = " " + " ".join(p for p in (utype, unit) if p)
    if unit and line.endswith(suffix):
        line = line[: -len(suffix)].strip()
    if not unit:
        unit = typed_unit(row)
    return (line, unit, result.get("zipCode") or ""), unit


def main(src, dst):
    key = os.environ.get("STHAN_API_KEY")
    if not key:
        raise SystemExit("Set STHAN_API_KEY first.")
    with open(src, newline="", encoding="utf-8-sig") as f:
        rows = list(csv.DictReader(f))

    calls, groups = 0, {}
    for i, row in enumerate(rows, 1):
        address = one_line(row)
        result, calls = standardize(address, key, calls)
        k, unit = match_key(result, row)
        result = result or {}
        row["standardized"] = result.get("fullAddress") or ""
        row["unit_used"] = unit
        row["status"] = result.get("deliverableStatus") or "No match"
        row["match_tier"] = result.get("matchTier") or ""
        row["_key"] = k
        if k:
            groups.setdefault(k, []).append(row)
        print(f"{i}/{len(rows)} {row['status']:<22} {row['standardized'] or address}", flush=True)

    next_id = 1
    for members in groups.values():
        if len(members) > 1:
            for r in members:
                r["duplicate_group"] = str(next_id)
                r["group_size"] = str(len(members))
            next_id += 1

    extra = ["duplicate_group", "group_size"]
    out_cols = [c for c in rows[0] if c != "_key" and c not in extra] + extra
    with open(dst, "w", newline="", encoding="utf-8") as f:
        w = csv.DictWriter(f, fieldnames=out_cols, extrasaction="ignore")
        w.writeheader()
        for r in rows:
            r.setdefault("duplicate_group", "")
            r.setdefault("group_size", "")
            w.writerow(r)
    print(f"{next_id - 1} duplicate groups; {calls} API calls this run; wrote {dst}")


if __name__ == "__main__":
    if len(sys.argv) != 3:
        raise SystemExit("usage: python dedupe_addresses.py input.csv output.csv")
    main(sys.argv[1], sys.argv[2])

The first run on the 50-row file, before we added the status rule, made 48 calls (the two exact repeats were sent once each). A second run with the rule read the saved answers, made no calls and ended with:

11 duplicate groups; 0 API calls this run; wrote grouped.csv

A shared standardized address makes two rows a candidate match, nothing more. Two companies can share a suite, and one customer can have two offices. Before you merge records, confirm with another field such as a customer number, a name or an email. Our real estate data guide uses the same rule for property records.

Sources

All read October 16, 2026.

• sthan.io US Address Verification API, 48 live calls for the 50 test rows, October 16, 2026 (every raw response saved with this article's working files).
• rapidfuzz fuzz.token_sort_ratio documentation: rapidfuzz.github.io/RapidFuzz/Usage/fuzz.html; package page: pypi.org/project/rapidfuzz.
• USPS Publication 28, Appendix C2, secondary unit designators: pe.usps.com/text/pub28/28apc_003.htm.
• sthan.io API documentation (free tier limits, API key header): sthan.io/docs.
• sthan.io US pricing: sthan.io/pricing/united-states.
• sthan.io Address Cleaner: sthan.io/address-cleaner.

Prices change; check the pricing page before you buy.