Cleaning steps
- Trim spaces and quotes, drop empty lines and anything that isn't an http(s) URL. In a CSV, the first URL on each row is used.
- Normalize conservatively: lowercase scheme and host, remove default ports and #fragments.
- Optional: remove tracking parameters, sort query parameters, unify the trailing slash, force https.
- Remove duplicates after normalization, keeping the first occurrence.
Frequently asked questions
Why is normalization conservative?
Paths can be case-sensitive and some parameters change the page. Only changes that never alter the destination are applied by default.
Is there a size limit?
Lists of hundreds of thousands of lines work in a modern browser. Nothing is uploaded.