This tool is used to remove duplicate URL and it is also a good filter. I used it when I do collecting sites to campaign and for SEO.

Duplicate Remover/URL Filter