and related: you'd be surprised how difficult it is to write a great web crawler. It's a classic "this looks easy!" computer science problem. It is *NOT* easy.