How to Write a Web Crawler in C# - ericsowell.comA few months ago I drastically changed how the urls on my site were built. I moved to using the ASP.NET 2.0 virtual path provider to make more friendly urls. See the discussions in April 2007 if you’re interested. There were several posts that month about it. One problem with a change like this is that it can wreak havoc on your urls, especially your relative ones. Using the url rewriting features
