Filtering out unwanted robot traffic
December 06, 2014
This week, we rolled out some changes to filter out some unwanted web traffic. While we want this site to be a useful resource to patients looking for prospective doctors, there are certain pages that we don't want crawled by bots, and we have indicated so via a widespread method known as robots.txt. It turns out that many of the visits to our site by robots do not abide by that convention. Consequently, we have implemented some changes to help restrict those visits. We're still in the early stages of these algorithms, and we're bound to be too aggressive in some ways, and not aggressive enough in others.
If you're a regular user seeing a page that your IP address has been banned, please let us know and we'll take a look. Please include your IP address when you contact us.