Improving the back end
February 17, 2013
One of the major things that we've been working on over the last couple of weeks has been the performance of our back end processes to understand data. There are actually a number of steps that our data crawlers have to go through before we can enable searches across the pages that they visit. Our crawlers need to visit the pages, extract relevant data, normalize the data, figure out which medical provider is being referred to, and store the normalized data. And repeat. Again and again. Our data coverage has grown to such an extent that we recently have had to remove a number of bottlenecks in order to ingest the data in a timely fashion.
It's often difficult to appreciate changes in the back end given that there's nothing visual we can point to. Over time, however, we expect that these improvements will enable us to scale better and to hone our algorithms.