Deep Web Search: Still Crappy after all these years

Deep Web Search: Still Crappy after all these years

It’s probably safe to say that we all love Google. A basic Google search is so effective and so danged useful that we mostly overlook the things Google DOESN’T do. Google gets blocked by anything that isn’t clean HTML or PDF. It can’t fill in a form to query a database. It can’t go past anything requiring a fee, a subscription, or a password. The part of the web that Google can’t reach is called the Deep Web, and it happens to be non-trivial. Best estimates put it around 500 times larger than the Surface Web, in terabytes. But that’s not what actually matters. What’s important is that much of the highest-quality content is housed there. There’s good stuff underneath the Surface Web, that we wish we could get at.

Every search is different, but depending on what we’re after, it’s always easy to recognize high-quality hits as opposed to low-quality hits. For example the last serious search I did (several hours) was learning about cloud backup and storage. Specifically I was shopping for an online backup provider which would also work well for retrieving encrypted & passworded documents using any Web-enabled device while traveling. This is a new hot area, with a lot of buzz, a lot of new startups, lots of flavors available, a ton of competition. Hard, in other words, to cut through the fog. In this case, a high-quality hit would be (more…)