I recall reading statements that suggested our profession was asleep at the wheel during the period when Google emerged as the dominant search engine. Why, some asked, did librarians fail to develop a better search engine since we are search experts? Well those folks may have overlooked just how much capital is needed to actually start a new search engine venture. Let’s just say it’s more than most academic libraries have in their budget for new, entrepreneurial ventures.
For some insight into what is involved in trying to develop a competitive search engine – one that can beat Google at its own game – read the New York Times article on “Looking for the Next Google” (originally appeared on 1/1/07 – sorry – no longer freely available) about some Silicon Valley search engine startup firms. There are two pieces of information that suggest the difficulty in taking on this task (these are taken from the article):
Since the beginning of 2004, venture capitalists have put nearly $350 million into no fewer than 79 start-ups that had something to do with Internet search, according to the National Venture Capital Association, an industry group.
Of dozens of search start-ups that were introduced in recent years, none had more than a 1 percent share of the United States search market in November, according to Nielsen NetRatings, a research firm that measures Internet traffic.
It’s easy to criticize academic librarians for failing to develop some sort of competitive search engine – or even for failing to come up with a good idea for one. But when you consider the odds against the success of such ventures – and even if you made the wildly optimistic assumption that a venture capital firm could be convinced to provide funding for a library-related entrepreneurial search engine (or even if we choose to band together and fund it ourselves) – it still seems far more sensible for academic libraries to hitch their wagon to the market leaders. In doing so we need to be clear about our end goals and how any partnership can further our achieving those goals.
That said, maybe we need to keep our eye on Powerset, Hakia, and other start-up engines that think they can be the next Google. Perhaps one of these new potential competitors of Google will be interested in doing more for academic libraries than just digitizing our content so that it can be used to increase advertising revenues. But the reality is that getting a share of that revenue pool (the market for Internet search is valued at $141 billion) is what the race to be the next big search engine is all about.
I have mixed feelings about this. Why couldn’t libraries pool their resources together to create a viable alternative? Sure there is an opportunity cost there, but what are you weighing this opportunity against? Google is built on grids of computing power … seems like an interesting model for pooling resources. And where are google’s roots … did they have all the VC money before they got started? VCs gave them scale and expanded their scope. Did they fundamentally improve the search technology? Perhaps, but I seriously doubt it …
“it still seems far more sensible for academic libraries to hitch their wagon to the market leaders” — and will those private sector market leaders uphold the values that we all too often take for granted in librarians? Will the nice ones stay nice? Please don’t approach this opportunity with a defeatist attitude and don’t give up yet!
Your truly,
One Naive Soul?
I’m not sure this is entirely accurate.
Most of the capital expenditure for a search engine is in the _hardware_ and physical infrastructure neccesary to support a bazillion users at once–as Google does, and as anyone that wants to challenge Google wants to do. There’s a bit of unusual hardware needs in order to index ‘the whole internet’, but probably within the realm of much more feasible for a small organization—certainly the huge amounts of hardware Google has are mostly becuase of the number of concurrent users it wants to support, not the size of the index.
A library, of course, isn’t neccesary trying to support umpteen million concurrent users. And Google when it started wasn’t either, it started fairly small and scaled up.
These new challengers might be spending lots of money on “R&D”, but the basic idea behind google was invented by it’s inventors while they were in grad school, getting maybe a grad student stipend, the end. Without VC money.
Now, to challenge Google in the commercial marketplace succesfully is a HUGE thing–Google’s success has raised the barrier of entry to _compete with Google_, and to serve umpteen million concurrent users, for sure. If you can pull it off, it also has a huge financial prize. Which is why the commercial marketplace is putting lots of money into it.
But that doesn’t mean that search is an inherently expensive or resource-intensive endeavor. It’s not.
Now, I’m not sure that the library community should have been expected to develop an _internet search_ solution, if that’s what we’re talking about. But I don’t think lack of capital is what it’s about.
As we go forward we will see many different strategies employed to improve search in a variety of domains. Some might argue that is already the case… Google’s web search strategy relies heavily on link analysis on a massive scale. For libraries the current challenge IMHO is exploiting our presumably rich structured data, not network mining, at least not yet. So if all this VC money is focused on the public web search problem, where presumably big $$$ can still be made, then I’m not convinced that the academic community will necessarily see big gains in the search space in the short term. For example, it’s possible that the market leaders will continue to develop amazing technology that will not really help people make sense of large piles of digitized texts (assuming that is a problem we want to solve).
I think academic libraries should look both inside and outside its community for search solutions. The key is to find companies that are building tools that can solve problems in our domain. And even if you find these companies, what kind of economic model is needed to make such partnerships successful? “Goodwill” just doesn’t cut it.