Search and Ye Might Find

Adam Rogers is a senior tech correspondent at Insider – and he’s been thinking and writing about what’s known in the industry simply as “search.” For the last decade, people have been grumbling about not being able to find things online, both in our private data and on the public web, despite ever-evolving algorithms. Ever since humans started writing stuff down, the struggle has been in how to organize it all so that its contents wouldn’t be lost in the stacks. Search has always been an attempt to fix that problem.

And long before we could search for things online, Google was essentially a person: a reference librarian. If you wanted to find something on, say, growing vegetables – you could go to the gardening or farming sections of the library. But in all the thousands of books in that huge section you’d quickly get overwhelmed. That’s where reference librarians and archivists come in. They’d take your topic and help you narrow it down even further, applying their own nuanced knowledge and specialized training to help you search better and find exactly what you’re looking for. That’s how search operated for centuries – by topic, mediated by human to human interaction.

Once the web started spawning search engines, all of that began to change, largely for the better, at least initially. But even in the early 2000s Google started getting bogged down by monetization and people trying to game the system. One way this played out was with SEO, or Search Engine Optimization. which is basically the set of tools for getting a web page ranked higher in specific search results. In the early years of SEO, webmasters would often stuff their webpages with keywords to get them to rank higher, regardless of actual relevance – and the strategies keep evolving. As the internet expanded, more and more spammy or irrelevant web pages would come up in search results. Google’s AI helped filter those out by emphasizing high quality results, like dot edu links, or BBC articles, for example — these would rank higher in a search result than, say, an indie blog, in part because many other sites would be linking to the more well-known pages. This system worked, at least in part, and for a while. But spammers started to figure out workarounds, resulting in a kind of continual arms race between ill-intentioned actors and Google’s algorithms.

For more on this back and forth, and the problems that still persist, check out Why Search Sucks by Adam Rogers. In the piece, he goes into another big problem with the “private” side of search — people, for instance, searching their own email. Without a wealth of data to come through that’s publicly available, private search AI is slow to learn. And private searches are becoming more and more essential as more and more of our emails, photos, and memories get stashed away.