Curation: The algorithm of our human essence

There is no doubt in my mind that the topic of curation and the Internet, is an important one and that it will be a dominant topic in 2011.

Curation is important because we are reaching the limits of what can be achieved through algorithms and machines in organising and navigating the Internet.

Aggregation looks like curation but it’s not.

I define curation as a person, or a group of people, engaged in choosing and presenting a collection of things related to a specific topic and context.

Aggregation employs software (algorithms) and machines (servers) to assemble a collection of things related to a specific topic and context.

Aggregation tools can be employed by curators but the human act of curation adds a layer of value that aggregation alone cannot provide.

A good example is Techmeme, the news aggregator run by Gabe Rivera.

Techmeme uses an algorithm to find and publish links to the most important tech news of the day.

For many years it was just Gabe and his algorithm. But now the site employs six people sifting through the results thrown up by Techmeme’s algorithm and looking for news that machine aggregation alone cannot find.

Techmeme added people because it can produce a superior, curated product, than by machine aggregation alone.

Contrast Techmeme’s curated approach with that of Google News’ aggregation. At the bottom of each Google News page you will find this statement: “The selection and placement of stories on this page were determined automatically by a computer program.”

And that’s the reason Techmeme wins out over Google News because the quality of machine selected stories is not as good as that of a team of people curating the news together.

The human element is important but it’s expensive, which is why Silicon Valley tech companies favour software and machines. Silicon Valley investors fund businesses that are “scalable”, that can be expanded by simply adding more servers and software.

People-based businesses are not scalable in the same way — you need to hire more people to do more. People add complexity, require management, vacations, and labour costs always trend up. In comparison, servers and software costs keep falling and their productivity increases.

The cheapest way to add a human layer is to improve machine aggregation by taking clues from what people do online, what they share, what they write, and what they discuss.

That’s why tech companies talk about the “social web” and social graphs. These are ways of improving their technologies without having to employ people.

A good example is Flipboard, the popular iPad application. It searches for content by looking at what a person’s Facebook friends and Twitter followers are sharing, and then publishing the content in an easy to browse magazine format.

However, such approaches reach a natural limit. Social networks share a wide variety of different content. Variety is ambiguous but curation needs to be focused.

Also, the technology of aggregation constantly trends towards becoming a commodity. It becomes a baseline rather than a differentiating factor. The value-add becomes the human layer.

And there are additional challenges to machine based aggregation: It becomes ever more easily gamed, as can be seen in the quality of Google search results and its unending battle with people seeking to manipulate the its results.

Google is forced to keep changing its algorithm in a bid to shakeout those that figured out how to game its system.

Techmeme, with its human curators, doesn’t need to worry about others gaming its algorithm because commercial spam won’t get published — except by its sponsors.

The evolution of the algorithm and machine can be viewed as a way to use technology to distill the human essence. What is it about us that cannot be qualified and quantified within an algorithm?

I see curation as a one part of that human essence, a natural human activity that cannot be matched by technology. And curation is where the value lies in improving the organisation and usefulness of the Internet.