Open innovation in collaborative filtering


Netflix has just announced a $1 million prize to whoever can improve the accuracy of their movie recommendation engine. To enable people to design an improved recommendation engine, they’ve provided their users’ ratings of 100 million movies, an extremely valuable database. This harkens back to Canadian gold mining company Goldcorp’s initiative, whereby they publicly released the geological data on their properties, and set up a competition with prizes for whoever could give them the best recommendations on where to dig for gold. Other open innovation initiatives such as Innocentive match a whole series of people looking for innovation, again providing pre-specified rewards for meeting specific parameters. Some note that the prize will mean a lot of people work for free, and it’s arguable that if you can indeed do better than the other competitors, you’ll be able to make more than $1 million from it commercially anyway. The size of the prize indicates the value in enhancing the accuracy of collaborative filtering, as I’ve written about many times before. If Netflix can more accurately recommend a movie to its customers, the more likely they will stay with Netflix. For companies with other business models, greater accuracy directly impacts sales and revenue. More and more energy and resources will be going into this space. Netflix has chosen to combine two of my passions – open innovation and collaborative filtering – so I will be very interested to see the results from this. Details of the prize are at, which will provide a progress chart on how the competing teams are doing.

IBM drives open business


This is fantastic. IBM has announced that it will voluntarily publish its patent filings on the Internet. The upside is that it is taking the lead in creating a clearer, less confused, higher quality patent landscape. The (potential) downside is that it exposes its technology directions and strategy to its competitors as they develop rather than once they’re implemented. The great thing about this kind of leadership in creating openness and transparency in business is that it usually is a ratchet – once it becomes more open, it is very hard for it to go back. There are plenty of businesspeople who hate the idea of open business, and fight it in every aspect of how they do business. Too bad for them. We are shifting to a world of open business at a very tidy pace, and I for one think that’s a great thing. Embrace it, or have a tough and miserable time fighting the trend.

Microsoft spins off social networking site


Back in November 2003 I blogged about a social networking project within Microsoft Research called MyWallop. At the time, fuelled by what I call the first phase of social networking, I speculated that something like this could become an opt-in part of Windows, unleashing an extraordinary ability for people to create useful connections across all PC users. The social networking space has – for now – moved on to focus on the big succcess stories like MySpace, which are based on personal expression, social identity, and entertainment, rather than business or utilitarian links (a space which has not gone away and will return in a different guise). It turns out that Microsoft sat on the project for several years, then finally decided to spin it out, with the expectation that it would be able to do better outside Microsoft’s walls than within them. Given what’s happening with some of Microsoft’s other initiatives, a wise choice. Wallop has now raised $13 million in venture capital, and just released the product in beta, with the intent of taking it to a full release early next year.

Wallop has a significantly different business model to other social networking sites. They have decided to eschew advertising, despite the massive deal MySpace won with Google. Instead, they charge members for the ability to customize their personal space. The personal spaces in MySpace’s are very basic and difficult to personalize extensively. As other commentators have noted, Wallop’s positioning is closer to CyWorld, which has had massive success in South Korea and recently launched in the US. CyWorld enables members to personalize their spaces to mimic their own homes, or create fantasy rooms. However in Wallop, members can also make money – if visitors buy the entertainment or features they have selected to place on their site, Wallop takes just 30%, leaving the rest of the revenue to the member. The idea is that the space becomes a marketplace for a wide range of digital expression, across music, avatars, art, animation, and more. Significantly, the entire site was developed in Flash, which enables Flash developers to create and sell artifacts within the site. In short, Wallop is a very interesting experiment to see whether people will both want to build networks in this environment, and spend money on content there. I think it will do at least fairly well, as it represents a real alternative to the existing social networking sites, but the critical issue here is scale. How well it does will significantly impact which way the social networking space evolves from here, as companies uncover what business models work or don’t work, and copy and refine the ones that do well.

Giving structure to citizen journalism


Jay Rosen has just announced that Reuters is giving $100,000 to NewAssignment.Net, which provides a more structured model for open source journalism, bringing together amateur and professional media creators. The money will be applied to hiring a full-time editor for the site. Jay gives some background to why Reuters is choosing to support this project:

Part of the background to the gift is a speech given March 2nd 2006 by Reuters CEO Tom Glocer to the Online Publishing Association. It was called “The Two-Way Pipe.” Glocer said the news industry “faces a profound challenge from home-created content-– everything from blogging and citizen journalism to video mash-ups.”

In 2005 he and his colleagues were worried about a shift in power they saw coming, but “it was about the consumer as editor,” Glocer said. “You get the news you want when you want it, either pulled by something like an RSS feed or a Tivo box or pushed by the media company.” This was a legitimate demand. And while companies like his are still catching up with that demand “our audiences have already moved on-– now they are consuming, creating, sharing and publishing.” Consumers as producers! That’s a power shift more confounding than the explosion of choice.

His puzzle: “If users want to be both author and editor, and technology is enabling this, what will be the role of the media company in the second decade of this century?” As Scott Karp pointed out (his blog is about the next era in publishing) Glocer’s answers to that question weren’t very revolutionary.

But some of his observations were keen. “On the day the Tsunami struck, Reuters had 2,300 journalists positioned around the world, mercifully none were on those beaches,” he said. “On that fateful day we also had 1,000 stringers around the globe – but none of them were there either.” The only way to get the story was from amateurs to whom the tools of media production had been re-distributed. His conclusion: “You have to be open to both amateur and professional to tell the story completely.”

NewAssignment.Net is explicitly joining traditional and social media in creating a “pro-am”, or professional-amateur model. For example, amateurs can collectively be given fact-checking tasks, or issues identified by amateurs can be passed on to professional journalists. For a long time I’ve believed that what is happening and will happen is a merging and integration of traditional mass media and emergent social media. In our Future of Media Report, we explicitly discussed how the “professionalism” of traditional media both provides standards and expectations, and is also a limiting box. NewAssignment.Net looks like a particularly promising model, though the details are not yet clear. However irrespective of how successful it is, the space in which it is playing, of bringing together the resources and capabilities of amateurs and professionals, is where much of the action will be moving forward in the world of media.

[UPDATE:] Chris Ahearn, head of media at Reuters, has posted his comments on the deal. An extract:

Professionals and amateurs will cooperate to produce work that neither could manage alone, using “open source” methods to develop assignments and help bring them to completion. The idea is that people will contribute to stories that they believe will make a difference.

While the Internet is rapidly transforming the world of traditional media, it also presents amazing new possibilities in terms of strengthening the investigative arm of journalism. The Internet is the perfect vehicle for galvanizing the public to become more involved in reporting.

Can technology create world peace?


I was interviewed for a podcast today by the enormously energetic Sanjana Hattotuwa, for his ICT for Peacebuilding blog. Sanjana’s overview of our broad-ranging discussion is here, and the podcast is here.

While we covered a lot of territory, including reputation systems, getting traditional media to adopt social media technologies, global innovation networks, and more, the heart of the conversation was about technology’s ability to support peace and a better world. This is a particularly pointed issue in Sri Lanka, where Sanjana is based, with violence now on the rise again. Certainly social media in particular gives us all access to a far broader range of views and opinions. Yet this doesn’t necessarily change people’s attitudes. The increasing polarization of the political debate in the US, and arguably globally, suggests that the rise of social media is not a universal panacea. Claims were made when both the telegraph and the telephone were introduced, that they would help bring about world peace. In fact, it can be easier to dehumanize people and opinions when discussion is intermediated by technology, thus creating more extreme expressions of disagreement or even hate. For decades it has been recognized that a key factor in people’s social networks is how diverse or similar their connections are (what Everett Rogers called heterophilous and homophilous networks). Certainly being exposed to more diverse people and views can help to temper views, yet this often has to be in a face-to-face context, where it is far harder to focus on abstract differences and to ignore people’s humanity. Unfortunately, I believe there are some aspects of humanity that mean it will not be soon before we stop killing each other. I also don’t doubt that much hate will continue to be expressed in the online world. Yet part of the evolution of humanity that we are increasingly exposed to other cultures and views. Most countries around the world have reached the stage where they are truly multi-cultural. From being exposed to people further afield primarily through mainstream media and entertainment, the shift is now to direct interpersonal communication, and accessing voices, writings, and videos created by an extraordinary assortment of individuals. Greater appreciation and tolerance is, in the long run, a fundamental outcome of this (without expecting world peace overnight…).

Job advertisement: Exceptional Executive Assistant/ Professional Support


I am currently advertising for someone to work at the core of my team, based out of Sydney. I’m advertising online, through word-of-mouth, and it makes also sense to put this on my blog too. Please do pass this on if you know anyone you think may be interested. Blog readers are encouraged to apply! Special talents in online tech, PR, research, consulting processes etc. will have ample scope for application. Full job ad details below.

Read more

Interview on MySpace and social media


Yesterday I was interviewed on ABC Radio National’s Counterpoint program about MySpace and social media, together with Sebastian Chan, the web services manager for Powerhouse Museum – the interview is available for streaming and download (listen from 39:10). I talked about how social media is unleashing our latent humanity, by providing us with new ways of interacting and relating. We now have a multiplicity of new ways of communicating, all of them additional to the existing communication channels we have. We are exploring and discovering who we become as individuals and a society through using the new tools of social media. Second Life is a powerful manifestation of this, and we can expect to live significant parts of our life in alternative worlds. MySpace is a specific manifestation of how young people’s attention is shifting. They are spending less time on TV and traditional media, and more on their relationships. Companies can reach people within the social media where they spend their time, far better than through traditional advertising. Businesspeople are finding that they can build their networks online at least as well as they can in the usual cocktail parties. Gaming is becoming a social and relationship space, absolutely not just for young people and geeks.

Impressions of Influence Forum 2006


I’ve just got back from the Influence event, held in the lovely Hunter Valley wine region north of Sydney. A very interesting experience, and as far as I’m aware a unique event globally. Phil Sim has been running Mediconnect, doing events and an online PR exchange for some years now. These bring together Australian technology journalists with tech vendors and PR. The company is also involved in the Asian market. This year the event has been renamed Influence, and broadened to Include not just journalists, but also other influencers such as analysts and new media players. The influencers come for free, while vendors and PR pay handsomely for sponsorship and attendance. The structure is highly conversational, with a series of panels on both enterprise IT and consumer IT topics, where the panelists give a five minute intro which is then opened out into Q&A and discussion by the entire room. The reason the event is, as far as I’m aware, unique, is the scale of the Australian market. It is possible to bring together in one venue a critical mass of the tech journalist community. Most have been in the industry for a good many years, they mainly know each other, and they value the chance to get together.

I asked people there their perception of the value of the event. Several PR people said it was fantastic, as wherever else could you ever access 50 top journalists for two days? But the journalists get a lot of value and stories from the event too. The concept is great – value for everyone here as there are truly interesting discussions of where technology is today in bringing together the right people. It does take the right execution, too, of course, and Phil clearly has the network and respect it takes to pull it off. Everyone I spoke to at the event was extremely interesting. No duds. Phil also is immensely oriented to the conversational, which is a very welcome relief to all as still most Australian conferences have very traditional formats.

Just a few fragments of what struck me from what I saw and heard:

* Met William Pramana – I just found out his blog is ranked #17 on Technorati – not bad for a guy hanging out in Western Sydney… Over a lunch chat on what Web 3.0 will be like, he suggested that the ability to share site revenue with participants and contributors to the site will be part of it. Very thought-provoking.

* Duncan Riley, head of operations of the blogging network b5 media, said that 80% of their 135 bloggers are female. Great to meet Duncan at last and hear his views as a top-tier blog businessman. His first post on the conference includes his thoughts on my presentation and the Web 2.0 panel.

* In the blog world as much as in traditional publishing, people are concerned about editorial independence and credibility. This is a topic that will last forever. How is information paid for? Does that influence its credibility?

* Dan Warne of APC talked about how journalists blogging makes news more granular. Smaller pieces are more acceptable and expected. You don’t take the time to craft a longer piece until later.

* In the IT-Enabled Innovation session, Andrew Lamble of Dimension Data spoke about collaborative sourcing, multisourcing, alignment of objective, tapping innovation in high trust relationships. Hey, that’s what I talk about too. I’ll be spending a lot more time on this sort of stuff soon – good to see this is very solidly on the business agenda, as it should be. Some people in the session didn’t seem to understand the idea of open innovation. People had better get this soon – you haven’t a hope if you rely solely on your own internal innovation capabilities.

Unfortunately had to leave a bit over half way through the event to get back to other commitments. I’m sure I’m missing lots more good conversations. Great to see this sort of thing happening in Oz – look forward to the next one.

Web 2.0 and user filtered content


Tomorrow I’m heading off to the Influence conference run by Phil Sim’s Mediaconnect. The event brings together media and other influencers (I believe I’m labelled a “new media influencer” there) and corporates, discussing current trends in key technology sectors. I’m on the Web 2.0 panel tomorrow, so I thought I’d briefly capture here my introductory comments, on my chosen topic of

User Filtered Content.

The user filtering landscape

+ The primary focus recently has been on the explosion of user generated content, with Wikipedia, MySpace, YouTube and many others just the vanguard of an immense wave of content creation, unleashed by accessible tools of production and sharing. We are moving towards a world of infinite content, further unleashed by the vast scope of content remixing and mashups.

+ With massively more content available, we need the means to filter it, to make the gems visible in vastness of the long tail. Fortunately, Web 2.0 is in fact just as much about user filtered content as about user generated content.

+ As far more people participate in the web, as technologies such as blogging, social networking, photo sharing and more become easier to use, the collective ability of the web to filter content is swiftly growing, and will more than keep pace with the growth in content.

User filtering mechanisms

Clicks indicate popularity of specific content within a site (with many caveats).

Links are stronger and more valid votes on the value of content.

Ratings provide explicit opinions on quality.

Tags describe content with words, locations etc.

Web-wide and site-specific filtering

There are two primary ways of implementating user filtering: taking data from across the web, and from within one site.

+ Google’s PageRank is a seminal example of web-wide user filtering, where people’s aggregated linking behaviors enable people to find relevant content. Technorati more explicitly shows how many blogs link to other blogs or blog posts, to indicate their authority. Techmeme draws on the timing and relationship of new links to uncover current conversations.’s book recommendations kicked off site-specific user filtering, notably by identifying related titles. Slashdot was for several years the primary site that enabled communities to select stories and rate each others’ commentary.

+In two years has reached over 1 million daily visitors with its core model of user filtering of content. Copycats or similar sites such as Reddit, Meneame, and Shoutwire have abounded. Finally AOL-owned Netscape launched a Digg copy, providing mainstream media endorsement of the model.

+Content sites such as YouTube, Flickr, MySpace, and Odeo all embed user filtering as core features of their services.

What’s next for user filtering

+ Effective user filtering will have increasing value, and there will be more plays in this space. Network effects will apply strongly to site-specific filtering, however this will not preclude new players with better models gaining traction quickly. The move by Netscape to hire active raters away from Digg is an attempt to accelerate shifts.

+ Social search engines such as Eurekster and Yahoo!’s Search Builder indicate the next level of sophistication of search, enabling filtering aggregation of specific communities rather than the web at large.

+ Tools such as Last.FM and Yahoo!’s Launchcast will, with permission, use extremely detailed personal taste profiles to provide content filtering for individuals.

+ New mechanisms will emerge that draw on people’s web activities, tagging, specific communities, and combine these perspectives in various ways to create more refined user filtering. This filtering will increasingly be designed to be relevant to people with particular interest profiles and individuals.

A manifesto for the newspaper industry


Tom Mohr, formerly president of Knight-Ridder Digital before its sale in June, has just published Winning Online – A Manifesto, proposing that the US newspaper industry should merge into a single industry-wide network, at least for its digital assets. He suggests that there is $4 billion of additional revenue to be gained by 2010, primarily by gaining targetted advertising. Part of Tom’s argument starts from the fact that in the US market, all but a few newspapers are local. It is important to note that there are very different dynamics in just about every other country in the world, where the newspaper markets are dominated by national players, so this is a particularly US view. Other key aspects of his case are the combined power of the newspapers’ advertising salesforces, and the negotiating power they have collectively, for example being able to withhold their content from the content aggregators such as Yahoo!. However Tom’s last point is that this collective work will take leadership. Indeed. In a network economy, leadership is required to show the potential of collaboration, and to bring participants together to create and share collective value. Having closely studied similar situations across many industries (for example FXall, RosettaNet, XBRL, CPFR), I simply do not believe that the industry structure or the participants in the US newspaper industry will allow this plan to make any headway. Reflecting on Tom’s piece, Don Dodge thinks that newspapers and magazines will die as soon as their current readers die. No way. E-paper will revitalize them in a new form. Don is in a minority of people who are happy to read news and entertainment sitting in front of a screen or on a portable device. Once the current print publications are digital, they will have a new lease of life.