Every day I am amazed afresh by the transformative power of the Web. Today I have discovered Many Eyes, a site hosted by IBM’s AlphaWorks. It combines open participation with a wonderful set of visualization tools. As such anyone can upload data sets, and then create sophisticated visual representations of those data sets, including scatterplots, tree maps, histograms, bubble diagrams, network maps and far more. Anyone can then either reuse the data sets, create new visualizations, add comments, or blog about the visualizations. To try it out I created in around one minute a bubble diagram representation of the frequency of words in the English language (See below for the non-interactive diagram – I won’t link directly, as I think generating the diagram is rather resource-intensive – have a look at the visualization gallery that includes it). In the first edition of my book Developing Knowledge-Based Client Relationships, which was published in 2000, I wrote about both data visualization and concept visualization (which uses visual representations to convey concepts rather than information). Both of these will be fundamental in a world in which we are swamped with information. While I haven’t spent as much time on visualization over the last years, I am shifting back towards this space, not least in facilatating clients in easily understanding and responding to strategic issues.

In Tim O’Reilly’s very interesting post about the site, he asked Martin Wattenberg and Fernanda Viegas, the people who conceived Many Eyes, about their inspiration. Fernanda calls it ‘“social data analysis,” in which “playful, social exploration of data leads to serious analysis”. Martin says that their goal is to “democratize” visualization. These are seriously valuable tools being provided for free to the community at large, where one person can use the tools for their own purposes, then have their ideas be taken up and developed further by others. I’ve long used AlphaWorks as one of the best and earliest examples of open innovation. It’s great to see them both offering this kind of value to the community, and have this fully integrated into their business models. Note that another social data visualization site, Swivel, launched before Many Eyes – it doesn’t appear to have as rich visual functionality, as Brian Dennis notes, but has far more data sets uploaded for people to play with.

I have always said – particularly to those who don’t understand blogging – that blogs are not necessarily important individually, but in aggregate they are massively powerful. The “blogosphere” pulls together what millions of talented people around the world are discovering and thinking. Collectively, blogs enable us to collaborate to filter and uncover the most worthwhile news. As I wrote in my second book Living Networks, we are currently all participating in the birth of a global brain, and the world of blogs allows makes visible our collective stream of consciousness.

In that vein, Techmeme is one of the top three sites I refer to – often several times a day – to discover what is the most interesting technology news of the moment. It is an automated site that tracks a continually evolving list of the top few thousand blogs. It uses a complex algorithm to pull out both the most discussed news items or blog posts at that time, and the current conversations between top bloggers that these have sparked. Because of its exclusive scope, you can be sure that the comments are interesting and relevant. More importantly, the site uncovers conversations, discussions, points of difference, disagreements, creating a view on the news that is far more than the sum of its parts. Techmeme’s sister sites – memeorandum for politics, WeSmirch for celebrity gossip, and Ballbug for baseball – fulfil the same role for other topics. Memeorandum in particular provides fascinating insights into the American political debate, and how topics are viewed by partisans of both left and right.

Danny Sullivan of SearchEngineLand has just interviewed Techmeme founder Gabe Rivera, providing some fascinating insights into Techmeme. I was very surprised to discover that Techmeme has only around 30,000 daily unique visitors, a tiny amount compared to sites like Digg.com, with around 1.5 million per day. Apparently Techmeme is still a tool used by the cognoscenti, including many journalists who use it to discover news stories. However in my mind Techmeme and its sister sites rank alongside Technorati as the most valuable tools to uncover the power of the blogosphere. Joshua Jaffe of The Deal says he’s convinced that Techmeme will be acquired by Google for a stupendous sum. I certainly have no doubt that Techmeme or similar tools for tracking insightful online conversations will soon come to the fore.

Netflix has just announced a $1 million prize to whoever can improve the accuracy of their movie recommendation engine. To enable people to design an improved recommendation engine, they’ve provided their users’ ratings of 100 million movies, an extremely valuable database. This harkens back to Canadian gold mining company Goldcorp’s initiative, whereby they publicly released the geological data on their properties, and set up a competition with prizes for whoever could give them the best recommendations on where to dig for gold. Other open innovation initiatives such as Innocentive match a whole series of people looking for innovation, again providing pre-specified rewards for meeting specific parameters. Some note that the prize will mean a lot of people work for free, and it’s arguable that if you can indeed do better than the other competitors, you’ll be able to make more than $1 million from it commercially anyway. The size of the prize indicates the value in enhancing the accuracy of collaborative filtering, as I’ve written about many times before. If Netflix can more accurately recommend a movie to its customers, the more likely they will stay with Netflix. For companies with other business models, greater accuracy directly impacts sales and revenue. More and more energy and resources will be going into this space. Netflix has chosen to combine two of my passions – open innovation and collaborative filtering – so I will be very interested to see the results from this. Details of the prize are at netflixprize.com, which will provide a progress chart on how the competing teams are doing.

News is just out that Microsoft’s Zune mp3 player, due out before Christmas to compete with Apple’s iPod, will have social networking capabilities, in addition to its core features of a 30GB disk and a 3 inch screen. Zune users will have the option of using the device’s inbuilt WiFi to send and receive music, playlists, videos, and photos to up to four other players. They can either broadcast these to any Zune player within range, or only to those of their selected friends. If they have the broadcast feature switched on, anyone permitted within range will be able to listen simultaneously to what they’re listening to.

This social networking feature, together with the device’s WiFi capabilities, is the only really significant feature difference to the iPod, and it is one that actually could shift users to the Microsoft player. Particularly for young people, music is fundamental to their identity and relationships, and sharing music is truly at the heart of their social networks. Sharing around musical preferences was the initial premise behind MySpace, and while it has gone quite a bit beyond this, it was the seed and still is at the centre of the largest human social network ever to exist. However I have a few concerns about how the feature is implemented on the Zune. One is that broadcasting to four people is not enough to really enable true social networks. It makes it a little bit more a gimmick than a feature to have it so limited in scope, though it can still act as a social glue for smaller groups. Another issue is probably related, in that WiFi is very energy-hungry. I have not seen any figures on battery life with WiFi turned on, but I suspect that the device won’t be able to last very long while it is broadcasting music, making it far less mobile. It may take, sometime down the track, the use of fuel cells or alternative wireless technologies for this kind of music social networking to be a broadly used application. A final issue is that, given this is Microsoft, we know that there will be solid Digital Rights Management (DRM) in place. In fact the release specifically says that people will be able to share “promotional copies” of songs, which will be just a fraction of what people have available on their players. Given these factors, the question remains whether this feature will prove to be a key differentiator for the Zune here in an extremely competitive marketplace, but I don’t doubt that mobile musical social networking will get massive uptake at the right time, when it’s done right.

Google has just released maps for use on mobiles, that indicate traffic congestion with four color levels from green to red, across 30 US cities. This is one of those applications that has been obvious forever, and it’s only been a question of time until it’s implemented well (which is not quite yet). When people are navigating traffic and choosing alternate routes, they have until now been guessing which way to go, having available at best a trickle of information in from the radio. In fact, traffic information is one of the main reasons that people listen to local radio. Once you can get far superior traffic information from other sources, you might as well go to the radio that gives you your preferred music or talk, which is unlikely to be local radio. Next steps include not just current traffic intensity, but also predicted traffic intensity. As I wrote in my book Living Networks, UK company Applied Generics has a product called RoDIN24, that anonymously monitors the movement of mobile phones relative to cell towers in order to provide extremely detailed and live views not just of where traffic is slow (mobile phones moving slowly), but also where traffic is converging to. Beyond that, computers will be able to predict reasonably accurately how long different routes will take, enabling drivers to make route choices without gazing at screens too much. Of course, these predictive devices will play off against each other – if every one made the same recommendation to their drivers, that route in turn would become congested. But in the long run, in congested urban traffic we will see the different possible routes taken even out, so that it takes a similar time whichever of the major possibilities you choose. Resource Shelf gives an overview of other traffic data resources. The US dominates, with some services also in the UK. As with good GPS mapping, there will be a several year lag for effective mobile traffic services to reach most other developed countries. As with many of these applications, it is the cost of mobile data that is a key driver. Cheap mobile data in the US is driving these kinds of applications. In countries where mobile data is very expensive, including Australia, it will, unfortunately, take a long time for mobile applications such as traffic data to take off.