We noted in an earlier blog post that “Big Data” handling is becoming more practical for mainstream organizations, thanks to new offerings from, amongst others, EMC, Microsoft, IBM and Oracle. But society’s needs for innovation must be balanced with privacy and other issues.

An interesting blog item from EMC’s CTO, Chuck Hollis recently pointed out that Big Data is now becoming – or should be – a major part of the public policy agenda, even figuring on the agenda at Davos last week. The more data is being generated, and the easier it gets to process that data, the more that companies and society can potentially benefit. For example, individuals and society benefit from science and medical research. Companies benefit from being able to design better products or better customer service leading to improved brand loyalty. But at the same time,Big Data technology makes “Big Brother” surveillance and privacy intrusion easier.

Coincidentally, last week also saw the unveiling by EU Justice Commissioner Vivane Reding of major reforms proposed for the EU’s 1995 data protection rules, aiming to strengthen online privacy rights. These include a ‘right to be forgotten’: people will be able to delete their data if there are no legitimate grounds for retaining it; and the proposal that EU rules must apply if personal data is handled abroad by companies that are active in the EU market and offer their services to EU citizens. Google, Facebook and Twitter are no doubt top of mind here, but all Cloud service providers will be looking hard at this in particular. But all organizations who collect and use personal data

Of course these new rules are not driven by “Big” data per se. But the rise of online data is certainly a major force behind them, and the increasing desire and capability to store and analyse – say – information about consumers’ desires and behaviours.

Meantime, EMC – which has done more to popularise the term and concepts of Big Data than probably any other firm – helped to bring Big Data management and processing even closer to the mainstream this week, with an announcement that it has built the “Hadoop” big-data file system into its Isilon storage product. This combines with its Hadoop-enabled Greenplum analytics system to provide a powerful integration of storage and analysis capability. We see this as an attractive proposition for many organisations.

So – whatever your views on the pros and cons of storing and processing ever larger amounts of data, it continues to be the case that it’s getting easier to do in an enterprise context – to store and retrieve, to make robust solutions and to analyse what you’ve got.

And for many companies, that’s good news. Technology like EMC’s can give important competitive edge. It’s up to the authorities to decide how it should be regulated – and products like EMC’s just serve to make it more important or urgent that they do.