Big Data, big deal ?

The opportunities that "Big Data" can offer civilisation is immense.

However as a civilisation, applying our collective minds towards the greater good has never been our forte, progress in its broadest sense has been more akin to a clown on stilts (while being charged at by a raging bull) than an Olympic sprinter. So my take on human society is dear reader, rather cynical, but in stating that what grounds do I have to be so prejudiced ?

Well perhaps I am being over harsh,  a negative bias is, I believe, hard-wired into our everyday thinking and  I blame our ancestral past, where  dragons.. pestilence and or disease plus a potential rival to contend with, not forsaking the wrath of an unhappy deity (s) was the order of the day, so a pretty stressed existence whichever way you'd care to look at it, and from that crucible of conflict the spirit of an age or culture, has both fashioned and continues to fashion each successive generation.

Now "civilisation"  accrues a common "utility"  call it purpose, even if it has some inherent flaws such as the creation of "wine lakes" and  "butter mountains"that co-exist (elsewhere on the same planet) with childhood mortality associated with malnutrition.

Advances in aviation and space technology are also utility's driven by civilisation and have ushered in the advent of consumer electronics notwithstanding two world wars and the sacrifice of millions for its accelerated progress, hardly a promising example of progress except under duress, remember "there be dragons"?

In WW2 The Battle of Britain could also be considered as partly won due to a convergence of  information technology and communications (as it was) in creating a proto "Big Data" system.

Although the genesis for Big Data arose out of a need to rationalise huge data sets as generated by the ever growing convergence of disparate IT systems, the proving of Big Data has already offered spectacular opportunities. I mentioned the battle of Britain but to bring it up to the modern day  I dare say many Caribbean beach side bars have regaled numerous ex-bankers extolling their newly founded wealth gleaned from the assiduous monitoring of data trends signifying  the fiscally un-initiated into the collective slaughter house of "mis-sold" financial products.

Western economies are tottering towards collapse while remedial actions has that  fantastical euphemism of hyper inflation  - "quantitative easing" something that even two world wars could not achieve, well actually I stand corrected, Germany after WW1 did apply a similar mechanism and that fiscal policy helped engender the seeds of the next global conflict, hmm is it me or is there a distinct variance between advances in technology and advances in economics ? 

In managing a Charity e-commerce site,  the need to study three or more months worth of Face-book, Twitter, or Google analytical data in order to understand what potential consumers are talking and thinking about when visiting the site is key to our business model. However data warehousing with a suite of  mid range servers from which to manage or extrapolate the relevant trend lines was beyond our meagre budget.

To date, what we could afford was a standard shared hosting service, which continually had issues in both reliability, support, and after a major loss of service -when the site was down for over two weeks - the management board conceded that my proposal to migrate over to a dedicated  platform was now the most expedient path to recovery.


Even if the server didn't crash (as often as it did)  a visiting user to the site may still  be greeted by a BANDWIDTH EXCEED error web page. due to neighbouring spammers effectively hijacking the same Web host for their nefarious Junk mail activities.

So aside from the problems inherent with low cost hosting services  due to the prevailing economies of scale moving over  to a Cloud based platform / Big Data infrastructure,  has defiantly worked out in the favour of SMEs's, such as ours.

Charities  scrabble for "market" share, and I'm referring to demographic trend-lines which can determine a whole range of marketing and promotional strategies for good causes on to data feeds that can assist farmers in developing countries e.g. why should a coffee grower in Columbia be limited to his local market (and buyers agent monopolies et all) for sales when he could  promote and sell his produce internationally by the simple expediency of Internet access and community web site?

Such distribution of data services will empower the smallholder wherever they happen to be and help stop the food processing giants from hijacking the production chain and extrapolating profit margins at the expense of both producer and consumer,  consumers can start to utilise Big Data services in order to compare and contrast food prices, people of the world unite via your smart phones !

Similarly E-learning via the Internet over capable form factors and  cellular data networks can recover one to one learning where large class sizes such as in the developing world preclude such benefits from taking place.
A schoolchild can have  one device (laptop/Tablet) to learn upon for the duration of their schooling years saving ongoing costs  as in books and research material, to me this a beautiful vision.

Another example of Big Data being gathered over a period of time, involves studies conducted into human activity and the build up of green house gases into the atmosphere, which are critically close to the tipping point whereby the natural recycling of the gas by the planet simply cannot occur.

There are the seeds of optimism such as with The Human Genome project http://folding.stanford.edu/ which ably demonstrates how technology can be applied in tackling inherent disease, by the simple expediency of clicking a few icons http://greatergood.com/  to create some good in the world.

In general, humanity is more reactive than pro-active in developing technology, such as the economic collapse to force car manufactures to implement greener cars, oil based energy and transportation technology can only go so far and personally those times are now over,   the "data" has been telling us this for years, you don't need to be Einstein to figure out that pumping out the earth's lubrication system is unsustainable but greed and consumerism guided by marketers' and financiers who applied "Big Data" in a way to enhance personal gain are leading us up the garden path to a literal Hell on earth.

For over fifty years, the UK government has monitored the spread of disease in cattle and this wealth of statistical data has been turned on its head  to "justify" a recent decision to implement a cull on wild badgers. Even though the data says  there is only a 16% rise in Tuberculosis  when  badgers and livestock are in close proximity, the blinkers are defiantly on when the same data makes it clear  that a badger is as likely to be infected by the cattle as the other way round. So rather than taking a strategic approach to managing the spread of a disease, a tactical solution- proven not to have worked over the last fifty years is ignored- and implemented instead.

 Time energy and money completely wasted,  here is one example of Big Data having no effect when its against vested political lobbying groups, likewise with greenhouse gas and climate change issue.

We have a great resource in Big Data but risk short-changing the benefits it can have for humanity and the planet unless access is available not just to a select few but to those who would benefit the most.

 "Big Data is a natural resource" Big Data is now part of the human condition just as our ancestors utilised their habitat, Big Data should be an ecology we all share in to enhance our lives, the recent announcement that we now have over 7 billion people on this planet should not promote dread and fear over competition for natural resource's instead it should be utilised by a "Big Data World model"  where each birth contributes to a  mind pool which (no pun intended but darkly ironic) sufficiently nourished could bring about stunning innovation into our lives, but it's not the considered way.

 Big Data can easily absorb such events and turn them to humanities advantage, and that's the Key; unity and shared technological access plus innovative thinking on a literal world wide scale.

The fact is, what efficiencies are made with data mining systems as regards our food supply, power etc. are for the most part not being passed down the line in as equitable manner as they could be but disappear into gross profit margins instead.

Ultimately, no one will be immune from this misuse and its consequences. In the west we are all "quantified",  information about us is being traded 24/7 by who and for what reasons we are utterly unaware of.
That data has no IP (intellectual property) ascribed to it, and considering your activity has generated a value  in one form or another it most defiantly should have.

Something to consider as we gasp the last lung full of clean air from the atmosphere,  so what for "Big Data"?

How Big is Big Data ?
The future of energy ?