This post is a bit long, but it ends with why your personal life will be spied upon by your company’s competitors.  Curious?  Read on.

Information – data turned into meaning – is going to (further) disrupt just about every industry there is.  Many of the strong will become weak.  Tiny upstarts, like Davids, will topple Goliaths.  And this could all happen at the expense of your personal privacy.

I know, I know, the “information revolution” has been predicted for decades.  Except that like many predictions, exactly when & how the predictions come true will differ from what’s first assumed.  “Video calling” has been predicted as inevitable since the 1950’s.  But nobody saw Skype as the inevitable means for it to come true.

To see the future, let’s look at what has happened on Wall Street and how the same is about to happen in other industries.

What’s happened on Wall Street

Fortunes on Wall Street are made through arbitrage.  Years ago, the winners were those who had better research teams: who had the better information about a company’s stock?  Currency arbitrage followed: who can spot temporary asymmetries in currency prices and execute a trade the fastest?

Across every tradable instrument, the arbitrage game on Wall Street has moved from high-latency (you know something I don’t, for days or weeks at a time) to near-zero-latency.  New fiber optic networks have been built to shave milliseconds from the average trade.  And entire data centers have been relocated to close proximity to these new networks.  Whatever advantage you have must now be exploited in real time.  Or it’s not an advantage.

As the arms race of zero-latency arbitrage has unfolded, two other trends have been its critical enablers.  The first is data acquisition.  Every trade on every exchange can be captured and analyzed, alongside reams of other data about the companies, markets and countries those pertain to those financial instruments.  The scope and scale is increasing by orders of magnitude.

• An example of scope: the latest algorithms even ingest Twitter feeds in real time to discern investor sentiment.  Including your latest Tweet about some stock you like
• An example of scale: banks and securities firms store more information per company than any other industry (see this great report from McKinsey)

The second enabler is the ability to make sense of the collected data.  No longer do freshly-minted MBA’s toil into the night to build Excel models with quaint calculations like “EDITDA” or “Return on Invested Capital”.  Today, PhD statisticians build algorithms so complex that they themselves have trouble making sense of them in use.  Michael Lewis’ “The Big Short” is a great read on this topic.

From Wall Street to Main Street?

Wall Street started its journey by going on a data collection binge, from which it developed its algorithms, from which it automated its trading such that milliseconds mattered.

It’s happening now in online consumer businesses.  Wonder why Google, Facebook et al are under such scrutiny for collecting your data to the point of privacy invasion?  Because they understand that data is the raw material that feeds their statisticians, that feeds their algorithms, that feeds their money-making.

Except “money-making” in this case is the price they can charge an advertiser for an ad on one of their web pages.  The more relevant the ad to a consumer’s interests, the higher a premium the ad will command.  How to discern consumer interest?  Profile the heck out of them.

It’s happening elsewhere too.

The “Measurement Wars”

New companies, or ones that successfully reinvent themselves, will start their innovation and disruption journey by gathering reams of data, then finding the relationships between that data.

The forthcoming data acquisition binge is going to amplify online privacy issues.  Collecting competitive intelligence will border on spying on other companies’ employees.

It will also tempt companies to sell their internal data to others.  Could we see new “data merchants” emerge?  Would you like to know the kilowatt-per-hour energy consumption for every household in America?  Someone would, and would like an electric utility to sell them that data for some commercial advantage.

Orwellian, or progress?

Every boundary that separates individuals, companies, cultures and countries will be subject to elimination or reduction.  Our ability to learn, empathize and understand differences across these boundaries will be exponentially enhanced.  Which is good.

The ability to use that same understanding to exploit others will also increase exponentially.  Which is bad.

My take?  We have dealt with many past innovations that could be used to exploit someone or something.  Each time, a new equilibrium was established and humankind moved on.

But it was the period of rapid transition, and resulting destabilization, that was the most dangerous.  We are in such a period now, I reckon.

Governments must understand the coming hunger for massive data collection, and act to mitigate the risks, if we are to emerge on the other side unscathed.  Or even emerge better off.  But what are the risks, and what are the remedies?

Spy vs. spy

We’re seeing the privacy issue play out now in the consumer sector.  Facebook has been scolded by the Federal Trade Commission for its privacy abuses.  Politicians in the European Union search for ways to legislate much more stringent consumer protections.  I won’t cover this ground because the media has done so many times over.

But we have yet to come to grips with the implications of how companies will compete with each other in the Measurement Wars.  The old rules were about the theft of intellectual property.  But what about when a company profiles its competitors’ employees?  Such as their Tweets, Facebook activities or LinkedIn profiles?  Individual persons will be deeply profiled as part of compiling a dossier of competitive intelligence gathering.

This is where it gets creepy.  Heck, if you were imaginative, you might think that things I write about in my blog pertain to issues I experience at work.

And you wouldn’t be all wrong.

Credits:

I have borrowed from the great work done by others.  In particular, Michael Lewis and McKinsey as cited previously.  Also, O’Reilly Media for their extensive coverage of the underling technologies of “Big Data” and their Big Data’s use in consumer online businesses.