What is Big Data?

Big Data

I remember manually plugging 16kB (that’s right “kilobyte”) memory chips into an original IBM Personal Computer.  The crunching sound they made was unforgettable.  I was ecstatic that I had increased the total memory to 128kB, and then to 256kB … WOW!   At that time that was a lot of memory.  Who could ever need to store that much data?  That was “Big Data“.

Let’s fast forward to today.  The memory on your handheld smartphone and other devices is measured in Gigabytes (GB).  Over 2.5 quintillion bytes of information are generated every day.  Every move, every transaction, every image, every event and every location in everyone’s  lives are recorded.

That is Big Data!

How Big is Big Data?

There are so many pieces of information flying around the electronic universe that the numbers are staggering.  Did you know that across the planet:

Big Data
  • We send almost 200 billions emails every day
  • Over 2.5 billion pieces of content are processed on Facebook every day
  • Over 500 million tweets are generated every day
  • Bitcoin processes over 200,000 transactions per day
  • Google processes over 3.5 billion search queries daily
  • And only 0.5% of it is actually analyzed and used

That is a lot of activity requiring intense edge computing capability.

So What is Big Data?

So does it just refer to a lot of data?  It seems to be much more than that simple interpretation.  Gil Press wrote “12 Big Data Definitions: What’s Yours?” at https://www.forbes.com/sites/gilpress/2014/09/03/12-big-data-definitions-whats-yours/#7721c66e13ae.

There are many, many definitions but for our purpose I will reference the Wikipedia definition as follows:  “Big Data represents the information assets characterized by such a high volume, velocity and variety to require specific technology and analytical methods for its transformation into value.”

In 2001 Doug Laney articulated the concepts of volume, velocity and variety in describing Big Data.  I believe this to be critical to the understanding of Big Data because it is not just about the amount.

Volume is the amount of data.  The sources are vast and ever increasing which contributes to the increased volumes.  Smartphones, wearable devices, sensors, laptops, social media, electronic tracking and transactions all generate information and are prevalent and everywhere.

Velocity amplifies the amount because of the speed of streaming and tracking.  Every second that you use your GPS means a signal is being transferred between satellites and your device to track your location and give you directions.  E-Commerce ordering means you want instantaneous response and the immediate transfer of information.  And real time tracking means it is constantly being streamed between devices and computers.

Variety is the third dimension.  No longer are just numbers and letters being transferred.  Video content, texts, audio, sensor information and unstructured (ie. doesn’t reside in a database) sources all contribute to the amount of Big Data.

I believe it is clear.  Big Data is more than just big volume.  Anyone who is trying to amass and analyze this kind of information and transform it into valuable information understands the nuanced challenge of dealing with Big Data.

The Challenges

The Digital Supply Chain has as it’s core the requirement to have end-to-end electronic connectivity and visibility.  This is absolutely a basic requirement to enable the Supply Chain of the future.  But to make that happen we need to be able to effectively manage Big Data.

What are the Challenges?


In many cases businesses do not yet have enough information.  To truly manage an end-to-end Supply Chain you must have end-to-end visibility in real time.

But to make that happen you must have electronic connectivity.  This connectivity must extend to all of your suppliers, all of your internal operations, all of your service providers (eg. carriers and distributors), and all of your customers.


In order to handle all of this you must have the requisite storage and I/T management capability.  Further you must have the processing capability to analyze and manipulate it.  There are many service providers that can provide this capability so it doesn’t have to be an internal capability.


As the saying goes, “Garbage In, Garbage Out”.  You need accuracy.  You need full transparency from all of your sources.  And you need a speed of refresh which supports your analytical and decision making requirements.


As mentioned in the definition of Big Data one of the areas of focus is variety, particularly as it pertains to unstructured data.   Your information may be coming into you in many different formats from many different sources.  How you are able to take all of that and convert it into a singularly interpretable format is the challenge.  For many companies just receiving basic structured data from many different suppliers in different ways is challenging enough.


Once you have your Big Data now you need to translate this into actionable information.  This is most often beyond the capabilities of manually managed spreadsheets.  You require an Advanced Analytics engine that can either promote or assist in decision making, or in the extreme use Artificial Intelligence in decision making.


Because an end-to-end Supply Chain requires the provision of information from sources outside of your company, in a transparent manner, you have a duty of responsibility to protect that information and guarantee its security.  That entails ensuring privacy with sufficient controls to assure all parties that their information is safe in your hands.

Management Support

All of this work requires a Strategy, Vision, Executive Support, Resources, and the Financial funding necessary to make it all come to fruition.  This is perhaps the biggest challenge with respect to Big Data.

Being able to collect Big Data and translate it into information for decision making holds the promise of improving efficiencies, reducing costs, improving competitiveness, increasing customer satisfaction, and decreasing time to market and time to money.  But all of that needs to be thrown into a business case for approval before you can proceed any further.


The concept of the Digital Supply Chain is certainly here.  The reality of it may be close or far away depending on your company.  Between that concept and that reality is the requirement for a Big Data strategy.

The level of electronic connectivity and the creation of more and more data will continue to increase.  At some point your company will have to get on board with a Big Data management strategy or be left behind by your competitors.

At some point not that long ago companies had to make the leap to invest in electronic email for their employees.  Today that seems like such a basic no-brainer decision but at the time it was a huge leap of faith from an investment standpoint.

Now we are at a similar inflection point.  An investment in Big Data management may seem optional but in the future any surviving, and thriving, company, will have a Big Data management capability.  Big Data will be as common as having email.

Originally published on July 17, 2018.