Lessons from the Edge of the Abyss! Surviving a System Implementation!

Lessons from the edge
SC Viral The Pandemic Diaries

The company I had just joined was nearing the finish line on the implementation of a new WMS system.  They had been working on the system change for a few years.  Now implementation was only a few months away.  There were going to be lessons from the edge of this experience.

As the implementation date drew closer one of the key Warehouse management leaders, heading the project from the beginning, left the company.  We forged ahead and implemented the system on April 1st.  That’s right, April 1st!

And that’s when the “fun” began!

Heading to Launch!

The company had spent millions of dollars on the new system, upgrades to the physical warehouse facility, and the enormous number of resources that were dedicated to the project.  The company that was providing the new system and the company that was implementing the physical warehouse changes seemed to  have dedicated a lot of time to the project also.

And it was clear that the upgrade was mandatory.  The old system could not handle the demands of new products, new channels, and greater capacity that the company’s strategy required.  There was no turning back.

So it was with great anticipation that the old system was shut down, intensive, last stage employee training was conducted, and final system testing was completed.

Inventory had been sent out to all channels in advance to cover 3 weeks worth of forecasted demand, in case there were any issues in starting up the system.  Everyone seemed confident that this was a sizeable buffer in the supply chain.

And then on April 1st everyone came into the Distribution Centre to see the new system in action.  It was going to have advanced transactional capabilities and efficiencies combining the promise of the latest software with a revised warehouse conveyance system, and an improved picking and packing process to match.

The system was turned on.   And nothing happened!

All Hell Breaks Loose!

The I/T team and the Business folks quickly scrambled to find out was going on.  With a few tweaks here and there the new conveyance system slowly started to move boxes.

The boxes of goods were put on the new conveyor system at various points and wound their way throughout the warehouse on their way to a pick to light (PTL) system.  At the PTL area operators were to pick goods out of the incoming boxes and put them into boxes slated for outbound shipping locations. These outbound boxes would be routed through packing and on to shipping.

But the pace of box movement was painfully slow.  And when boxes got to quality control points almost 100% of the boxes were automatically rejected.  It seems that the information detailing the contents of each box was not “travelling” as fast as the slow moving boxes.  So when the boxes got to the automatic quality checkpoint the contents that the system thought were in the box were not the same as what was in the physical box.  Everything was being rejected.

We had a massive problem.  As I said to the I/T team, “How can the electrons containing the information move slower than the physical box that was slowly moving 100 feet by conveyor?”  Something was severely wrong.

There were a myriad of other problems as well.  The trays on which the boxes were placed, to move around the conveyor system, had such a low profile that they created traffic jams of boxes.  Boxes would fly off the conveyor whenever they bumped into each other or when around turns.  Employees, including myself, were constantly fixing log jams in the conveyor system.

Sensor issues, conveyor speed problems, in-line weigh scale problems, cubiscan issues and more constituted the abundance of physical system issues.

It turns out that the contractor who installed the conveyor system had also subcontracted the software work that ran the conveyor to another company entirely.  As a result there were constant communication issues between the WMS system and the conveyance system.

On top of that the WMS system itself was one of the first of its kind installed by this company.  There were a number of flaws which we only painfully discovered when we went live.

The combination of software and hardware issues meant that our production in the first week of cutover was only about 5% of what it would have been in a normal week.  In the second week we were about 10% of normal.  And in the 3rd week we were about 20% of normal production.

Our inventory buffer had now been exhausted in our channels.  And we were no where near where our throughput numbers had to be.  Very soon our channels would be screaming about a lack of inventory.  This would quickly translate into lost sales.

The heat was on high!

All Hands on Deck!

The CEO was understandably becoming increasingly restless.  The channels were very upset.  The sales team was beside themselves.

The Supply Chain and I/T teams were all hands on deck.  We escalated to the very top of both the software and hardware companies to ensure that we had not only all of their attention but all of their resources at our disposal.

We worked with our internal teams to prioritize which goods needed to move first.  While we couldn’t ship everything that the channels needed we could ship the highest, fastest selling goods first which would keep them somewhat satisfied and helped to buy us more time.

Week after week we solved more and more problems.  Many times this would unearth other issues.  But slowly and steadily our production numbers started to return to ever higher levels.

The Distribution Centre looked like a disaster zone.  Boxes were everywhere in every condition from receiving through to storage and on to processing, quality control, packing and shipping.  The piles of goods that were rejected at quality control stations were up and down every aisle and in every corner until we found the time to go back and resolve those issues.

It was at that time that we recruited a new candidate to run the Distribution Centre.  As a part of the interview process I toured the final candidates through the warehouse to gauge their reactions to the very obvious disaster that was in front of our eyes.  The candidate I selected was the one who looked at the situation and responded with enthusiasm and saw all of the opportunities.  

If looking at this situation garnered that reaction I knew that this was a powerful leader who would not be scared off by anything.

The Aftermath and Reflections

Within two months we had recovered the entire backlog.  The channels were full again and they were happy.  The new system still had its flaws but we were now processing goods at a level that far surpassed the capabilities of the old system.  In the coming months we would stress the system to the maximum during the Holiday season.  And we would set records for throughput, speed and cost reduction.

But getting to that point was phenomenally painful.  Most everyone goes through at least one problematic system implementation in their career.  And for me this one was at the top of the list.

But as I thought back on how we persevered through this incredibly stressful experience there were several observations that struck me as key for us to have not only survived but thrived during this period.  These were the Lessons from the edge of the abyss in a system implementation and tech rollout!

Customer First!

We always kept our customer at the top of our minds.  We knew that we were not meeting all of their needs but that fact was our driving force to keep pushing for improvements at a fast and furious pace.  We prioritized their requirements, communicated with them constantly, and slowly but surely restored their confidence in us as we eliminated the backlog of deliveries.

On The Field Behaviour!

Myself and all of our leaders were “On the Field” all the time.  I put every other responsibility on the back burner.  I spent most of my time on the Distribution Centre floor, moving boxes, fixing log jams, talking and working side by side with our employees.  By spending this time actually in the operation I was able to observe what was really going on which informed the actions and decisions we needed to take.

I didn’t need to be in meetings making decisions from afar.  I needed to be on the front line.  And it created a level of goodwill, morale and resolve that could not be achieved otherwise.

Focus and Perseverance!

There were so many problems, so many upset customers, and so many Executives wanting answers that it would have been easy to be overwhelmed and give up.  The stress levels were off the charts.  And the time commitment we needed from those involved was enormous with many employees working 15+ hour days, day after day after day.

But through it all we remained focused.  Failure was not an option.  We needed to tackle the problems in order of importance.  We needed to focus and direct our resources.  And we needed to take one step forward every hour of every day.

I put everything else on the back burner.  I missed a lot of meetings.  Other projects were falling behind.  Costs were going through the roof.  But getting this system up and running was the single most important issue in the entire company.  An undivided focus on this project was what was required.

As the saying goes “When the going gets tough, the tough get going”.  And our team exemplified that motto.    Our employees worked tirelessly and selflessly to get done whatever was needed.

These were the lessons from the edge of the abyss.

Lessons from the Edge in Conclusion

People are implementing new systems and upgrades every day somewhere around the world.  Many of these will go smoothly.  Many others will not.

In our case there was a tremendous amount of testing and planning done in advance.  But when the switch was finally turned on there were just too many unforeseen issues and scenarios which came up.

The key for us to survive this system cutover was in how we reacted.  The leadership, dedication, focus and commitment from all of our employees allowed us to prevail and ultimately succeed.

Most importantly going through this “battle” together brought our teams closer than ever before.  Even though you could see exhaustion at every turn there was an ever increasing resolve to break through all barriers.  We were learning lessons from the edge of the abyss and reacting in real time.

We built trust, goodwill and morale in a way which could only have come from going through this fire together.  We learned a lot of lessons from the edge of the abyss.

And I could never have been more proud of being a part of that team We learned lessons from the edge of the abyss!

Originally published on February 6, 2018.