Expert Column: Big Data, Bigger Profits

Retail is a Big Data business - perhaps more so than any other, especially in today’s digital landscape, with so much information generated from so many different sources. And while many industries produce large amounts of data, for most, the analysis of that data – while potentially critical for competitive advantage – isn’t necessarily a core, mission critical function of the business.

Retail is different. Large retailers base their business on conducting millions of small, profitable transactions involving many customers, with tens of thousands of items sold across many physical locations, online and via catalogs. At the core of retail operations rests a single goal: to offer products that customers want through the right locations and channels at the right time at the right price.

Sounds simple, right? As it turns out, behind the appearance of simplicity is one of the business world’s longest-standing and toughest Big Data challenges. Let’s take the example of a 500-store grocery chain. According to the Food Marketing Institute, the average grocery store stocks a meager 42,686 distinct items. Keeping track of those items across 500 locations on a daily basis means logging 21,343,000 inventory records every single day.

Of course, to analyze trends and performance, and to make smart decisions about future customer demand, we need to look at that data over a longer period of time. The industry standard (which leaves something to be desired) is to retain detailed data for the current year plus two years of history, for a maximum of three years of historical data. Three years of daily inventory history for our average grocery store chain means that in order to conduct fundamental category and merchandise analysis,  we need to be able to richly and nimbly manage and analyze a whopping 23 billion rows of data.

That’s before we’ve added data about sales transactions and customers into the equation --  and this really is the key. For retailers to succeed, they need to understand the customer experience: the items each customer saw (and didn’t see); how these items were visually presented; how they were priced; the interactions with staff; and the efficiency of the checkout line. They then need to compare that experience with actual sales – which is the measurement of the customer’s response to their experience.

To do so, retailers need to ask and answer questions like:

• How is my customer’s transaction affected when an item is out-of-stock?
• What assortment of products will drive increased sales?
• How can I craft offers that will drive additional loyalty from my customers?

It is here where data begins to get extremely big. Let’s look again at our same 500-store chain. Across the entire chain, our retailer will have approximately 3.2 million unique customers, each visiting their store 1.7 times per week for a total of 283 million annual transactions. Compared to our 23 billion rows of inventory data, 283 million doesn’t seem like a lot, right?

But people don’t shop for “transactions.” They shop for items and groups of items that will allow them to make a complete meal, stock up on necessities, or complete their shopping list. If the retailer doesn’t help the consumer achieve their goal, then the customer is dissatisfied and their loyalty is negatively impacted. They may buy fewer items during their shopping trip or – worse yet – they may abandon their purchase altogether and shop somewhere else -- permanently.

Therefore, to understand and optimize the customer experience, retailers need to compare the line items within every customer transaction along with the profile of store inventory that existed the day of the transaction. They need to understand how the various in-stock positions, item assortments, presentations and promotional offers in effect during each transaction positively or negatively impacted the customer experience. Additionally, the most ambitious retailers seek to go beyond analyzing individual trips, using customer loyalty data and other customer-centric insights to connect individual transactions across an entire lifetime of behavior for each shopper. This allows them to leverage bigger picture insights in order to craft an irresistible experience for customers.

Making this happen requires looking deep down into the details of customer purchasing habits, and sifting through incredibly large and complex combinations of data to identify what really matters. Doing it successfully requires Big Data analysis and modeling tools that allows business users to easily ask their questions, while breakthrough technology acts behind the scenes deliver answers quickly. Questions like:

• What combinations of items indicate high-value customers?
• Did my promotional strategy drive increased sales or margins?
• For which key items must I ensure on-shelf availability in order to delight my targeted customers?
• Is my shopper purchasing new items in new categories, indicating increased loyalty?
• What items tend to sell together and how can I use that insight to increase loyalty?

When you hear retailers talk about the mandate to move from a merchandise-centric retail operation to a customer-centric retail world, these (among many others) are the questions that they are challenged to ask and answer. With the complexity of calculations and volumes of data that reside underneath those questions, success in the modern retail industry also requires success with Big Data. Really, really Big Data!

Jed Alpert is VP of marketing at 1010data, a provider of a cloud-based platform for big data discovery and data sharing, used by hundreds of the world's largest retail, manufacturing, telecom and financial services enterprises.