Discovering Merchandising Insights with Testing and Optimization

Tests can not, and should not, stand in for human analysis and intuition. Just like human intuition can not, and should not, be confused for data-driven test results.

By first digging into historical data, retailers can use the knowledge they gain to deduce what types of behaviors can be expected as a result of certain changes. They can then formulate educated hypotheses and measure their tests using the proper KPIs, which will result in productive and insightful tests.

Combining human analysis and intuition with the learnings from the tests themselves is how retailers can extract the most value out of their optimization testing.

Here is a step-by-step breakdown of how to interpret and use data before, during and after the testing process for optimal results.

First Identify Business Targets and Areas for Growth

An appropriate starting point for merchants is to identify which business goals they want to optimize for, and to analyze their site data in order to discover which onsite elements directly influence those goals. For example, if a merchant knows that they want to increase average order value then they should start by looking at different site elements that directly impact either the number of products purchased per order or the price tier of products that are commonly purchased. These elements could include things like product recommendations shown on either their product detail pages or their cart page or the products they showcase on the top of their category pages. Once the merchant narrows down which elements they believe directly affect their performance in terms of the chosen goal, they can then begin hypothesizing which variations of those elements would be best to test.

Next Create Element Variations That are Data-Driven

The most important mistake to avoid when forming hypotheses for which variations to test is confirmation bias.

For instance, if a merchant is mostly known for selling products to women, it doesn’t make sense to test male-focused imagery or copy against the status quo due to the fact that the majority of the site traffic probably consists of women. It is an inherent bias to choose test variations knowing which test will win, as the test will only confirm what you already know. Conversely, if a merchant starts with their purchase data and digs into that data by segment, they can often discover certain trends that can help them form data-driven hypotheses which will deliver useful insights.

An example of this would be a merchant who discovers certain brand or product affinities of first-time customers and decides to test if bringing those specific brands or products to the top of their homepage (or other key landing pages) drives more conversions than when other content is in that real estate.

In other words, will there be an increase in conversions if certain brands or lines that first-time converters often purchase are emphasized when consumers first visit the merchant’s site or not?

This is one small way that merchants can use data in combination with their own analysis to better inform the variations they build tests around.

Finally, Derive Deeper Insights Than The Usual KPIs

Testing and optimizing site elements for certain KPIs, like conversion rate or average order value, is something most merchants are familiar with today. The truly interesting insights come from when merchants use products like Nosto’s Merchandising Insights feature to dig into segment, brand, or product specific results that tell a more detailed story.

Let’s say a merchant is known for one particular line of products (as evidenced by what customers buy most) but needs to sell their other lines as well. They can then form a test around different ways to push site visitors to purchase the less popular lines of products.

For example, assume that an imaginary merchant is known for street wear, but they also carry athletic attire that sells way less frequently. If this merchant were to set up an A/B test of two variations of a homepage banner that each show one of the different lines of clothing and set the test to optimize for conversion rate, then the test would show that the banner that displays the line that already sells at a higher rate wins.

However, if the merchant were to separate the customers who clicked one of the banners into two separate segments (those who clicked the street wear banner and those who clicked the athletic attire banner), then they could begin pulling more useful insights into what exactly the banner variations accomplished.

For instance, are visitors who clicked the athletic attire banner more likely to actually purchase athletic clothing?

If it turns out that those who click the athletic clothing banner do, in fact, buy athletic attire at a higher rate, then the merchant now knows a way they can drive more site visitors to purchase that line of clothing. Sure, the athletic clothing banner may not drive as many overall conversions as the street wear banner, but if the merchant can figure out a way to know when to present that banner to customers who will click it then they know by doing so they greatly increase their chances of those customers converting on a line of products they need to sell more of. 

In short, by digging a layer deeper into test results, and not only focusing on general KPIs, merchants can derive deep product and brand insights that can inform complex ecommerce merchandising strategies.


For online retailers, implementing an effective testing strategy is often a slippery slope. On one hand, there are the retailers who understand the importance of testing and optimizing but liberally implement tests without any real direction or goal in mind. These tests often lead to inconclusive results or unhelpful insights due to the fact that those running them aren’t sure what they’re looking to learn.

On the other hand, there are retailers who hold very strong beliefs and only implement tests in order to confirm their biases. These tests are unproductive because those who are running them don’t really give them room to do what they are meant to.

Combining human analysis and intuition with the objectivity of test results, though, is how merchants can discover the full potential of testing variations of their onsite experiences to optimize performance.

To get started with our latest offering, get in touch with our team (via demo request or the chat box in the bottom right corner).

Watch a 4 minute demo
Let us demonstrate why over 2,600 leading retailers worldwide have chosen Nosto’s AI-based personalization platform.