A/B testing of Category Merchandising – Klevu

We are very pleased to introduce a highly sought-after feature: A/B Testing, or more accurately Multivariate Testing. You can now configure one or more variations of a category merchandising campaign and run them alongside each other, to see which one performs the best.

For example, you could compare Klevu’s internal AI and Machine Learning versus your own manual merchandising, or see if boosting a particular product brand yields more conversions than another, or any other rules you wish to configure via our visual merchandising tool.

You can even configure the customer split percentage within the Klevu, specifying how many customers should see variation A vs B (or C, D or E), allowing you to try out smaller experiments for a few days on a small portion of your customers, or run larger tests with a head to head 50:50 split over the course of a few months.

In this article, we will introduce how you can do it and the most commonly asked questions (FAQs).

Prerequisites:

Klevu A/B Testing is a part of Category Merchandising, so you must have an active Klevu subscription which includes the Category Merchandising solution.
Currently, we only support merchants using our Klevu Template JS, as this is where we have implemented the customer-facing A/B test logic. If you are on JSv1 and wish to discuss your options, please contact support.

Step 1: Create an A/B Test

You can find A/B testing within Smart Category Merchandising > Visual Merchandising inside the Klevu Merchant Center (KMC). Alongside Default Rules and Scheduled Campaigns, you will see a new tab labeled A/B Test.

After clicking on this tab, the next page is where you would see any draft, completed or running tests you have already configured. Click on Create a New Test (or New A/B Test) to get started.

Objective / Goal / Test Name

The first thing to do is decide why you are running this test, by selecting one of the objectives available. If you’re trying this for the first time then you can select “I'm just playing around” and move on, but in general you should give your test an objective and a meaningful name, and bear those things in mind whilst configuring your test variants.

We will also use your chosen objective to highlight which variant is the strongest, although you will still be able to view all analytics data regardless of what you choose here.

Schedule

Next, define when your test will run. You can configure a test to start immediately by setting the start date as Today, or schedule to start automatically on some date in the future. You can also select any duration between 1 and 90 days.Use the quick links on the right to select the most common time ranges.

Click Proceed once you have completed your scheduling, then once you are happy with everything on this page, click on Save & Next to continue.

Step 2: Setup Variants

This is where you can configure up to five variants to run alongside each other, with their own merchandising rules, allowing you to analyze and compare which of your strategies is the strongest.

You cannot modify the base variant on this screen, as this variant represents your default rules which are already configured elsewhere in Klevu, via the Default Rules and Scheduled Campaigns tabs. Your base variant will always track whatever merchandising takes place in those tabs.

Next you can add additional variants, and even rename them and add a description to remember what that variation is focusing on. Each test can have up to five variants, including the base variant.

Step 3: Setup Merchandising Strategies for Each Variants

There are two types of strategy available for your variants:

Manually Merchandise
Self / Machine Learning

The first allows you to manually configure rules which will apply only to this particular variant, for example boosting high margin products, de-boosting certain brands, or pinning top products. These rules will take precedence over the Klevu AI self/machine learning, which will also apply.

Secondly you can choose a single variant to use only Klevu self/machine learning. In this case you will not be able to configure any manual rules at all, which can be useful to compare how Klevu AI performs and whether you should be spending time on merchandising or just letting Klevu AI run on its own.

If you choose manual merchandising, you will then select Click Here to Merchandise to continue.

Variant-level Visual Merchandising

Those of you who have used our Smart Category Merchandising tool will already be familiar with this screen. The only difference being you are configuring rules which will only apply to the variant you are editing. For more information on how to use our visual merchandising tool, please click here.

When you are happy with your changes, click on Save Variant in the top right to return to your test.

Step 4: Traffic Split between Variants

Once you have named your variants and given them a description, next you should decide which percentage of your customers should see which of your campaigns. All values must add up to 100. For a pure A/B test you can leave the traffic split as 50% base and 50% variant B. For a smaller experiment, you may allocate 90% to base and just 10% to variant B.

You cannot delete the base variant, but if you want to exclude it from your test you can allocate a Traffic Split of 0%, meaning that for the duration of your test no customers will see those rules.

Step 5: Schedule/Start A/B Test

In the bottom right you will notice two buttons:

Save as Draft
Start Test

Use Save as Draft, if you are not quite ready to start your test and come back to make changes later. What happens after you click the second button, Start Test, will depend on your schedule.

If your test start date is today, then the test will start immediately, and you will see the test in a running state. If your test was scheduled, the test will enter a scheduled state. You can still edit the test in this state, but once it enters the running state you can no longer edit it.

It can take up to 30 minutes from the test entering a running state until your customers are allocated with variants and they start seeing results, so please be patient for the first half hour. If you believe your test is still not running after one hour, please contact Klevu support.

Interpreting the Results

Whilst a test is running, Klevu is constantly collecting the interactions customers are making in terms of clicks and conversions. The analytics data you see on the following screen is aggregated once per hour and presented so you can actively monitor your tests in close to real time. You can even stop a test if you don’t like what you see!

For any Running, Stopped or Completed tests, you can access the analytics of a test by clicking the analytics button:

Klevu will automatically highlight the winning variant depending on the objective you defined, but you can also select a different objective to see which variant would have won in that scenario, eg. switching between Click Through Rate or Conversion Rate.

The data available for each test is as follows:

Category Views: The number of times the category(s) have been viewed.
Product Clicks: The number of product clicks generated from those categories.
CTR: The click through rate, calculated from the above two metrics.
Product Conversions: The number of sales generated after interactions with those categories.
Sales: The total of the sales generated from those product conversions.
CR: The number of clicks that lead to purchases, the conversion rate.

The product conversions, sales and CR value aggregation is delayed by up to 48 hours, so please bear this in mind when reading the results. We therefore recommend you wait 2 days after a test has completed before fully evaluating the sales data.

You can also switch between variant and category view, to get an idea on how your variant rules performed on a category by category basis, as you may find that something works particularly well on one category but not on another, and you may wish to pick and choose which rules you take away from the experiment.

Applying the Results

Now that the test is complete and you have found your winning variant(s), the next step is to decide what to do with those results. Klevu will not apply anything automatically, so you can take your time and analyze things per category.

The action you will take next depends on the outcome of the test, detailed below. You can look at this from a variant level or category by category by using the available tabs, and decide what to do next:

If the base variant wins

Your existing scheduled campaign or default rules are already tuned better than your test. There is no action to be taken. You may like to tweak something and try another test.

If Klevu Self / Machine Learning Wins

This outcome suggests that you should trust Klevu AI, and remove any manual merchandising rules you have configured. If you tested against the base variant, you may want to go and edit your default or scheduled campaign to remove manual rules.

If Manual Merchandising Wins

In this scenario, the manual merchandising configured for this particular variant were the strongest, so you may want to export them so you can apply them to one or more categories. You will find a … on the right hand side where you can export the rules. Once you have them downloaded, you can follow the same process for importing Klevu visual merchandising rules, detailed here

FAQs

How are my customers allocated a variant (eg. A or B)

Once an A/B test enters the running state, it can take up to 30 minutes for a customer to be allocated a variant A or B (or C, D or E). Once they have been allocated that variant, they will keep it for the duration of the test. For example if you set up an A/B test where only 10% would see B, of 10 customers, 9 of those would be allocated variant A and 1 would be allocated B. We also monitor the customer activity afterwards to ensure a fair split (see true allocation logic below). This is all invisible to the customer of course, all they will see is the products from the campaign you have configured.

How does Klevu ensure the tests are fair? (True Allocation Logic)

Since your tests are category-specific, and not all customers visit every category, Klevu uses something called “true allocation logic” to ensure your tests yield meaningful and fair results. For example, if you specify a 50:50 split between A and B on the Shoes category, it is important that test represents a fair split between customers who actually visit that category. Therefore, if a customer is allocated variant A but doesn’t actually visit the Shoes category, Klevu will take this into account when distributing future variations to customers, favoring variant A slightly until the balance is restored.

Can I use the same category in two different tests?

No. You can only schedule a single test for any particular category for a given timeframe. You can schedule multiple tests on the same test to run concurrently, but not simultaneously.

Can I stop a test mid-way?

Yes, you can stop a test, but please bear in mind there is no way to restart it, so make sure you want to stop the test before you do. Simply find your test in the listing, click the … on the right hand side and select Stop. Once stopped, whichever variant had the best results up until that point will be highlighted as the winner.

How can I start a test instantly?

You simply need to set the start date as today. Once you save and publish the test, you will see it enters a running state instantly. Please note, however, that it can take up to 30 minutes for your customers to be allocated variants for this test.

Can I still merchandise my default rules while an A/B test is running?

Yes, and in fact this is a good way to compare your default merchandising rules against the Klevu AI, where you specify one variant as “Self/Machine Learning”, then continue to merchandise manually throughout the test. You can even add another variant to compare against a fixed set of rules too. Note that whilst it is possible to merchandise the default rules, you will see a warning whilst any tests are running, just so you are aware that your changes could impact any running tests. It is not possible to modify the merchandising rules of any A/B Test variants once the test is running.

Which merchandising rules get applied when a test completes?

When a test completes, the category(s) you have tested will revert back to their state as if there was no test running. You must manually decide whether to apply the rules of one of your variations to that category after a completed test. Therefore, at the end of an A/B test, one of the following will happen:

Another A/B test may begin, if one was scheduled for the same category.
A scheduled campaign may begin, if one was schedule.
Otherwise, the default rules will apply for that category.

One of my variants is performing badly, can I remove it?

Once a test is running it cannot be edited. We recommend you stop the test and create a new one, omitting or modifying the poorly performing variant, or reducing the customer allocation percentage to run a smaller test for that variant.

Can I delete a test?

Yes, you can delete a stopped or completed test, but please be aware that you will lose the view of analytics data associated to that particular test. Klevu will still keep the analytics data it needs, but you wont be able to see the A/B test results.