Split Test Node

The Split Test Node can accomplish several tasks:

1. Randomly select a variant for A/B or multivariate testing
2. Choose the winning variant that has been getting the most engagement
3. Reassign the probability of selecting a variant to favor the winner

Recommended places to use this node: 

• Selecting Twitter responses 
• Selecting different subject lines and images for outgoing emails
• Selecting different texts and images to display on a personalized website

A Producer split test node with four available variants, with red being the favored one.

 A Split Test Node is a two-part node comprised of a 'Producer' and a 'Consumer'.

A Producer will assign and randomly selected variant to the transaction it is processing at that time and save two outputs - 'Selected Variant' and 'Consumer ID'.

• The 'Selected Variant' is the variant that was randomly selected by the node - in the example above, it is “Blue”
• The 'Consumer ID' is a hashed string that must be passed on in the process and eventually received by a 'Consumer' node to see which variant is the winner. For convenience, the variant name is appended to the end of the hash, and each section is “split” by the underscore for easier identification.

A 'Consumer' graph will digest the Consumer ID generated by the 'Producer' node to understand which variant is the winning variant. 

When implemented, the flow could look like this:

In the example above, the Consumer node is placed in the same graph, but it would realistically be placed in a separate graph with the Xponent API added. As such, the flow would look like this:

• Producer selects the variant and consumer ID, and sends it out to the world (email, webpage, twitter response)
• The user engages with the email/webpage
• Xponent's API graph picks it up, and the consumer node will process 

If you click on the Producer node in the graph, and look at the right panel, you will see the Producer results panel. This shows the number of iterations across all variants and the number of responses.

Producer results panel

 Split test Options

Two options can be enabled based on the use case.

1. Adaptive Learning
2. New Test on Deploy

Adaptive Learning 

By enabling Adaptive learning, we can change the distribution based on the winning variant, and reassign the probability of selecting a variant to favor the winner. Trials can be ongoing, allowing you to choose a winning variant and change the distribution infinitely. To stop the ongoing trials, you can stop the deployment of the graph.

Adaptive learning enabled

• Minimum Iterations Produced : Default value - 1000. At least 1000 transactions must be produced before variant distribution is adjusted based on adaptive learning
• Minimum Responses Consumed : Default value - 100. At least 100 responses (from the 1000 produced) must be consumed before variant distribution is adjusted based on adaptive learning
• Selection Certainty : Default value - 95. The confidence level required to select the winner. If this number is lower, the winner is decided quickly but with less confidence. The winner is chosen using a Z-Score significance test. 

Post the Minimum Responses Consumed; the A/B Split Test Node will select variants based on the winning ranking and the Post Selection Distribution.

Choosing a Winner 

For a winner to be chosen, all of the following conditions have to be true: 

• The minimum number of iterations for an individual variant must be greater than or equal to those specified AND
• The minimum number of responses in any variant must be greater than those set AND
• The potential winner response rate must differ from the other options by at least the confidence level given when applying a Z test.

For Example:

a winner will be chosen if:

• After 50 iterations AND
• After 10 responses have been consumed AND
• One of the four options has a response rate that is significantly different from the others at a 90% confidence level

Once a winner has been chosen, the post-selection distribution choice will then take effect - with reference to the screenshot above - 61% of the post-test variants will be the winning variant, 27% will be from the second place choice, and the third place will represent 12% of the selections. The fourth-place variant will not be used once a winner has been chosen. 

Create a New Test on Deploy

On selecting this option, every time the graph is restarted or redeployed, the split test node will be reset, and everything (winning variant) that was learned so far will be reset.

The node in a graph can be changed from Producer to Consumer by clicking on the two-way arrow attached to the node in the graph. 

The name of the existing variant cannot be edited

Producer results panel updates approximately every minute

There is no way to get the count of the iterations and response values in the schema or public variable.  Look at it on the results panel for the counts.

If an email is opened multiple times, we get multiple responses. So, the number of  iterations can be greater than the responses.

Creating a Split Test

Step 1

When creating a New Item, choose the 'A/B Split Test' Node from the 'Learning category'. This will lead you to a new screen with a blank Split Test Node to configure.

Configuring a blank split test node

 Step 2

• Click Create Variant at the top right-hand side corner of the screen and type the desired likelihood of selection for that variant. 

If you create a variant, you don't need to use the trash can icon to the right of the proportion to remove it. 

 Step 3

Place the 'Producer' node in the process. You can select the node type to be a 'Producer' or a 'Consumer' by selecting the two-way arrow at the bottom right corner of the node. 

Step 4

In a different process that is used to track email, tweet, or web engagement, put the 'Consumer' node in the process and configure it to digest the Consumer ID from the right field of the schema.

---

Validation Warnings

: All the invalid notifications will be displayed after expanding the Invalid Drop-down.