A/B/n testing enables the testing of multiple channels and content variations within retention or lifecycle-based campaigns. The system automatically assigns customers to different test variations in a round-robin manner, ensuring an even distribution.

If a control group is included, customers are also allocated to this group, but they do not receive any messages. However, their events and behaviors are still tracked to compare their activity against those who participated in the tests.

The experiment runs for a predefined duration, configured by the user. Once the test concludes, the user can select the best-performing action set and apply it to:

• The remaining segment (for one-off campaigns).
• The remaining duration of a dynamic campaign.

Experiment Automation

Auto-Publishing

• When enabled, the system automatically analyses the test results and publishes the best-performing action set to the remaining segment.
• The selected KPI determines the winning variation.
• No manual intervention is required.

Manual Approval

• When enabled, the system identifies the best-performing test but requires user approval before publishing the final actions.

Publishing Experiment Results

Once the experiment ends, the best-performing actions can be published as campaign actions:

• One-off campaigns: The winning actions are applied to the remaining target segment (all customers except experiment participants).
• Dynamic campaigns: The campaign continues for its specified duration with the published action set.

Key Performance Indicator (KPI) Selection

A Key Performance Indicator (KPI) is a quantifiable metric that measures the effectiveness of an experiment.

• A KPI must be selected for auto-publishing.
• The CRM will evaluate the tests based on this metric to determine the best-performing action set.

Actions and Scheduling

Action Types

An action represents a single execution of:

• Email
• SMS
• Push Notification
• Webhook

Available Scheduling Options

Step-by-Step Guide: Creating an A/B/n Test

This example outlines the process for a one-off campaign; however, A/B/n testing can also be applied to dynamic campaigns.

Create a New Campaign

1. Navigate to Campaigns → Retention.
2. Click New and select the campaign type (One-Off or Dynamic).
3. Assign a name and add tags for easy search and organization.
4. Set the campaign start date.
5. Click Save, then click Next.

Define the Target Audience

1. Select the target segment for the campaign.
2. Click Next.

Activate Experiment Mode

1. Enable Experiment Mode to initiate A/B/n testing.
2. By default, a control group is included.
   • To remove it, click Delete Control Group (top-right corner of the control group section).

Add Test Variations

1. Click Add a New Test.
2. Select the appropriate actions and content.
3. Repeat this process to add additional test variations as required.

Configure Experiment Parameters

1. Set the experiment duration (in days).
2. Define the percentage of customers to be included in the experiment.
   • For dynamic campaigns, the system determines participation based on the duration, as the total segment size is not predefined.

Define Experiment Preferences

1. Click Next twice or navigate to Campaign Preferences.

2. Disable Use Global Settings.

3. Expand Experiment Preferences and choose:

   • Automated publishing (publishes the best-performing action set automatically).
   • Manual approval (requires user confirmation before publishing).

4. If auto-publishing is enabled:

   • Select the objective KPI for determining the winning test.
   • In this example, the goal is the highest number of depositors.

5. If manual approval is enabled:

   • The system will recommend the best-performing test, but the user must manually approve the final selection.

A/B/n testing in CRM allows users to test multiple content and channel variations, ensuring data-driven decision-making in retention and lifecycle campaigns. By automating performance analysis and allowing for real-time optimisations, businesses can maximise engagement and optimise campaign effectiveness.

Publishing the Best-Performing Action Sets

Once an A/B/n experiment concludes, the best-performing action set can be published and applied to the remaining campaign audience. This ensures that the most effective messaging and engagement strategy is implemented for one-off and dynamic campaigns.

Step-by-Step Guide

Access the Experiment Section

• Navigate to the Experiment section within the campaign.

Publish the Best-Performing Action Set

• Click Publish Actions under the test that performed best.
• This will select and apply the action set from the chosen test to the remaining campaign.

Review Analytics (Optional)

• Click View Analytics to analyse the experiment results before making a final decision.

Finalise and Save

• Once Publish is clicked, the selected actions will be copied to the Actions section of the campaign.
• Click Save to confirm the changes.
• Ensure the campaign status is set to active to apply the published actions:
  • For one-off campaigns: The action set will be executed on the remaining segment (customers who were not part of the experiment).
  • For dynamic campaigns: The action set will continue for the remaining campaign duration.

By publishing the best-performing test, campaigns can be optimised for maximum engagement and effectiveness, ensuring that data-driven decisions guide the messaging strategy.