Website A/B Testing (Static vs Dynamic Product Images)

Goal:

The goal of the A/B testing is to determine which variation, A or B, leads to better overall performance for the e-commerce website. The primary metric to focus on could be the conversion rate, i.e., the percentage of visitors who make a purchase on the website. Additionally, secondary metrics like average order value, bounce rate, and time spent on the product pages can also be considered to gain more insights. Variant A provides static product images to customers whereas Variant B provides dynamic images which changes based on customer's customization selection.

Business Requirements:

• Improve Conversion Rate: The main objective is to increase the number of visitors who complete a purchase, thereby boosting overall revenue for the e-commerce business.

• Enhance Customer Engagement: The website should strive to keep visitors engaged and interested in the products, increasing the likelihood of a purchase.

• User Experience: The website should be easy to navigate and visually appealing to encourage visitors to stay and explore the products.

• Data-driven Decision Making: The A/B test should be conducted scientifically, with sufficient sample size and duration to ensure reliable results.

A/B Testing:

The control group is exposed to the current version of a website or product (Variant A). The treatment group experiences the new version or variant being tested (Variant B).

CR (A) = 11.81% and CR (B) = 13.17%

By looking at this, we can see that Variant B is more effective.

Checking statistical significance:

• We are using Bernoulli distribution (convert/purchase or do not convert/purchase)

• Control group purchased: (n)ct * (p^)ct = 15640 * 11.81% = 1847 = (x)ct

• Treatment group purchased: (n)tr * (p^)tr = 15640 * 13.17% = 2060 = (x)tr

• Test statistic follows Z-distribution

• Hypothesis:

Null Hypothesis (H0): There is no significant difference between Variant A and Variant B. Any observed differences are due to chance or random variation. i.e. d = 0, (p^)ct = (p^)tr

Alternative Hypothesis (Ha): There is a significant difference between Variant A and Variant B. The observed differences are not due to chance but are a result of a real effect.

Calculations:

d = (p^)tr - (p^)ct = 0.1317 - 0.1181 = 0.0136

p = (xtr + xct)/(ncr + ntr) =12.49% = 0.1249

Pooled SE = Sqrt(p(1-p)((1/nct)+(1/ntr))) = 0.0037385

Test statistic (TS) = ((p^)tr - (p^)ct ) / SE = 0.0136 / 0.0037385 = 3.6378

Critical Z-score (alpha = 0.05) = 1.96

If TS > 1.96 or T < - 1.96, reject H0

Since TS = 3.6378 > 1.96, we reject H0

Therefore, test is statistically significant

Conclusion and Recommendation:

After running the A/B test for a sufficient duration (2 weeks) to collect meaningful data, let's analyze the results:

Variation B (E-commerce with Dynamic Images) performed better:

• Conclusion: The A/B test indicates that the inclusion of dynamic images on product pages positively impacts user engagement and conversion rates.

• Recommendation: Implement Variation B on the e-commerce website to capitalize on the interactive and visually appealing nature of dynamic images. Continue monitoring metrics to ensure consistent performance.

It's essential to consider other factors such as page loading speed, image quality, and the nature of the products being sold. It's also worth running periodic A/B tests in the future as user preferences and technology evolve.

To further optimize the e-commerce website:

• Personalization: Use data from user behavior and preferences to offer personalized product recommendations and experiences.

• Customer Reviews: Integrate customer reviews and ratings on product pages to build trust and influence purchase decisions.

• Mobile Responsiveness: Ensure the website is fully optimized for mobile devices, as mobile shopping continues to grow in popularity.

• Streamlined Checkout Process: Simplify the checkout process to reduce cart abandonment rates and improve overall conversion.