Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.orderprotection.com/llms.txt

Use this file to discover all available pages before exploring further.

Setting Up Your Test Settings in Shopify

Before you can start your A/B tests within the Order Protection app, you need to configure a few settings.
1

Enable a New Custom Pixel

From Settings, click on Customer Events.Once in the Customer Events settings, click Add custom pixel. Name it “Order Protection A/B Testing” or something similar to know the testing is originating in the OP app.
2

Add the Code Script

Once the pixel is added, add the custom script below.
analytics.subscribe('checkout_completed', async event => {
  const STORE_URL = 'kysonteststore4.myshopify.com';
  const API_BASE = 'https://api.production.orderprotection.com/v1';

  function transformSettings(settings) {
    return {
      variants: settings.variants,
      settings: {
        default_op: settings.automaticallyAddToCart,
        dynamic_price: true,
        dynamic_price_range: settings.rules.map(rule => {
          const term = rule.terms.customer || rule.terms.brand;
          return {
            value: rule.min * 100,
            max: rule.max * 100,
            insuranceCost: term.type === 'percentage' ? term.amount : term.amount * 100,
            isPercentage: term.type === 'percentage',
            paidBy: rule.terms.customer ? 'customer' : 'brand',
          };
        }),
        splitTestSettings: settings.splitTestSettings || null,
      },
    };
  }

  function extractSplitTestFromWidgetConfig(widgetConfig) {
    const configArray = Array.isArray(widgetConfig) ? widgetConfig : [widgetConfig];
    const configWithSplitTest = configArray.find(c => c?.splitTestSettings?.token);
    return configWithSplitTest?.splitTestSettings || null;
  }

  async function fetchWidgetConfig(widgetType) {
    const response = await fetch(`${API_BASE}/widget/config?id=${STORE_URL}&type=${widgetType}`);
    if (!response.ok) return null;
    const widgetConfig = await response.json();
    return extractSplitTestFromWidgetConfig(widgetConfig);
  }

  async function checkWidgetViewed(testId, token) {
    const response = await fetch(`${API_BASE}/tests/${testId}/${token}`);
    return response.ok;
  }

  async function getWidgetStyleSplitTest(testId) {
    const widgetTypes = ['CHECKOUT', 'CART'];
    const results = await Promise.allSettled(widgetTypes.map(fetchWidgetConfig));
    const splitTestSettings = results.find(r => r.status === 'fulfilled' && r.value)?.value || null;

    if (!splitTestSettings?.token) return null;

    const wasViewed = await checkWidgetViewed(testId, splitTestSettings.token);
    if (!wasViewed) return null;

    return splitTestSettings;
  }

  async function getNonWidgetStyleSplitTest(testId) {
    const storeResponse = await fetch(`${API_BASE}/quote/insurance?store_url=${STORE_URL}`);
    if (!storeResponse.ok) return null;

    const rawStoreSettings = await storeResponse.json();
    const store_settings = transformSettings(rawStoreSettings);
    const splitTestFromStore = store_settings?.settings?.splitTestSettings;

    if (!splitTestFromStore?.token) return null;

    const viewedWidgetResponse = await fetch(`${API_BASE}/tests/${testId}/${splitTestFromStore.token}`);
    if (!viewedWidgetResponse.ok) return null;

    return splitTestFromStore;
  }

  function sendTestEvent(splitTestSettings, orderId, purchasedOp) {
    fetch(`${API_BASE}/tests/event`, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({
        type: 'checkout:completed',
        testId: splitTestSettings.id,
        sessionId: splitTestSettings.token,
        metadata: {
          isOriginal: splitTestSettings.isOriginal,
          settingsType: splitTestSettings.type,
          orderId,
          purchasedOp,
        },
      }),
    }).catch(err => console.error('Failed to send test event:', err));
  }

  async function initSplitTest() {
    try {
      const testResponse = await fetch(`${API_BASE}/tests?store=${STORE_URL}`);
      if (!testResponse.ok) return;

      const hasSplitTest = await testResponse.json();
      if (!hasSplitTest?.active || !hasSplitTest?.test) return;

      const isWidgetStyleTest = hasSplitTest.test.type === 'WIDGET_STYLE';
      const testId = hasSplitTest.test.id;

      const splitTestSettings = isWidgetStyleTest ? await getWidgetStyleSplitTest(testId) : await getNonWidgetStyleSplitTest(testId);

      if (!splitTestSettings?.token) return;

      const purchasedOp = event.data.checkout.lineItems.some(
        item => item.variant.product.vendor === 'Order Protection',
      );
      const orderId = event.data.checkout.order.id;

      sendTestEvent(splitTestSettings, orderId, purchasedOp);
    } catch (error) {
      console.error('Error in initSplitTest:', error);
    }
  }

  initSplitTest();
});
It should look something like this.Before you save the file, you need to swap out the following in quotations const STORE_URL = ‘kysonteststore4.myshopify.com’; with your Shopify store URL. The store URL can be found at the top of the left navigation in settings as well as at the top of the current custom pixel page you are populating. This URL must be your myshopify URL for it to work properly. The Shopify URL will be something similar to: {store_name}.myshopify.com. Save the custom pixel and you’re done! 🙌
3

Starting an A/B Test in the Order Protection App

Now that you have configured your custom pixel, you’re ready to start A/B testing. Log back in to app.orderprotection.com, navigate to to the A/B Tests tab in the left navigation and click on Start New Test to get started. Session data will start rolling into the test analytics within a few minutes. Expect 1-2 weeks for results to reach statistical significance depending on your store traffic volume.

What You Can Test

When creating a new A/B test, you’ll choose from three test types. Each type isolates a different variable so you can measure its impact on conversions and revenue. Creating a new A/B test

Widget

Test whether the Order Protection widget defaults to on or off for customers at checkout. This is the most common test type for measuring opt-in vs. opt-out behavior.

Price

Test different pricing structures to find the optimal balance between customer conversion and revenue. Compare your current pricing against a variant with adjusted rates.

Widget Style

Test different widget designs and layouts to see which style drives the highest engagement and attach rate.

Configuring Your Test

After selecting your test type, you’ll configure the test details. Each test type has its own configuration options.

Pricing Tests

When you select Price as the test type, you’ll see your current pricing displayed as the Original variant. This is pulled directly from your store’s existing price settings and cannot be edited within the test — it serves as your control group. Create a Variant with adjusted pricing to test against the original. You can modify both the flat rate and percentage-based pricing tiers for different order value ranges. Configuring a pricing A/B test
Once you start a test, you will not be able to edit the variants to maintain the integrity of the test. You can always end the test and start a new one if you need to adjust your pricing.

Widget Style Tests

Widget style tests let you compare two different widget designs to determine which performs better. Before setting up this test type, you’ll need at least two widget configurations created in the Customizations section.
To learn how to create and customize widgets, see the Widget Customization guide.
Navigate to Customizations > Widgets to view your available widget configurations. You’ll need both a published widget (your control) and at least one additional widget to use as a variant. Available widgets in Customizations When you select Widget Style as the test type, a widget picker will appear showing your available widget configurations. Each widget has an Add To A/B Test button — select the widgets you want to compare. Selecting widgets for your A/B test Once selected, you’ll see a preview of each variant showing the control widget and the variant widget side by side. Widget style test with control and variant

Checking Your Results

Once your test is running, you can monitor results in real time from the A/B Tests tab. Click on any active or completed test to view the results dashboard. A/B test results dashboard
All metrics are calculated starting from when the A/B test was launched — not from when your store was created or when Order Protection was installed. This ensures a clean comparison between your test variants.

Understanding Your Metrics

The results dashboard shows a summary row at the top with key winner indicators, a time-series chart for tracking trends, and a detailed comparison table. Here’s what each metric means:
MetricWhat It Measures
ViewsThe number of times the Order Protection widget was displayed to customers in each variant.
ConversionsThe number of completed checkouts per variant, regardless of whether Order Protection was purchased.
CVR (Conversion Rate)The percentage of widget views that resulted in a completed checkout. Calculated as Conversions / Views.
Attach RateThe percentage of orders that included Order Protection, per variant. Calculated as Protected Orders / Total Orders.
ProfitEstimated profit generated from Order Protection fees based on your revenue share agreement.
Views represent the total number of times the Order Protection widget was rendered and visible to a customer during their shopping session. Each unique session where the widget loads counts as one view.The results table breaks views down per variant, so you can see whether traffic was split evenly. A significant imbalance in views between variants may indicate that the test needs more time to normalize.
Conversions track the number of completed purchases (checkouts) within each test variant. This metric captures all orders — both those that included Order Protection and those that did not.The CVR (Conversion Rate) is calculated as:CVR = Conversions / Views x 100The CVR Compare to Original column shows the percentage-point difference between each variant and the original, making it easy to spot whether a variant is helping or hurting overall checkout completion.
Attach rate is the most important metric for most A/B tests. It measures the percentage of total orders in a variant that included Order Protection.Attach Rate = Protected Orders / Total Orders x 100For example, if 3,409 out of 5,171 orders in the Original variant included Order Protection, the attach rate is 65.93%. A higher attach rate means more customers are opting in to protection, directly increasing revenue.The Attach Rate Winner in the summary row highlights which variant achieved the highest attach rate.
Profit is calculated based on your revenue share agreement with Order Protection. This represents your store’s estimated earnings from Order Protection fees collected during the test period.The calculation considers:
  • Total Order Protection fees collected per variant (based on the pricing tier and order values)
  • Your revenue share percentage — the portion of each fee that goes to your store
  • Claim costs are not factored into this metric; it reflects gross revenue share only
The Profit Winner in the summary row indicates which variant generated the most revenue for your store. When evaluating profit, consider it alongside attach rate — a variant with slightly lower attach rate but higher pricing may still generate more total profit.

Variant Details

At the bottom of the results page, you’ll find Variant Details showing the exact configuration for each variant. This includes:
  • Whether Order Protection was on by default
  • Whether the No Order Protection option was provided
  • Any widget style differences between variants
  • For Price tests, the full pricing tiers and rates for each variant (flat rates, percentage-based rates, and order value ranges)
Use this section to recall exactly what was being tested, especially when reviewing results from older tests.