How to tell if your website change helped

What should you do before you change anything?

Three things, and they take a minute. After the change it's too late to do them honestly.

1. Choose one number. Contacts per 100 visits, orders per 100 visits, or forms sent per 100 started. Use a rate, because a busy month inflates a count. Pick the one the change is meant to move.
2. Write the expected result and the date. For example: "Shorter form, live 16 October. Expect forms sent per 100 started to rise." Put it on your analytics timeline if your tool has one.
3. Change one thing. If you ship a new form, a new heading and a new price together, you won't learn which helped.

Choosing first matters. Look at ten numbers afterwards and one will have moved by luck, and you'll want to believe it.

Which dates should you compare?

Use equal stretches of full weeks on both sides, so each has the same weekdays. Don't count the change day itself. Avoid anything that makes one side unusual: a sale, a public holiday, a press mention, a big email. If you can't avoid it, note it, because it may explain the result better than your change does.

Don't stop as soon as the number looks good. Rates drift up and down every day, so if you check daily and stop on a good day, you'll often find a win that isn't there. Decide the end date first. Two full weeks is a sensible minimum for a small site, and it's often not enough, as the table below shows.

How do you test whether the difference is real?

Use the standard test for comparing two proportions, which NIST's handbook gives in full. In a spreadsheet, put successes and totals in cells:

x1 = after: successes     n1 = after: total
x2 = before: successes    n2 = before: total

pooled:  =(x1+x2)/(n1+n2)
z:       =(x1/n1 - x2/n2) / SQRT(pooled*(1-pooled)*(1/n1+1/n2))
p-value: =2*(1-NORMSDIST(ABS(z)))

We read the p-value the way our analyst's own test does. It writes the verdict in words so nobody has to remember:

A worked example: is 6% to 6.9% an improvement?

Suppose you changed something on our demo marketplace on 16 October, and compared 5 to 15 October against 16 to 31 October. Here are the dashboard's visit outcomes for the two periods:

Converted visits rose from 109 of 1,830 (6.0%) to 198 of 2,867 (6.9%). In a report, that would read as a rise of 16%. Anyone would call it a win.

Run the sums: z is 1.29, and the p-value is 0.199. It could easily be chance. You can see it another way, using the 95% ranges our test also reports: 5.0% to 7.1% before and 6.0% to 7.9% after. They overlap heavily.

These dates also break the rules above: 11 days against 16, and the second period holds a launch day with 471 visits, which changes the mix of sources. So even a clear result would need a closer look. That's the second check.

How big a change can you actually detect?

Before you run anything, see whether your traffic can answer the question. We ran the test above for a range of sizes. Here's the smallest result that passes p under 0.05, with the same number of visits on each side:

With 1,000 visits per side and a 5% rate, you need a jump to about 7.1% to read it as real, a gain of over 40%. A site with 500 visits a month needs months to judge a modest change. That's a real answer: if you can't measure it, make the change because it's plainly better, then watch for harm, and don't pretend you've proved it.

What else could have moved the number?

Even a clear result only says something changed. Before you credit yourself, check:

• The traffic mix. Split your rate by source on both sides. If each source is flat and the total rose, you got more of a better source, and your page may have done nothing.
• Device. A share shift toward phones or computers can move a rate on its own.
• Season and campaigns. Check the same dates last year if you have them, and your marketing calendar.
• Tracking. A new tag, a renamed button or a consent change can move the number without changing anyone's behaviour.
• A side effect. A shorter form may send more enquiries of lower quality. Glance at what the enquiries are like.

When should you run a proper test instead?

When it's a big decision and you have the traffic. Showing the old and new versions to visitors at the same time removes season and campaigns from the question, since both sides live through the same days. For a change you can't split, such as a price list or a new menu across the site, before-and-after is what you have.

How MetricMind shortens this

• The comparison is built in. Pick custom dates and turn on the previous period. A period still running is compared up to the same clock time.
• Notes on the chart. Drag across a day to add a note, or connect GitHub so releases appear by themselves.
• Ask "is this better, or is it chance?" The analyst runs this same test and shows the query it used. Questions come with every plan, with a monthly allowance.
• Experiments split visitors for you. You state what you expect and one deciding metric, you see how long it needs before you start, and no verdict appears before the planned sample and seven days.
• Fix checks re-measure a finding's fix after some days and say whether it held. These come with Business and Scale. Written investigations start with Pro.

It can't tell you why a number moved if the cause is offline, and it can't make a small site's numbers bigger. For the noise you see from day to day, read real change or normal swing. To pick what to change first, start with visitors who don't become customers.