Case Study #1 – lostroombcn.com

Case Study #1 – lostroombcn.com

History of conversion rate optimization agency for Escape Rooms.

About a year ago an idea to create “the best” service for conversion rate optimization was born. A couple of the first months I spent getting closer to legal regulations, preparing designs, registering the domain, configuring the servers and infrastructure as well as some other stuff. Then the first programmer was hired. He created a part of the service during the first month and disappeared during the second. Then I was looking for marketers (nowadays people say it’s important to create content and advertisement). Then the first programmers were hired for the full work day, the marketer was changed, a drop of blood and sweat and voila – 10 months later we got an MVP. And it was so viable, that we almost immediately found a client (no surprise here actually, he is a friend).

That was a long story short. Now the real first-hand example how to run experiments.

Beginning.

We start our work with the Growity code installation and existent events and goals on the client’s website. One part of our service is an enhancer for Google Analytics. It adds measurement of interactions with different elements on the website including an actual view of the element, clicks, depth of scroll, whether the page is seen (not just opened in a tab). All that is needed for proper conversion optimization.

Test Zero or experiment for the sake of experiment.

To make sure the whole process was correct we ran a fake experiment which did not change anything on the website. We just changed one symbol in the footer. That helped us understand that the control and experiment groups were split correctly and conversions on different checkpoints through the checkout/order process were the same (or more precisely the difference is not statistically significant)

Test One or why you cannot trust CRO-systems.

Competitor analysis revealed a very interesting thing – they hide the website unless the execution code is loaded (for instance VWO). It helps to avoid flickering effect when the first original element is loaded and then the test is executed. It looks like a usual website loading. Nothing wrong with that except one big thing – the system cannot measure how this affects conversion, if it does of course (usually yes, it does https://www.fastcompany.com/1825005/how-one-second-could-cost-amazon-16-billion-sales).

So, it was time for us to find out the truth. And you know what? It relatively reduces the overall conversion by 5% (4.63% with the significance interval 90%).

Google Optimize does not add this to your website which makes it more difficult to use, but safer regarding of conversion. If you feel confident with this type of systems and you don’t need more than 3 simultaneous tests, use a free version of Google Optimize).

In Growity service we solved the problem in a different manner. For each test, we generate so-called Style Guard which prevents flickering on the level of CSS-styles. An individual approach to each client and test – that’s what makes us proud of what we are doing.

Test N or why you cannot trust even yourself.

Naturally, we worry about business no less than its owner. That’s why we check results and analytics on a daily basis. This process helped us find a couple of crucial issues. Well, what did you expect from MVP… we would say if the issues were on our side. But no, the problem was on the client’s side. On the website lostroombcn.com (btw, please visit when in Barcelona) every visit was tracked twice. This is a pretty popular bug – please check that your Google Tag Managers and Google Analytics are not overlapping.

The second issue was much trickier. We could not find N percents of orders in our Growity. We checked every possible combination of factors and localized the problem – iPhones. Obviously, we were thinking our code was broken. Nope. Well, another suspected reason was adv. blockers. Nope again. Mysteriously our code did not run at all. So, we checked the actual source code of the page for iPhones and got really shocked. There was no Growity code there. How? It turned out it was an old-good bug on the client’s website. And the problem with this issue is that CRO-system just can’t detect it – because the code does not exist for some circumstances (in our case – iPhone, yours can be different).

Next time, we will write about all 5 running tests and their results of course.

Stay tuned!