Frank Hopkins spends his days as an Experimentation Analyst at the BBC. We caught up with him after Numberwang, and he happily shared with us what he gets up to at the BBC on a day to day and gave us some interesting insight into how to be successful in all things experimentation!
Frank HopkinsBBCNumberwangMy name is Frank Hopkins and I am an Experimentation Analyst for the BBC. For the last 5 years, I’ve worked as an “experimenter” of some capacity; with a background in exercise mechanics and statistics.
Frank HopkinsBBCMy role focuses on best experimentation practice across all BBC products – ranging from iPlayer to CBBC and Newsround. This often requires me to get in and amongst teams and help in the experimentation process, and on any given day I could be helping with hypotheses building, prioritisation, QA’ing experiments or performing post-hoc analysis to harness meaningful insight from experimentation data.
iPlayerCBBC NewsroundDue to the size of the BBC, work is both genuinely and generally exciting on a day-to-day basis. As part of the BBC, I have worked with a variety of experiments, which have differed in size and complexity; from large-scale MVT experiments and homepage re-designs down to smaller and more nuanced changes on sites, such as re-ordering modules and prioritizing certain content items.
For those of you who work in digital, you may be familiar with the term “experimentation” or may have heard it being thrown around. If not, I will briefly define experimentation and what people mean (or think they mean!) when they say “let’s test this”.
"Experimentation, in a digital context, refers to the execution of a controlled, balanced experiment on a page, site and/or app where equal distribution of traffic is delivered to two or more variations of the same entity."
The purpose of experimentation is often misconstrued and it’s important to stress where its true value lies. To help “product” people understand the true value of experimentation and how it should be implemented, I take myself back to my days as an applied scientist, studying Biomechanics.
In an applied science context, the body of literature is examined, a gap or unanswered questions are identified, a reasonable hypothesis is presented, appropriate methods for answering the hypothesis are conducted and statistical analysis is performed. Following the results, conclusions are drawn and future research is proposed. In theory, this is how experimentation should be conducted in a digital and/or business setting. Experimentation tools should be used to test scientifically or numerically backed hypotheses. Such backing takes many forms, including holistic analysis performed by central analysts and/or data scientists, field/market research, qualitative analysis or UX analysis.
The important thing to understand, is that having any of the above is better than having no data to back a hypotheses; that’s when it can easily get into the realm of “hunch-based” hypotheses. This is something to steer away from, because we don’t want to invest our time and development resources into an assumption.
Moral of the story: some data is better than no data, because we are always working with a certain volume of numbers (either limited or in abundance), but the statistical rules applied to their significance are governed by man-made rules; so even if a hypothesis is backed by < 100 data points in a market research survey, that’s better than thinking “this might just work”.
Top Tips for Experimentation Analysts:
translationOptimizelyIf you have a hunch that changing something within a digital product may have a positive impact on performance metrics, perform analysis on the site and gather some data to rationalise your assumptions; this will help in understanding whether you are either utterly correct, or you are completely shooting in the dark.
Either way, both of these are useful manifestations because, as both analysts and human-beings in general, we don’t question how we’ve come to a certain conclusion – so it could be a useful exercise to simply validate or dismiss your assumptions.
Thank you to Frank Hopkins for sharing his inside knowledge with us, we really look forward to following your experimentation journey! If you are interested in a role like this, please see our Analytics Jobs and check out our North West Marketing Analytics Sector Skills Survey!
Frank HopkinsAnalytics JobsNorth West Marketing Analytics Sector Skills Survey!herehereherehereclick here.