Fundamentals of gameplay metrics work (pt. 2/2): explorative vs. hypothesis driven
How do we work with gameplay metrics at the fundamental level? In the first part of this post the concepts of analysis (“breaking down things”) and synthesis (“putting things together”) were introduced as two fundamental approaches used here at Game Analytics to help categorize and define workflows. In the second part, we dig deeper and look at the factors driving the work.
Both synthesis and analysis can be applied to explorative work (where we look for patterns in data) or hypothesis-driven work (where we have an idea what the answer is and need to confirm or reject the idea).
Alternatively, we could have a hypothesis stating that the amounts of player deaths on a certain map correlates to the perceived difficulty level of the map. Checking metrics data on player death events with feedback from research study participants can either lead to confirmation or rejection of the hypothesis, possibly leading to the formulation of a new hypothesis.
A commonly applied method in game data mining to answer these kinds of questions are prediction analysis – the application of specific algorithms to predict something, e.g. which users that will convert to paying users, or when a person will stop playing (more on prediction analysis in future posts).
In practice, as soon as you move outside of the kind of questions that can be answered with synthesis, a quick analysis or standard algorithms, e.g. “what is the number of active users today?” or “what is the average playtime for level 22?”; you often end up mixing hypothesis-driven and explorative work.
Purely explorative questions are in our experience rare – a game developer usually does not have the luxury of throwing a dataset at some people and tasking them to see what interesting stuff they can find. This is not to say that purely explorative analysis of gameplay metrics data cannot be useful, but it is often a kind of blue-water research that companies have a hard time justifying the expenditure of.
Hypothesis, explorative, synthesis, analysis … why do we care about these terms?
The reason is that the fundamental ways we can approach gameplay metrics analysis are ordered according to these terms. They provide us with a means for classifying methods, and a terminology to use when discussing gameplay metrics work, something that is crucial for maturing analytics practices. Finally, this kind of structure provides guidance on planning how to answer particular problems – e.g. considering whether a problem is best solved analytically or using simple synthesis, whether we already have an idea about what the answer is and should test it, or not – and so forth.
Acknowledgements: The ideas and many examples from this post stem from a text that Alessandro Canossa from the IT University Copenhagen; Janus Rau Sørensen, lead game user researcher at Square Enix, and I wrote for an upcoming book on game telemetry.
About the author
Anders Drachen is a veteran Data Scientist and Lead Game Analyst at Game Analytics. His work in the game industry as well as in data and game science is focused on game analytics, business intelligence for games, game data mining, game user experience, industry economics, business development and game user research. He is one of the most published experts worldwide on the topic of game analytics, user research, game data mining, and user profiling, having authored more than 60 research publications on game analytics, user testing, and business intelligence in game development, and co-edited the first book on the topic.