What’s the scientific verdict on game-based assessments?

No longer the new kid on the block: What’s the scientific verdict on game-based assessments?

It’s been about a decade since Game-Based Assessments hit the scene (at least in the UK) making promises to fundamentally disrupt the world of psychometrics and recruitment. Back in those heady days, Game-Based Assessments (or GBAs – a three-letter-acronym that never caught on quite as well as SJT) were very much the smashed avocado on sourdough to the traditional fried breakfast of psychometric testing.

Did the initial buzz and marketing promise of the new technology catch on? In the VHS world of psychometrics, were GBAs the Blu-Ray or the Betamax? 10 years on, we can step back, catch our breath, and review what the science and practice of GBA have to tell us.

Fad or thing?

When a new psychometric or technology becomes the hot, new, must-have assessment it takes a while for the dust to settle enough for us to understand whether it’s a passing fad or an actual thing that will alter the landscape before becoming a part of it.

Because I’m very old (old enough to remember the New Kids on the Block and VHS/Betamax at least), I seen this wheel turn quite a few times. New models and technologies like EQ, SJTs, adaptive testing, big-data create a storm of breathless articles, over-subscribed conference sessions, a sudden rush-to-market of competitor products before cooling down as case studies, user experience, and research combine to clarify how enduring the new thing will be.

In terms of workplace assessments, this means understanding:

• Does it actually work the way it claims?
• Does it offer genuine or distinctive advantages for users and candidates?
• Are there both pragmatic and evidence-based reasons for adopting it?

 

What is game-based assessment?

Despite being around now for quite a while, there remains pretty widespread misapprehension about what a GBA actually it. And not just among practitioners, but also in research. Reviews of scientific literature in 2021 and 2022 looked back over preceding years of research and found that many papers confounded three overlapping approaches:

• Gamification – where we apply techniques and touches from gaming mechanics into more traditional forms of assessment, in order to encourage completion rates
• Gameful design – when developers build new assessments based on gaming mechanics
• Game-based Assessment – where developers build a game to test a specific psychological construct

It’s confusing right? To bring it to life a bit, you can use gamification techniques to encourage people to do something they might not other wise do voluntarily. People might be more likely to throw their rubbish in a bin if it has a target drawn on it. You make the behaviour you want to elicit into a game. In testing or assessment, you can do the same by adding gamification touches – think whistles, bells, rewards.

In gameful design, you don’t need to encourage people to complete it, because you’ve designed the test to be as game-like as you can, while still measuring personality/ability on a traditional way.

Game-based assessments are built as games. But as part of their development process, the data captured during gameplay is used algorithmically to generate scores relating to psychological constructs, So, a game of Space Invaders could be built to measure extraversion. Or a game about collecting gold coins might measure impulsiveness.

 

Do GBAs meet their early claims?

When GBAs hit the assessment world, there was much discussion about potential upsides and downsides.

Benefits of game-based assessment:

1. They are definitely games. They test without looking like a test and arguably offer an engaging, visual, interactive, and candidate friendly way of delivering assessments; certainly compared to a maths test
2. Being a game, they may help candidates to feel more relaxed and less anxious compared to being tested
3. They’re down with the kids. Younger, digitally native candidates may feel more at ease during the recruitment process
4. Can differentiate your recruitment process from competitors – why use boring old psychometrics when you can use a game?

 

Downsides of game-based assessment:

1. Games cost a fortune in R&D. More expensive to design, develop and implement vs more established psychometric technologies
2. Not suited to all test takers? Potential to exclude people who don’t want to play a game to job
3. Not suited to all employer brands? Not every organisation wants to represent its culture and values through a game when attracting and assessing new workers
4. Potential to disadvantage less tech savvy candidates at a disadvantage, or place high demands on candidate’s technology
5. Practice makes perfect – performance on games generally changes with practice while the psychological trait being measured does not. You can be better at Space Invaders without becoming more extravert.

 

Does it offer genuine or distinctive advantages for users and candidates?

For assessment and psychometrics professionals, an enduring difficulty with GBA is the mysterious link between the gameplay and the psychological score it produces. While a test is pretty transparent once you look under the bonnet (you can see where the score comes from), with GBAs the measurement is based on a blackbox full of data-points gathered during the candidate’s gameplay. This form of ‘opaque’ assessment is nothing new (clinical assessments have long used the technique of disguising what they’re measuring to stop people ‘faking-bad’), so we know from research outcomes that it makes it much more difficult to correlate scores from assessment with actual behaviour and job performance.

The literature reviews of GBA research identifies mixed results in terms of these outcomes, while more traditional approaches to assessment (even those using gameful design) continue to offer more tangible evidence of performance prediction.

For employers, potential concerns suggested by the research papers over the last 10 years relate to issues around legal issues and the effect of GBA on inclusion and diversity. Currently there is a shortfall of evidence in terms of the extent to which GBA attracts or deters protected groups in recruitment processes. Additionally, research confirming the predictive power of GBA compared to established assessment methods fives a mixed picture.

For many candidates, the opportunity to play a game rather than take a test seems to offer a self-evident benefit. The video games industry as long out-performed the film business, underlining the dominant position of gaming in popular culture and entertainment. However, not all candidates are the same and consideration should be given by employers to the valuable talent who might be alienated by a GBA route to employment.

 

Are there both pragmatic and evidence-based reasons for adopting it

Ten years down the road, there is now research available to help guide evidence-based decisions about whether to adopt, or retain, a GBA approach within an assessment and selection process.

But evidence-based decisions can also be guided by additional considerations. In terms of GBA, assessment users can also ask:

• Will it offer my candidates an inclusive, engaging experience that will convert into better attraction and higher offer-acceptance rates?
• Does the cost-benefit ratio offer advantages over alternative methods of assessment?
• Does it differentiate my employer brand from my competitors in the employment market?
• Will the results give me meaningful information to help me make better hiring decisions?
• Can I adjust the assessment for candidates with needs and conditions to match reasonable adjustments in the job?
• Is it accessible to all candidates, with a varying access to technology?

The bottom line is that these are questions a user should pose before adopting any new assessment, regardless of whether it is a GBA or established form of psychometric. The responsibility is for the platform provider to be able to provide the evidence you need to make your decision.

 

The future of assessment

Game-based Assessments were the future once. Ten tears later, they are still with us, But for test users looking to be early adopters of the next big thing, what lies head? For this we need to consult our crystal ball. So, cross my hand with gold and wait while the mists begin to clear….

• What games can often lack in terms of real-life emulation is presently offered by more behavioural/situational judgement type assessments. However both AR and VR already offer the potential for assessments to be delivered to candidates in an engaging and immersive environment. The main roadblock currently is the absence of a routinely available means of remote testing
• Blockchain based assessment maybe increasingly used by candidates to minimise the number of tests they need to complete for recruitment and selection processes.
• Candidates can complete a psychometric once, blockchain their results and make them available to different employers on request. This democratises the assessment process and gives candidates ownership of their personal, psychological data
• Overthrowing the machines. The use of AI/machine-learning based decision making in recruitment has been found to embed human biases, which has seen some high-profile adopters of this technology pullback from its use. These experiences, along with tighter regulations over automated decision-making processes may see a sea change back towards more human, accessible, transparent forms of assessment and decision making

 

Sources:

Examining Game-based Approaches in Human Resources Recruitment And Selection: A Literature Review and Research Agenda (2021). https://hdl.handle.net/10125/70773

Game?based, gamified, and gamefully designed assessments for employee selection: Definitions, distinctions, design, and validation (2022) DOI: 10.1111/ijsa.12376

https://blogs.lse.ac.uk/careers/2020/10/19/game-based-assessments-what-why-and-how-to-succeed/