OPTO – NEW NORMS 2022

We have continued the development and maintenance of OPTO, ensuring continuation of OPTO as a high-quality solution.

This time we have looked at the norms, both updating existing norms and creating new norms. And due to the fact, that OPTO is being sold vastly in many countries, we now have data enough to create 11 different norms. This means, that Master can now offer customers 10 local norms, and an international norm based on more than 16.000 responses. This is an enormous amount of data, and it secures excellent norms across countries. 

Maintaining high quality of our solutions involves updating norms based on the newest data reflecting the local population of the norm. But what is a norm? Why is it so important when we look at test results? And why can scores change when a norm is updated? This article will go into depth answering these questions.

What is a norm?

A norm can be described as a baseline we use to look at an individual’s results. In psychological measurements we are interested in relative scores, where we can compare scores and results to others.

Let’s imagine a test, which can measure how extroverted a person is. If a person scores 15 on that test, what does that mean? Is the person then an extrovert? We cannot really conclude anything just from the raw score alone. To gain more information we need a group of people who have also taken the test, who we then can compare the score to. Such a group is called a norm group.

To get a norm group we let at least 300 people take our test, we now find that the average score on the test is, let’s say, 11. We can now conclude that our test person with a score of 15 is more extroverted than the average person of the norm group. In Master we convert these raw scores into a recognizable scale as the STEN scale (or Standard Ten scale) using the mean and standard deviation of the norm group. This enables the comparison of an individual´s score to other scores using the same norm.

For a norm group to be representative, is must include enough responses from a sample of people that reflect the target population. For instance, a norm for a specific country should mirror the demographics of the population in that country (gender, age, educational level etc.). Generally, norm groups should at least include 300 responses for high stakes decisions, and any norm that is less than 10 years old and has more than 1,000 representative responses, is considered by the European Federation of Psychologists' Associations (EFPA) to be excellent in relation to making high-stakes decisions such as recruitment (Evers et al, 2013). We want the norm group to be representative of the population. To ensure representativeness, we weigh the norm group by the demographic of the population. Meaning, that if the demographics for Denmark, show that there are 52% women in the working population, but we only have 49% of the test responses from women, then we give their responses more weight, so we accomplish having the same weighted distribution as the actual demographics.

Why are norms important?

Norms help us to ensure, that an individual’s responses are always compared to a representative group. This is a way to beat unconscious discrimination in a work population, which could be the case if women for example were compared to a group consisting of only men. Or if people from lower income jobs, were only compared to people with high income jobs.

The norms we choose in a recruitment setting, can have high impact on the results from the candidates. And when choosing norms, the test administrator should consider who they want to compare the candidates to. If a recruiter is looking for a person to be a part of a Spanish workplace, it is important to use a Spanish norm. Even though the candidate might be from another country, the recruiter should be interested in understanding how that person will interact in a Spanish setting. If the recruiter is looking for an extroverted person, the recruiter would want to know if the candidate is extroverted in a Spanish sense, as this might differ from other cultures.

Typically, a test provider does not have local norms to cover the whole World but having high quality solutions means having a diversity in local norms. And having an international norm, that truly includes a large international population. This is also a focus from the external agencies like British Society of Psychology (BPS) or Det Norske Veritas (DNV) when they review our solutions, and accredit their use in HR.

Why can scores change?

There are roughly speaking 3 elements, which have an impact on the score, and can eventually change the score of a candidate:

First, when using a specific norm group, it is important to bear in mind that responses are always keyed to that group. If you are using a norm group based on for example, Spanish people of working age, all results are interpreted in that respect - how the test person compares to Spanish people of working age. It therefore follows, that interpretation of the individual can change significantly based on who we compare it to, meaning by which norm is selected.

So, a raw score of 15 might be above average when comparing to a Finnish norm group, and the person will in that case get a result of for example 7 on the STEN scale, and you could label this person as extroverted using the Finnish norm. But when using for example a Spanish norm, 15 might be below average, and when we then convert the raw score into a STEN scale using the Spanish norm it might result in for example 5 on the STEN scale, and you would label the same person as less extroverted, even though the personality of the person is the same, and the measurement is the same. But that is because the result is keyed to the chosen norm group, and when changing norms, we change the baseline of comparison.

Secondly, when we update an existing norm, we use the most recent data we have, to give the best updated picture of the norm group. We do so, as we know that norms change over time. For example, age groups change over time, as age groups are composed of different people with different life experiences (Woods, R. and Clare, L., 2015). Therefore, we expect a general shift in the average of people’s answers over time. And when large life experiences occur, we would expect this to have a general effect on a population. Therefore, it is crucial, that norm groups are regularly updated and that they are never more than 10 years old (Evers et al, 2013).

And third, the way we use norms is based on complex algorithms, that from time to time is reviewed and updated to reflect the most recent research on the area. This is the case with OPTO in the norm review, where we also have updated the algorithm, we use to calculate the conversion of test results using norms into STEN scores.

The result of the norm update is that customers, users, and test takers can rest assure, that OPTO is up to date, and Master International can guarantee the high quality of the solution for many years to come. Of course, the work is never finished and the ongoing process of updating and ensuring high quality is what we in the psychology team at Master International live for. For now, we are extremely proud of the many norms we now offer (10 local + International), and the large amount of data that is the basis of the norms (+16,000), and we look forward to future updates coming.

Tabel 1: OPTO norms available as per April 2022.

Want to know more about OPTO: LINK

References:

Evers, A., Muñiz, J., Hagemeister, C. Høstmælingen, A., Lindley, P., and Sjöberg, A. (2013) EFPA Review model for the description and evaluation of psychological and educational tests. Version 4.2.6

Woods, R. T. and Clare, L. (edit.) (2015) Handbook of the Clinical Psychology of Ageing. John Wiley & Sons.