Methods

Results

A priori subscales

Internal consistency varied considerably across the five subscales. The Concern subscale had high reliability (α = .854), whereas the Regulations and Internet subscales exhibited very low reliability (α =.169 and .159, respectively). The Appreciation and Knowledge subscales showed moderate consistency, with Cronbach’s alphas of .643 and .586, respectively.

As shown in Table 3, there are significant correlations between Concern, Regulations and Internet, but not for Appreciation and Knowledge. Further, Regulations correlates with all factors except Knowledge, whereas Internet correlates with all but Appreciation.

Table 3

Correlations Between A Priori Sub Scales

	Regulation	Internet	Appreciation	Knowledge
Concern	.183**	.153**	-.020	.042
Regulations		.300**	.170**	.041
Internet			.003	.053*
Appreciation				-.156**

Since several of the Evers factors showed poor internal consistency in our data, we present a boxplot in Figure 1 for the visual presentation of the different factors. The whiskers overlap considerably, suggesting a high degree of covariation. Moreover, numerous extreme studentised outliers (Grubbs, 1969) – those exceeding twice the interquartile range – along with a number of outliers beyond 1.5 times the interquartile range – significantly affect the Appreciation, Internet and Regulations factors. The factors therefore seem neither sound nor useful.

Figure 1

A Priori Factors Boxplot

Discussion

The main aim of this study was to report on attitudes towards and concerns about test use and test quality from a survey among Norwegian psychologists. Our primary finding is that test quality and regulations are major concerns, as indicated by high ratings on items relating to the need to document test quality, concerns about improper test use and a clear endorsement of more restrictions on who may use tests. This result is reflected in various aspects of the results and echoes the concerns raised in a previous Norwegian study on test use (Vaskinn et al., 2010). They reported that 96% of respondents were ‘somewhat’ or ‘very’ concerned about test quality, a sentiment that was also communicated in our survey. They also revealed differences in the degree to which practitioners checked for quality themselves, whereas our study findings could be interpreted as a call for more assistance in quality checking.

Our respondents reported satisfaction with the training they received in their professional psychology studies but also indicated that they acquired more knowledge after graduating. The Board of Test Policy recently conducted an informal review of university assessment training, and together with the current survey, we find support for our view that training at the basic level seems adequate. Given that challenges related to learning specific tests and acquiring practical assessment skills are addressed (Handler & Meyer, 1998), efforts should be directed more specifically towards providing easily accessible information on test quality to enable clinicians to make informed choices.

In Norway, the health authorities have explicitly delegated the responsibility for test quality evaluation to the individual clinician. However, Vaskinn and Egeland (2012) demonstrated that only half of their respondents checked test quality themselves. The Norwegian Psychologists Association has challenged this decision, arguing that it imposes an excessive demand on clinicians, given the workload and expertise required to evaluate a test thoroughly. The available documentation for many tests is often rudimentary, at times inaccurate (Mihura et al., 2019) or even non-existent, which is also a concern raised by our respondents. Some institutions have made efforts towards systematically evaluating the quality of commonly used test methods, but sustaining this work over time has proven difficult. Additionally, as noted by a reviewer of this manuscript, in a small country like Norway, we have a limited selection of translated tests from which to choose, rendering us more reliant on the quality of the few tests available. This highlights the need for more coordinated efforts to regulate and guide the field.

The lack of effort from clinicians to systematically evaluate test quality is not due to a shortage of engagement or competence; on the contrary, clinicians report that they are invested in assessment and recognise its value. Nevertheless, the challenging and comprehensive task of evaluating test quality requires time and effort with resources that can only be allocated through national initiatives. We therefore find it unreasonable that the individual clinician should bear this responsibility alone. Our current data indicate that the respondents support our view. Test quality and competent use are major concerns for the respondents, who appear frustrated with the current situation.

Several survey items show that our respondents are generally positive about computer- and internet-based testing, particularly among the younger generation. Interest in this issue may have waned somewhat. In clinical practice, paper testing remains widespread. Several factors may account for this, including the lack of appropriate licensing schemes that limit access to computer- and internet-based systems in hospitals and clinics; concerns over data privacy due to the possibility of data being stored on foreign servers; and cost. Practitioners may also be sceptical because they have limited oversight over the conditions under which patients complete tests at home or in other locations. This scepticism appears to be more pronounced among older practitioners, which may be expected. The lower level of concern about illegal copying within the field of work psychology may stem from the reduced prevalence of paper-and-pencil testing, as this is a field that has adopted computer-based testing to a greater extent.

The question of test use restrictions is a more contentious issue. Respondents expressed clear support for greater national and international regulation in this area, preferably through legislative measures. They are more divided on whether other professions and groups should be permitted to use tests based on demonstrated qualifications. There seems to be a clear sentiment towards restricting test interpretation to licensed psychologists. However, this remains a contentious issue among our respondents, reflecting tensions between those advocating for restrictions based on specific educational backgrounds and those supporting access for individuals with proven qualifications. Notably, older and clinical practitioners tend to be more restrictive in this regard, while younger practitioners in work and educational sectors are less so.

The usefulness of grouping the questionnaire items into sub-factors appears to be quite limited in our data. Only the factor Concern achieves an acceptable level of internal consistency; the other two factors are critically low. In addition, the factors correlate to a high degree. We therefore suggest that future analyses of the EFPA survey should focus solely on the individual items – or, alternatively, that analyses at the sub-factor level be conducted on the international data set as a whole.

A major limitation of the test use section of the survey is its restriction to the three most commonly used tests. This means that we do not have a comprehensive test use survey – only indications of test use. With this in mind, our results indicate that performance-based cognitive tests are the most frequently used, even though neuropsychologists constitute a small portion of the sample. Conversely, clinically oriented performance-based tests, such as the Rorschach Inkblot Method, are used relatively infrequently. We can only speculate that users of the latter may not consider themselves to be test-savvy, contributing to their underrepresentation in the current data, as psychologists in private practice also constituted a small subgroup of the sample. The only broad personality and psychopathology test that ranks highly on the list is the MMPI (Butcher et al., 2011). It is also surprising to see the continued use of the SCL-90 (Derogatis, 1971), despite its well-known, serious psychometric limitations (Groth-Marnat & Wright, 2016). Strictly speaking, the SCID, MINI (Sheehan et al., 2009), MADRS (Montgomery & Åsberg, 1979), AUDIT (Babor et al., 1992) and DUDIT (Berman et al., 2007) can be described as checklists rather than psychometric tests, but methods like these nonetheless provide structure to assessments.

Earlier data from the United States show that self-report methods are by far the most popular type used by psychologists (Archer et al., 2006; Norcross & Karpiak, 2012), followed by performance-based methods. All practitioners involved in emotional injury cases who responded to a survey (Boccaccini & Brodsky, 1999) reported using the MMPI-2, with 54% using the Wechsler tests, 50% using the MCMI-III (Millon Clinical Multiaxial Inventory-III; Millon, 1994) and 41% using the Rorschach Inkblot Method (Rorschach, 1921). Data from as far back as 1985 indicate a decline in the use of performance-based methods, such as the Rorschach and the Wechsler tests (Piotrowski, 2015; Piotrowski et al., 1985). This trend corresponds with the increased research on and use of self-report tests in recent decades (Baumeister et al., 2007). Nonetheless, there are also indications that the use of tests is stable. Wright et al. (2017) demonstrated that self-report tests dominate among practitioners, alongside ability tests. In their survey, the Rorschach Inkblot Method and other performance-based (‘projective’) methods ranked eighth and ninth in terms of usage.

More generally, variations in test usage are also evident across different parts of the world. Some methods have a relatively strong foothold in certain countries but not in others. We know first-hand that the Rorschach Inkblot Method has very limited use in, for example, Great Britain and Germany, whereas it is widely employed in Finland and the United States. Another example is the Wartegg Drawing Completion Test, which is frequently used by psychologists in Sweden and Finland but is practically unknown in neighbouring Norway (Soilevuo Grønnerød & Grønnerød, 2012). In a small country like Norway, test education and usage are more dependent on specific professionals who actively teach and promote the use of particular tests. This situation disproportionately influences which tests are used compared to larger countries. That said, studies also indicate that most tests in use are common across countries, at least in the Western sphere (Egeland et al., 2016; Muñiz et al., 2001).

The main limitation of our study is the response rate of about 20%. Surveys of this nature have generally concluded that European psychologists have a positive attitude towards testing. However, generalisations must be made cautiously, as the response rates are generally low (15–20% across most participating countries). Even if the dispersion across work fields and specialties is representative of the psychologist population, the potential for self-selection is apparent. We must therefore assume that those most invested in tests and assessments are also those most likely to respond to a questionnaire of this kind. We can at best presume that the results are representative of assessment-focused psychologists, and less so for the psychologist population at large. Another limitation is that respondents were asked to name the three most-used tests in their practice. A more complete list of tests would have been useful, as it would have revealed differences in test diversity among practitioners and provided a more comprehensive overview of the most commonly used tests.

Conclusions

In the Norwegian segment of the EFPA’s 2020 decennial survey of psychologists’ attitudes towards tests and test use, test quality and regulation are the primary concerns expressed by respondents. Despite a relatively low response rate of 20%, study results highlight the need for more accessible information on test quality and for increased regulation based on user competence.