Recent Assessment Centre Research Articles of Interest
In this section, some of the latest research on Assessment Centres (ACs) is presented. The reference is provided along with two or three take away points for practitioners and a link to the abstract. The aim is to allow for a brief, practitioner-oriented summary of the latest AC research along with some stated limitations, if necessary. It is not intended as a comprehensive or theoretical account of any given research article.
Monahan, E. L., Hoffman, B. J., Lance, C. E., Jackson, D. J. R., & Foster, M. R. (2013). Now you see them, now you do not: The influence of indicator-factor ratio on support for assessment center dimensions. Personnel Psychology, 66, 1009-1047. doi: 10.1111/peps.12049
Summary points
-
If you increase the number of indicators that you use to measure dimensions, then it is possible that you will see a modest increase in the contribution of dimensions to the assessment.
-
Exercise variance plays a key role in ACs, regardless as to how they are scored
Limitations
-
Limited number of samples were used
-
Only modest dimension effects were found
Abstract
The inability of assessment center (AC) researchers to find admissible solutions for confirmatory factor analytic (CFA) models that include dimensions has led some to conclude that ACs do not measure dimensions at all. This study investigated whether increasing the indicator-factor ratio facilitates the achievement of convergent and admissible CFA solutions in 2 independent ACs. Results revealed that, when models specify multiple behavioral checklist items as manifest indicators of each latent dimension, all of the AC CFA models tested were identified and returned proper solutions. When armed with the ability to undertake a full set of model comparisons using model fit rather than solution convergence and admissibility as comparative criteria, we found clear evidence for modest dimension effects. These results suggest that the frequent failure to find dimensions in models of the internal structure of ACs is a methodological artifact and that one approach to increase the likelihood for reaching a proper solution is to increase the number of manifest indicators for each dimension factor. In addition, across exercise dimension ratings and the overall assessment rating were both strongly correlated with dimension and exercise factors, indicating that regardless of how an AC is scored, exercise variance will continue to play a key role in the scoring of ACs.
Hoffman, B. J., Kennedy, C., LoPilato, A., Monahan, E., & Lance, C. E. (in press). A review of the content, criterion-related, and construct-related validity of assessment center exercises. Journal of Applied Psychology. doi: 10.1037/a0038707
Summary points
-
AC exercise scores, sans dimensions, correlate modestly with outcome performance, the implication being that you could just use stand-alone exercises as a scoring basis, but there will be a drop in validity typically expected from ACs.
-
AC exercise scores also correlate with externally measured psychological characteristics (e.g., general mental ability, personality), suggesting that there is a psychological basis for performance on them.
Limitations
-
This article is primarily focused on individual exercises, which are sometimes referred to as low fidelity simulations or performance tests (e.g., a single in-basket), in which modest criterion-related validity is commonplace.Thus, it could be argued that this is not actually about ACs per se, because ACs involve multiple exercises.
-
Although ostensibly about AC exercise-based scoring procedures, the article contains no data on task-based ACs, which represent a key type of exercise-based scoring in the AC literature.
Abstract
This study uses meta-analysis and a qualitative review of exercise descriptions to evaluate the content, criterion-related, construct, and incremental validity of 5 commonly used types of assessment center (AC) exercises. First, we present a meta-analysis of the relationship between 5 types of AC exercises with (a) the other exercise types, (b) the 5-factor model of personality, (c) general mental ability (GMA), and (d) relevant criterion variables. All 5 types of exercises were significantly related to criterion variables (ρ = .16–.19). The nomological network analyses suggested that the exercises tend to be modestly associated with GMA, Extraversion and, to a lesser extent, Openness to Experience but largely unrelated to Agreeableness, Conscientiousness, and Emotional Stability. Finally, despite sparse reporting in primary studies, a content analysis of exercise descriptions yielded some evidence of complexity, ambiguity, interpersonal interaction, and fidelity but not necessarily interdependence.
Speer, A. B., Christiansen, N. D., Goffin, R. D., & Goff, M. (2014). Situational bandwidth and the criterion-related validity of assessment center ratings: Is cross-exercise convergence always desirable? Journal of Applied Psychology, 99, 282-295. doi: 10.1037/a0035213
Summary points
-
Although the literature has, in the past, been trying to create consistency across AC exercises, this article suggests that aim might have been misdirected.
-
The researchers found that ACs that used different types of exercises tended to yield higher validities than ACs that employed similar exercise types.
-
The take-home point for practitioners is to make sure that your AC exercises are different from each other.
Limitations
-
Only three samples were included here and it would be encouraging to see more studies to help ensure replicability.
Abstract
This research examines the relationship between the construct and criterion-related validity of assessment centers (ACs) based on how convergence of dimension ratings across AC exercises affects their ability to predict managerial performance. According to traditional multi-trait–multi-method perspective, a high degree of convergence represents more reliable measurement and has the potential for better validity. In contrast, the concept of situational bandwidth suggests that behavior assessed under a dissimilar set of circumstances should result in a more comprehensive assessment of a candidate’s tendencies even though ratings are less likely to show high convergence. To test these opposing viewpoints, data from 3 operational ACs were obtained along with experts’ evaluations of exercise characteristics and supervisors’ ratings of candidates’ managerial performance. Across the 3 samples, AC ratings taken from exercises with dissimilar demands had higher estimates of criterion-related validity than ratings taken from similar exercises, even though the same dimension-different exercise correlations were substantially higher between similar exercises. Composites of ratings high in convergence did not emerge as better predictors of managerial performance, and validity particularly suffered. when derived from ratings that converged as a result of exercises with similar demands. Implications for AC design are discussed.
Lievens, F., Schollaert, E., & Keen, G. (2014). The interplay of elicitation and evaluation of trait-expressive behavior: Evidence in assessment center exercises. Journal of Applied Psychology. doi: 10.1037/apl0000004
Summary points
-
Looks at integrating theoretical and rating models in order to optimise assessments.
-
AC rating evaluation schemes should be developed with forethought given to how situational characteristics will elicit trait-related behaviour.
-
Practitioners need to consider how their exercises will elicit certain behaviours and rating approaches in ACs need to be developed with these ideas in mind.
Limitations
-
Some of the results around cognitively-oriented dimensions did not manifest according to expectations.
Abstract
In assessment centers (ACs), research on eliciting candidate behavior and evaluating candidate behavior have largely followed independent paths. This study integrates trait activation and trait rating models to posit hypotheses about the effects of behavior elicitation via situational cues on key assessor observation and rating variables. To test the hypotheses, a series of experimental and field studies are conducted. Only when trait-expressive behavior activation and evaluation models work in conjunction, increases in observability are coupled with increases in the inter-rater reliability, convergent validity, discriminant validity, and accuracy of AC ratings. Implications of these findings for AC theory and practice are formulated.
Meriac, J. P., Hoffman, B. J., & Woehr, D. J. (2014). A conceptual and empirical review of the structure of assessment center dimensions. Journal of Management, 40. doi: 10.1177/0149206314522299
Summary points
-
Presents a theoretical framework for AC dimensions
-
Uses quantitative approaches to summarise three major dimensions across AC studies, including administrative skills, relational skills, and drive
-
AC practitioners could use these broad summaries to categorise their own dimension frameworks
Limitations
-
It is known that exercises have an influence in AC ratings, yet they are not a focus in this study.
Abstract
Although the design, scoring, and interpretation of assessment centers (ACs) commonly focuses on job-relevant dimensions, over three decades of past studies have questioned the evidentiary basis underlying dimension-based interpretations of ACs. This review combines multiple approaches to examine the structure of AC dimensions. First, we consulted the AC, job performance, leadership, and personality literatures to articulate competing models of the dimensions underlying AC ratings. Next, meta-analytic confirmatory factor analysis (CFA) was used to compare the fit of these models to existing AC data. The results supported a model including administrative skills, relational skills, and drive. Third, socioanalytic theory was used as a basis to examine the nomological network of these three broad factors, specifically their relationships with general mental ability and the five factor model of personality. The analyses supported the nomological network of drive and administrative skills but less so for relational skills. These findings are discussed with regard to the construct-related validity of AC dimensions, the fidelity of ACs to the broader criterion domain, and the value of applying generalizable models to the analysis of AC ratings.