PLS-5 Validity Study – A Comparison of Test Scores
Authored by: Kristin K Smith, MA CCC-SLP
Child Find, Douglas County School District, Colorado
Following publication of the newly revised Preschool Language Scale – Fifth Edition (PLS-5,) Douglas County School District in Colorado purchased this assessment tool for the Early Childhood Education department. Use of the PLS-5 to assess children birth through kindergarten age began in in October of 2011. Child Find staff began questioning PLS-5 results almost immediately because test scores generated often did not appear to reflect language skills demonstrated by the test subjects. In one instance, a new 3 year old with less than 50 words in his vocabulary received an Expressive Communication score that fell at the 18th percentile, - a score within the “normal” range. In another instance, a child of 3 years, 8 months who exhibited social/communicative and language characteristics consistent with high functioning Autism received an Expressive Communication subtest score at the 14th percentile even though he was unable to answer basic questions and engage in conversation. In each of these cases, the administering SLP had expected resulting percentile scores to fall below the 7th percentile.
Following several assessments where Child Find SLP’s questioned PLS-5 results, this author initiated a dialogue through the online ASHA Discussions forum to determine whether SLPs on a national level were having similar experiences. With one exception, all respondents indicted that they too were generating scores that seemed inflated with relation to a child’s demonstrated skills. Concerns largely centered around PLS-5 Expressive Communication scores, though Auditory Comprehension scores at some age ranges were also identified as suspect. Kristin Smith, MA CCC-SLP, then initiated a study to compare PLS-5 results with other age appropriate language measures in order to document possible inter-test discrepancies.
About the study
Two commonly used assessment tools were selected for the comparison study based on test contents and/or customary usage within the school district for the identification of language difficulties in children. The Mullen Scales of Early Learning Receptive Language and Expressive Language subtests (Mullen) were used for comparison with children between the ages of 6 months and 3 years. The Clinical Evaluation of Language Fundamentals Preschool – Second Edition (CELF-P2) Receptive Language and Expressive Language composites were selected for comparison with preschool children over the age of three. Four speech/language pathologists from the DCSD Early Childhood department* contributed testing results for the comparison study. SLPs administered the PLS-5, as well as the other age appropriate test, to the same child within a two-week period. Percentiles were uses as the point of comparison because all three tests generated percentile scores, and the Colorado Department of Education had loosely defined a language handicapping condition by a 7% or lower score on a standardized assessment instrument. Twenty-seven children participated in the study, 4 of which were in a preschool program having been previously identified as language handicapped, and the remainder of whom had been referred to Child Find for a complete developmental evaluation due to parental concerns regarding their overall development. Of the 27 children participating in the comparison study, 17 were administered both the PLS-5 and Mullen while the other 10 were administered the PLS-5 and CELF-P2. One child over the age of 36 months was administered the Mullen Scales rather than the CELF-P2 due to the magnitude of the child’s perceived delays and the desire for more meaningful clinical information to be generated. Children participating in the study were selected at random based solely on the fact that they had been scheduled for a complete developmental assessment through Child Find or that their annual IEP review occurred within the window of time of the comparison study. Parents were asked if their children could participate, and written permissions were obtained prior to testing. The research study was also reviewed and approved of by the DCSD System Performance Office.
Results
Receptive Language Test Scores
Children under 12 months - 2 subjects: tests generated comparable scores. No children assessed in this age range were identified for language based support services.
Children 12 months to 36 months - 14 subjects: 10 or 71% received higher scores on the PLS-5 than on the Mullen Scales. 6 children qualified for educational support services, but only 4 were identified through the PLS-5
Children over 36 months given the CELF-P2 - 10 subjects: 8 or 80% received higher scores on the PLS-5 than on the CELF-P2. 3 qualified for educational support services, but only 2 were identified through the PLS-5.
For 18 of the 27 subjects (66%,) the PLS-5 scored higher than the comparison measure. In 26% of cases (7 out of the 27), PLS-5 Auditory Comprehension scores were more than 15 percentile points higher than the other test. PLS-5 test scores spanned from 8 percentile points lower to as much as 36 percentile points higher than the comparison test for an average of 8 percentile points higher. Nine children out of the sample were identified through the assessment process as having a significant receptive language delay and were offered Special Education support services through an IEP or IFSP. Five of those demonstrated characteristics consistent with an Autism spectrum disorder. Of the remaining 4, the PLS-5 only identified 1 child as having an educationally significant receptive language delay. The PLS-5 did not identify 75% of children with significant expressive language delays unless they were demonstrating disordered language characteristics consistent with Autism.
Expressive Language Test Scores
Children under 12 months - 2 subjects: tests generated comparable results with one subject while the Mullen scored significantly higher (27 percentile points) than the PLS-5 for the other subject. One child assessed in this age range qualified for language based support services and was identified by both measures.
Children 12 months to 36 months -14 subjects: 13 or 93% received higher scores on the PLS-5 than on the Mullen Scales. 10 qualified for educational support services, but only 3 were identified through the PLS-5.
Children over 36 months given the CELF-P2 - 10 subjects: 7 or 70% received higher scores on the PLS-5 than on the CELF-P2. 4 qualified for educational support services, but only 1 was identified through the PLS-5.
For 21 of the 27 subjects (78%,) the PLS-5 scored higher than the comparison measure. In 33% of the cases (9 out of 27,) PLS-5 Auditory Comprehension scores were more than 15 percentile points higher than the other test. PLS-5 test scores spanned from 22 percentile points lower to 36 percentile points higher than the comparison test for an average of 9 percentile points higher. Sixteen of the 27 children assessed in this sample (59%) were ultimately identified as having an expressive language handicap and offered Special Education support services through an IEP or IFSP. The PLS-5 only identified 6 of those children (22%) as having significant expressive language issues. Of particular concern was the 12-36 month age range. Of the 10 children in this range who received special education support as a result of the assessment, only 3 were identified by the PLS-5. Only one child in the 12-24 month age range was identified by the PLS-5 as having even “below average” language skills. No children in the 12-24 month age range were actually identified by the PLS-5, though 4 of the 6 children in this age range were made eligible for expressive language support by the Child Find team and were identified by the other language measure. Five of the 16 total children who ultimately received educational support services demonstrated characteristics consistent with an Autism Spectrum Disorder. The PLS-5 identified each of those 5 children with a less than 7% score, but it only identified 1 child out of the other 11 who received support services as having significant expressive language difficulties. In other words, unless the child was likely on the Autism Spectrum, the PLS-5 missed 91% of children with handicapping expressive language deficits in this sample.
Discussion
Scores generated by the newly revised PLS-5 are concerning for a number of reasons but primarily because the scores often do not appear to reflect the actual language skills, or lack there of, demonstrated by individual children. Problems with the standardization of the PLS-5 may explain decreased PLS-5 sensitivity.**
The standardization sample of the PLS-5 included delayed and disordered children. According to Pena, Spalding, & Plante, (2006), when the purpose of a test is to identify children with language impairment, inclusion of children with language impairment in the normative sample can reduce the accuracy of identification. Inclusion of these children causes a leftward shift in the mean, altering the mean and reducing the test's ability to determine true positives (sensitivity), thus reducing reliability.
According to the NYCDOH definition, "Negative predictive value is the probability that a person with a negative (normal) test result is truly free of disease," The PLS-5 provides insufficient evidence based on discriminant accuracy, as "Negative Predictive Powers" fall as low as 60%. (Values obtained from Pearson Assessments, see references section.) According to Plante and Vance (1994) the standard for discriminant accuracy is 80% or greater, where 80-89% is "fair" and 90% or above is considered "good" discriminant accuracy. As a result, it is not possible to determine the degree to which the PLS-5 is accurately able to identify and discriminate among children with language disorder and those without language disorder. For this reason, the validity of this test is compromised.
The PLS-5 standardization data uses information collected from the 2008 update of U.S. census figures. "While a normative sample based on the U.S. Census gives us an idea of the average performance of ethnic groups, it will not identify whether a child's communication skills deviate substantially from the norms of the child's speech community," (Crowley, Agosto & Castle, 2006).
This study would suggest that there are significant discrepancies between PLS-5 results and those generated by other established language measures. Per this study, as many as 75 -91% of preschool aged children with an educationally significant language handicap could go unrecognized if PLS-5 scores are taken on face value. The ramifications of using the PLS-5 to determine eligibility for school based support services could be tremendous depending upon the experience of a particular SLP and the rigidity of administrative guidelines with respect to special education qualifications. It is this author’s opinion that the PLS-5 should be recalled to address problems with standardization and test validity.
* Kristin K Smith, MA CCC-SLP * Lindsay Cohen, MS CF-SLP * Jessica Gates, MS CCC-SLP * Heather Hickman, MA CCC-SLP **Sources compiled by: Tatyana Elleseff MA CCC-SLP
* Kristin K Smith, MA CCC-SLP - Early Childhood Department – Douglas County School District, Colorado
* Lindsay Cohen, MS CF-SLP - Early Childhood Department – Douglas County School District, Colorado
* Jessica Gates, MS CCC-SLP - Early Childhood Department – Douglas County School District, Colorado
* Heather Hickman, MA CCC-SLP - Early Childhood Department – Douglas County School District, Colorado
**Sources compiled by: Tatyana Elleseff MA CCC-SLP - Lead Speech Language Pathologist, University of Medicine and Dentistry of New Jersey, Piscataway, NJ