PLS-5 - A Validity Study

By Kristin Smith posted 10-28-2014 04:35 PM

  

PLS-5 Validity Study – A Comparison of Test Scores

Authored by: Kristin K Smith, MA CCC-SLP

Child Find, Douglas County School District, Colorado

Following publication of the newly revised Preschool Language Scale – Fifth Edition (PLS-5,) Douglas County School District in Colorado purchased this assessment tool for the Early Childhood Education department.  Use of the PLS-5 to assess children birth through kindergarten age began in in October of 2011.  Child Find staff began questioning PLS-5 results almost immediately because test scores generated often did not appear to reflect language skills demonstrated by the test subjects.  In one instance, a new 3 year old with less than 50 words in his vocabulary received an Expressive Communication score that fell at the 18th percentile, - a score within the “normal” range.  In another instance, a child of 3 years, 8 months who exhibited social/communicative and language characteristics consistent with high functioning Autism received an Expressive Communication subtest score at the 14th percentile even though he was unable to answer basic questions and engage in conversation.  In each of these cases, the administering SLP had expected resulting percentile scores to fall below the 7th percentile.   

Following several assessments where Child Find SLP’s questioned PLS-5 results, this author initiated a dialogue through the online ASHA Discussions forum to determine whether SLPs on a national level were having similar experiences.  With one exception, all respondents indicted that they too were generating scores that seemed inflated with relation to a child’s demonstrated skills.  Concerns largely centered around PLS-5 Expressive Communication scores, though Auditory Comprehension scores at some age ranges were also identified as suspect. Kristin Smith, MA CCC-SLP, then initiated a study to compare PLS-5 results with other age appropriate language measures in order to document possible inter-test discrepancies.  

About the study

Two commonly used assessment tools were selected for the comparison study based on test contents and/or customary usage within the school district for the identification of language difficulties in children.  The Mullen Scales of Early Learning Receptive Language and Expressive Language subtests (Mullen) were used for comparison with children between the ages of 6 months and 3 years.  The Clinical Evaluation of Language Fundamentals Preschool – Second Edition (CELF-P2) Receptive Language and Expressive Language composites were selected for comparison with preschool children over the age of three.  Four speech/language pathologists from the DCSD Early Childhood department* contributed testing results for the comparison study.  SLPs administered the PLS-5, as well as the other age appropriate test, to the same child within a two-week period.  Percentiles were uses as the point of comparison because all three tests generated percentile scores, and the Colorado Department of Education had loosely defined a language handicapping condition by a 7% or lower score on a standardized assessment instrument. Twenty-seven children participated in the study, 4 of which were in a preschool program having been previously identified as language handicapped, and the remainder of whom had been referred to Child Find for a complete developmental evaluation due to parental concerns regarding their overall development.  Of the 27 children participating in the comparison study, 17 were administered both the PLS-5 and Mullen while the other 10 were administered the PLS-5 and CELF-P2.  One child over the age of 36 months was administered the Mullen Scales rather than the CELF-P2 due to the magnitude of the child’s perceived delays and the desire for more meaningful clinical information to be generated.  Children participating in the study were selected at random based solely on the fact that they had been scheduled for a complete developmental assessment through Child Find or that their annual IEP review occurred within the window of time of the comparison study.  Parents were asked if their children could participate, and written permissions were obtained prior to testing.  The research study was also reviewed and approved of by the DCSD System Performance Office.

 

Results

Receptive Language Test Scores  

Children under 12 months - 2 subjects:  tests generated comparable scores.  No children assessed in this age range were identified for language based support services.

Children 12 months to 36 months - 14 subjects:  10 or 71% received higher scores on the PLS-5 than on the Mullen Scales.    6 children qualified for educational support services, but only 4 were identified through the PLS-5

Children over 36 months given the CELF-P2 - 10 subjects:  8 or 80% received higher scores on the PLS-5 than on the CELF-P2.  3 qualified for educational support services, but only 2 were identified through the PLS-5.

 

For 18 of the 27 subjects (66%,) the PLS-5 scored higher than the comparison measure.  In 26% of cases (7 out of the 27), PLS-5 Auditory Comprehension scores were more than 15 percentile points higher than the other test.  PLS-5 test scores spanned from 8 percentile points lower to as much as 36 percentile points higher than the comparison test for an average of 8 percentile points higher.  Nine children out of the sample were identified through the assessment process as having a significant receptive language delay and were offered Special Education support services through an IEP or IFSP.  Five of those demonstrated characteristics consistent with an Autism spectrum disorder.  Of the remaining 4, the PLS-5 only identified 1 child as having an educationally significant receptive language delay.  The PLS-5 did not identify 75% of children with significant expressive language delays unless they were demonstrating disordered language characteristics consistent with Autism.

                                    Expressive Language Test Scores

Children under 12 months - 2 subjects:  tests generated comparable results with one subject while the Mullen scored significantly higher (27 percentile points) than the PLS-5 for the other subject.   One child assessed in this age range qualified for language based support services and was identified by both measures.

Children 12 months to 36 months -14 subjects:  13 or 93% received higher scores on the PLS-5 than on the Mullen Scales.  10 qualified for educational support services, but only 3 were identified through the PLS-5.

Children over  36 months given the CELF-P2 - 10 subjects:  7 or 70% received higher scores on the PLS-5 than on the CELF-P2.  4 qualified for educational support services, but only 1 was identified through the PLS-5.

 

For 21 of the 27 subjects (78%,) the PLS-5 scored higher than the comparison measure.  In 33% of the cases (9 out of 27,) PLS-5 Auditory Comprehension scores were more than 15 percentile points higher than the other test.  PLS-5 test scores spanned from 22 percentile points lower to 36 percentile points higher than the comparison test for an average of 9 percentile points higher.  Sixteen of the 27 children assessed in this sample (59%) were ultimately identified as having an expressive language handicap and offered Special Education support services through an IEP or IFSP.  The PLS-5 only identified 6 of those children (22%) as having significant expressive language issues.  Of particular concern was the 12-36 month age range.  Of the 10 children in this range who received special education support as a result of the assessment, only 3 were identified by the PLS-5.  Only one child in the 12-24 month age range was identified by the PLS-5 as having even “below average” language skills.  No children in the 12-24 month age range were actually identified by the PLS-5, though 4 of the 6 children in this age range were made eligible for expressive language support by the Child Find team and were identified by the other language measure.  Five of the 16 total children who ultimately received educational support services demonstrated characteristics consistent with an Autism Spectrum Disorder.  The PLS-5 identified each of those 5 children with a less than 7% score, but it only identified 1 child out of the other 11 who received support services as having significant expressive language difficulties.  In other words, unless the child was likely on the Autism Spectrum, the PLS-5 missed 91% of children with handicapping expressive language deficits in this sample. 

Discussion

Scores generated by the newly revised PLS-5 are concerning for a number of reasons but primarily because the scores often do not appear to reflect the actual language skills, or lack there of, demonstrated by individual children.  Problems with the standardization of the PLS-5 may explain decreased PLS-5 sensitivity.**

The standardization sample of the PLS-5 included delayed and disordered children. According to Pena, Spalding, & Plante, (2006), when the purpose of a test is to identify children with language impairment, inclusion of children with language impairment in the normative sample can reduce the accuracy of identification. Inclusion of these children causes a leftward shift in the mean, altering the mean and reducing the test's ability to determine true positives (sensitivity), thus reducing reliability. 

According to the NYCDOH definition, "Negative predictive value is the probability that a person with a negative (normal) test result is truly free of disease," The PLS-5 provides insufficient evidence based on discriminant accuracy, as "Negative Predictive Powers" fall as low as 60%. (Values obtained from Pearson Assessments, see references section.) According to Plante and Vance (1994) the standard for discriminant accuracy is 80% or greater, where 80-89% is "fair" and 90% or above is considered "good" discriminant accuracy. As a result, it is not possible to determine the degree to which the PLS-5 is accurately able to identify and discriminate among children with language disorder and those without language disorder. For this reason, the validity of this test is compromised.

The PLS-5 standardization data uses information collected from the 2008 update of U.S. census figures. "While a normative sample based on the U.S. Census gives us an idea of the average performance of ethnic groups, it will not identify whether a child's communication skills deviate substantially from the norms of the child's speech community," (Crowley, Agosto & Castle, 2006).

This study would suggest that there are significant discrepancies between PLS-5 results and those generated by other established language measures.  Per this study, as many as 75 -91% of preschool aged children with an educationally significant language handicap could go unrecognized if PLS-5 scores are taken on face value.  The ramifications of using the PLS-5 to determine eligibility for school based support services could be tremendous depending upon the experience of a particular SLP and the rigidity of administrative guidelines with respect to special education qualifications. It is this author’s opinion that the PLS-5 should be recalled to address problems with standardization and test validity. 

 

* Kristin K Smith, MA CCC-SLP * Lindsay Cohen, MS CF-SLP * Jessica Gates, MS CCC-SLP * Heather Hickman, MA CCC-SLP **Sources compiled by: Tatyana Elleseff MA CCC-SLP

* Kristin K Smith, MA CCC-SLP - Early Childhood Department – Douglas County School District, Colorado

 

 * Lindsay Cohen, MS CF-SLP - Early Childhood Department – Douglas County School District, Colorado

 

 * Jessica Gates, MS CCC-SLP - Early Childhood Department – Douglas County School District, Colorado

 

 * Heather Hickman, MA CCC-SLP - Early Childhood Department – Douglas County School District, Colorado

 

**Sources compiled by: Tatyana Elleseff MA CCC-SLP - Lead Speech Language Pathologist, University of Medicine and Dentistry of New Jersey, Piscataway, NJ

6 comments
28801 views

Comments

04-05-2016 03:52 PM

I'm grateful it's not just me feeling this way. Any follow up with the publisher to address the issue? What is everyone else using in place of it??

11-19-2015 12:21 PM

My job is evaluating ECSE children (age 3-5) for five rural Oregon counties. I have been using the PLS-5 on a daily basis. Using it, I seem to encounter more situations than I might expect where the expressive language scores are higher than the receptive language scores, and on several occasions there has been as much as a 15-30 point difference. While that is not unheard of, it isn't something I have tended to see often over the years. I have also had several instances in which the expressive language standard scores for children from age 2-6 to 3-6 have been in the high 70s to low 80s when those children use far fewer words than would be expected. This is frustrating, as our evaluation team sees up to 16 kids per week, and I'm the guy who decides whether or not they are CD eligible. I have to go by the test scores, but my judgment, based on years of experience, tells me not to. If this is a widespread issue, I hope the publishers can do something to remedy the situation. Our ESD has a somewhat limited budget for buying new test kits, but maybe we need to replace the PLS-5 and use something else for our evals. Sorry, Mr. Bear!

11-18-2015 04:26 PM

I have been frustrated with this test since it was released. Like Dana B., I am grateful for the research and would love to know if Pearson is going to address this issue. Also, I'd love to know of any recommended alternatives that can be used for children ages birth-to-3 years.

08-28-2015 07:35 AM

I have been expressing the same concern for the past two years!! I have had a multitude of 2.6-3 year-olds that are either non-verbal or at the one word level using only nouns to label and the standard scores fall within the "average" range while I know that there is nothing "average" about it. For those children, I will use a detailed and descriptive narrative comparing average skills to the students skills and have been able to qualify for services. It's a shame. This test has been highly regarded and has been an industry norm for so long we all use it.

07-23-2015 05:38 PM

I definitely find that this is consistent with my experience. Which is unfortunate because the PLS-5 is very expensive and was a waste of money. A non-verbal 2 year old will score in the normal range for expressive language. I have frequently had kids with ASD score very high expressively, but significantly lower receptive my, which makes no sense at all. But their delayed echolalia and memorized phrases will get them credit for items it really shouldn't. This makes it hard when trying to convince insurance companies that a child really is severely impaired and really does need treatment.

06-02-2015 12:32 PM

Thank you for pursuing this research and sharing your results. Has there been any follow up by the company regarding this issue with the PLS-5? Any other info? Are you still using this tool? If not, what are you using in it's place?