New research at N.C. State has shed doubt on the accuracy of readability formulas used by teachers, schools and textbook companies.
In a paper titled, “Can Readability Formulas Be Used to Successfully Gauge Difficulty of Reading Materials?” researchers have determined that certain formulas, which are used as indicators for judging the difficulty of reading materials and assigning reading levels for age groups, may not accurately correspond with the difficulty of the text.
“It’s not really a question of whether they are effective; it’s a question of whether they are what’s referred to as valid,” said John Begeny, associate professor of psychology and lead author of the research paper.
According to Begeny, validity is a term commonly used when understanding any type of assessment, which can include IQ tests, the SAT and End-of-Grade examinations.
A valid assessment means there is factual evidence based on research showing that it truly measures what it is supposed to measure, Begeny said. According to Begeny, readability formulas are meant to measure the relative difficulty level of text, but some research studies, including the recently published paper Begeny worked on, suggests that many of these formulas do not have good evidence of validity.
According to Begeny, the readability formulas incorporate factors such as the percentage of high-frequency “easy” words, or words in a predetermined list defined as familiar to most students in a particular grade. The formulas also contain the percentage of “hard,” or unfamiliar words, the average number of words per sentence, the average number of syllables per word, the number of single-syllable words, or number of words with multiple syllables.
Different combinations of these variables and different numerical associations are used to create the formulas, Begeny said. Unfortunately, the new research indicates that many of the formulas have poor percentages of accuracy.
Though the accuracy of readability formulas have been called into question for the last 50 years, according to Begeny, N.C. State researchers emphasized the relationship between the grade level and difficulty level. In other words, they attempted to discover whether the students’ grade level as determined by the formula corresponded to the actual difficulty level of their text reading.
Begeny and his team assessed the actual difficulty level through oral-reading-fluency performance, which tested how well students could read aloud. The problem with the assumption that grade level and difficulty level match up is that in many cases students are not reading at the appropriate level. Students that should be reading more advanced text are reading less advanced text, and vice versa, Begeny said.
This disconnect hurts students’ performances and hinders them from learning at their optimal level, the researchers said. The study did determine potential improvements to the system, including a plan to evaluate students each time a new reading assignment was made to determine the appropriate difficulty of the text, Begeny said.
The experiment itself consisted of 360 students from elementary school that were evaluated using the oral reading fluency score for six separate articles. Then, 10 separate formulas were applied to the article scores to determine their reading level. The students were then provided with their “appropriate” reading level texts and then reevaluated with the reading-fluency scores. The results varied from 79 percent as the most accurate, to 17 percent as least accurate, with the majority of them scoring less than 50 percent.
