Materials testing

Scoring the materials for difficulty of words and sentences

What readability formula did we use?

Overall, how did the materials score?

What are the individual scores?

Things to know about interpreting readability scores

What readability formula did we use?

• Like readability formulas in general, the Fry method assumes that longer words tend to be harder words, and longer sentences tend to be harder sentences. It estimates the difficulty of material based on counting the number of syllables and sentences in samples of text from the document.

   

• Since difficulty of words and sentences can vary in different parts of a document, scoring is based on drawing three 100-word samples and calculating the average score across these samples. For most documents in the resource bank, we were able to draw three samples. For the shorter materials, the Fry scoring is based on only two 100-word samples.

   

• Scoring was done only for materials written in English.

Overall, how did the materials score?

The materials scored as “easy” to “average” in difficulty of words and sentences (the Fry readability scores ranged from 4^th to 7^th grade depending on the material).

What are the individual scores?

Below we report scores separately for each piece of material. Since grade level scores are not nearly as precise as they sound, we report these individual scores in the following way:

• If the Fry grade level score falls in the range of 4th to 6th grade, we interpret this as meaning that the material uses “easy” words and sentences.

   

• If the Fry grade level score falls in the range of 7th to 8th grade, we interpret this as meaning that the material uses words and sentences that are of “average” difficulty.

“Difficult” words. Readability formulas assume that longer words are harder words. Specifically, words with three or more syllables are typically considered to be “difficult” words. What really matters, of course, is not the length of the words but whether the intended readers will know and understand the words that are used. So to help you judge the difficulty of the vocabulary in our resource bank materials, we show you the Fry score for each piece of material together with a list of words with three or more syllables that were in the samples of text that we scored.

Asthma Action Plan:

• Difficulty of words and sentences: Easy (4^th to 6^th grade).

   

• Words with three or more syllables in text sample that was scored: “medicine” (appeared twice),“normally,” “physically,” “tobacco.” 

Patient Questionnaire

• Difficulty of words and sentences:Easy (4^th to 6^th grade).

   

• Words with three or more syllables in text sample that was scored: “appointment,” “emergency,” “everyday,” “hospital,” “inhalers,” “medicine” (appeared twice),“medicines,” “overall,” “overnight,” “pharmacist,” “usually” (appeared twice). 

What is asthma and what can you do about it?

• Difficulty of words and sentences:Easy (4^th to 6^th grade).

   

• Words with three or more syllables in text sample that was scored:“irritated,” “medicine” (appeared four times),“regular,” “serious,” “whenever.” 

Why do you need two different types of asthma medicine?

• Difficulty of words and sentences:Average (7^th to 8^th grade).

   

• Words with three or more syllables in text sample that was scored:   “easier,” “irritated,” “medicine” (appeared 12 times),“relaxing,” “usually.” 

What to do when you have an asthma attack

• Difficulty of words and sentences:Average (7^th to 8^th grade).

   

• Words with three or more syllables in text sample that was scored: “easier,” “emergencies,” “emergency” (appeared three times),“medicine” (appeared four times),“medical” (appeared twice),“relaxing,” “serious,” “whenever,” “911” (appeared twice). 

What things cause asthma attacks for you? (asthma triggers)

• Difficulty of words and sentences:Average (7^th to 8^th grade).

   

• Words with three or more syllables in text sample that was scored:  “actually,” “allergies,” “animals,” “activity,” “bronchitis,” “detective,” “different,” “furniture,” “gardening,” “medicine” (appeared twice),“physical,” “tobacco,” “whenever.” 

• Difficulty of words and sentences:Average (7^th to 8^th grade).

   

• Words with three or more syllables in text sample that was scored: “activity,” “exercise,” “medicine” (appeared twice),“physical,” “possible” (appeared twice), “sensitive,” “usual.” 

Things to know about interpreting readability scores

• Readability scores are not measures of comprehension, even though they are sometimes interpreted that way.
  

   

• Readability scores reflect only one of many factors that affect ease of reading and usability of the materials. A readability formula provides a way to screen for difficulty of words and sentences, but it can’t take into account the life experience, literacy skills, and active search for meaning that people bring to the task of reading.
  
  
  • Like any readability formula, the Fry method focuses narrowly on what is easy to count at the level of individual words and sentences and ignores everything else.
    

     

  • Counts of syllables and sentences cannot tell you whether the layout is effective, or whether the writing is clear, cohesive, and well organized. These counts cannot tell you whether readers find the information appealing, easy to understand, and easy to use.

   

• Direct feedback from the intended users is the ultimate test. Since readability formulas only measure the length of words and sentences, it is not appropriate to use the results as a summary indicator or final standard for judging suitability of materials. To assess how well the materials in the Oregon Asthma Resource Bank were working, we conducted two rounds of interviews with consumers to get their reactions to draft versions of the materials. We used the results from this testing to improve the materials.