Fluency - Example Calculations

Overview

Below is a set of examples to show how a customer can use our service to calculate metrics like the WCPM or Accuracy.

What we provide

The follow information is provided in the results JSON object. Using this information we can enable customers to make WCPM and other calculations with ease.

The following calculations can be used to assess how well a child reads a particular passage. Using the different types, you can more accurately get how a child preformed and avoid situations where there is a lot of deletions at the end of a audio file and the child is punished for it.

Sample Equations

The following equations are using sample figures from the table below. These figures are example values that are typically returned by the Fluency endpoint .

Note:

The Formula use a mixture of data returned by the engine. The variables in the Name column are not returned by the endpoint. As mistakes and accuracy are defined by the customer, the below is a guide on how they can be calculated depending on your use case.

Calculating Accuracy

There are a number of ways to calculate Accuracy. This will be determined by your learning goals and activity type. Two sample options are listed below:

Option #1

correct_count % passage_read.

This ignores any insertion or repetition the child may has said while reading, this is only concerned how accurate the child said only the prompt text.

Option #2

correct_count % total_words.

This includes any insertions the child may has said while reading the prompt, which would lead to a more accurate score of how well the child read the prompt.

Calculating Read Percentage

Similar to Accuracy there are a number of different ways to calculate this. Two sample options are listed below

Option #1

passage_read % word_count.

This ignores any insertion or repetition the child may has said while reading, it also accounts for deletions so any words the child skipped, this gives an indication of how far the child got in the passage and is not an indication of how much of the passage the child actually read.

Option #2

words_said % word_count.

This does not include deletions but includes everything the child said, which would accurately tell how much of the actual passage the child said.

Calculating WCPM (Words Correct Per Minute)

There are a number of different methods for calculating WCPM below we have outlined one possible way. The response returned from the Fluency endpoint contains all the data points required to calculate a Words Correct Per Minute metric. For example:

Additional Features

The following features provide additional datapoints which can enhance and improve the results above.

Last Word Type

With the last word feature, we get some additional information which can be used to improve the results. This will mainly effect the WCPM as instead of using the audio duration for the timestamp, we can now depending on which last word type is used, use a timestamp of when the child actually stopped speaking.
Read more: Fluency - Last Word Feature

JSON

The Last Word JSON below is added to the root JSON returned by our API. With this we can get a more accurate timestamp of when the child either stopped reading the passage with READ or stopped talking with HEARD.

"last_word": {
	"end_timestamp": 5.135,
	"text_score_index": 2,
	"type": "WHOLE"
}

This feature allows you to choose between 3 options which will change the returned data counts.

Example Results

The following tables show results directly from the engine, for the three different types.

In this example the child has stopped reading from the passage after 31.14 seconds, they then speak a few more words which are flagged as insertions before stopping, meaning the last 24 seconds of audio is silence.

You will notice the amount of deletions for whole is 151, the majority of these occurring after the child has finished speaking, this leads to a poor Accuracy and WCPM.

Alternatively for read where we stop counting results after the last correct word, the number of deletions drops to 8, this indicates that the child skipped some words while reading. Because we do not count all the deletions at the end as we did in whole, the WCPM and Accuracy are now much higher and more accurate to what actually occurred.

In contrast to read, the heard option considers any insertions or substitutions detected after the last correct word, but similar to read does not consider deletions after the child stops speaking. This reduces the Accuracy somewhat compared to read as there were extra words spoken, but the Accuracy is still much improved in comparison to whole.

Please consult this Word Breakdown Table to see how the information is stored in the Word objects.

Self Corrections

With self corrections the customer will have to decide if the word was a self correction or not. This can be done by going through the results and checking each word that was marked as a self correction and comparing the time_since_previous with the last word to ensure that it is within your time frame for what is a self correction.
Read more: Fluency - Self Corrections Feature

JSON

"sub_types": {
	"self_correction": {
		"reparandums": [1]
	}
}

Pseudo code to calculate the number of self corrections.

for item in text_scores:
  
  if item['sub_type']['self_correction']:
    index_of_reparandum = item['sub_type']['self_correction'][0]
    
    time_since_previous = text_score[index_of_reparandum]['transcription_details']['time_since_previous']
    
    if time_since_previous < SELF_CORRECTION_THRESHOLD:
      ### We have a self correction!
      self_correction_count += 1

Calculations

With this information, users can now use it to discard reparandums from mistakes if they wish to not punish students for attempts. We can now get a true_insertion_count which will take away the number of self corrections or reparandums if you wish from the insertion_count.

true_insertion_count = insertion_count - self_correction_count

Example Results

With this change, anything which relies on insertion_count will have its score effected and will lead to improved accuracy if the self corrections are omitted.