Approaching the end of the school year means focusing on graduation rates, dropout rates, and other data suggesting trends for students.
Opportunities for considering statistics abound; but one must continue to examine the way that these statistics are actually used, by asking the right questions about the data.
For example: As teachers finish state testing regimens and head into final exams, it may be useful to see data related to average pay for teachers. Is it going up?
These figures from 1969-2014 suggest that it is:
Wow. Average teacher salaries have gone up by more than 700 percent since 1969, you might say, based on the numbers in this table.
What one sometimes forgets, though, is that inflation and other factors contribute to a need to calibrate salaries to current dollars, or what they represent in today’s terms. Here are the numbers, as calculated in 2012-’13 dollars (the latest year that the National Center for Education Statistics has reported).
This adjusted data tells a different story, with average salaries not changing so radically over the years. But notice, too, that in these charts, the intervals are not consistent; the first four are 10-year intervals, while the last three are 2-year and 1-year intervals. “What are the intervals in the data collection?” is another question that must be raised. Comparing the jump from 1969-70 to 1979-80 is not the same thing as evaluating the difference between 2009-10 and 2011-12.
A control chart that reflects this data may give a greater sense of information from the data, but since the increments are irregular, the chart is as specious as the original data:
The comparison points to the necessity for being careful with statistics. As Mark Twain noted, “There are lies, damn lies, and statistics.” Statistical methods when applied carelessly, distort what can be seen as reality. Sometimes this happens accidentally, and sometimes statistics are simply made up to support a pre-held conviction.
Here are some things to look for in order to verify that statistics are actually accurate.
- If dollar figures are given over time (as in the example above) be sure that the adjustment for CPI rates or other economic factors have been factored in.
- Check the vertical axis of any chart, to be sure that the intervals are consistent and reasonable. Statistics can be distorted by altering the consistency in the y axis. If, for example, some intervals are 5 and others are 10, be suspicious.
- Verify the source of the data in the chart. It may be anecdotal, as in the case of charting the number of items that women have in their purses. If it comes from a reliable source, you may have more confidence in its accuracy.
- Verify the operational definition for what is being measured (e.g., is a string bag considered a purse? What about a backpack? Or a wrist wallet?).
- Look at the sample and the integrity of sampling methods that have been utilized. If a sampler surveys a group of his friends, for example, be suspicious. The sampling should be truly random, and large enough to suggest trends.
- Use a control chart to assess the data for stability, using standard out-of-control tests that include patterns such as too many points around the mean.
While applying statistics in a way that promises accuracy demands more than these few rules, applying these standards will give a clue about the data that will help evaluate its integrity.