The formula -- in what is called a “value-added model” -- tries to determine a teacher’s effect on a student’s FCAT performance by predicting what the student should score in a given year, and then rating the teacher on whether the student hits or surpasses the mark.
For thousands of other Florida teachers who don’t even teach a subject assessed by the FCAT, value-added scores will be tied to the FCAT reading score of their entire school.
With its new report card, Florida is among 25 states that have turned to student scores on standardized exams to help evaluate teachers and set their pay. By 2014, it will become mandatory to do so under a new state law.
In Miami, the model will initially use results on the FCAT, which has gotten tougher, and will expand to include other tests that are being developed in every subject at every grade level.
Florida’s revamped teacher-evaluation system is all part of the education reform agenda pushed by the Obama administration, which is giving states $4.3 billion in its Race to the Top grant program to come up with new ways to grade teachers and tie student performance to paychecks.
In a way, it’s a race to find better ways to evaluate teachers. The new formula plays into a growing national trend of holding individual teachers accountable for student performance.
In Florida, the stakes are high. Top performing teachers can get permanent salary increases, while those with ratings near the bottom for two consecutive years can be let go.
In the past, teachers were traditionally evaluated by their principals or top administrators. The result was that most were rated the same: satisfactory. And state-issued school grades come mainly from school FCAT scores, which don’t recognize individual teacher’s impact, said state Rep. Erik Fresen, the Miami Republican who helped pass the controversial law known as Senate Bill 736.
“All we were looking at as a state and as districts was what the school did, not what the individual teacher did. This changes that paradigm completely,” he said.
In the new evaluations, half a teacher’s “grade” will be based on the new value-added formula, and half an evaluation that involves principal observations and professional development goals. Teachers who don’t instruct FCAT subjects will get grades based on the school’s FCAT reading performance in some counties.
In Manatee, no decision has been made about whether FCAT reading scores will be tied to the value-added model, said Pat Barber, president of the Manatee Education Association.
“We’re trying to come to a consensus within the committee as to what’s the fairest and easiest to explain,” Barber said referring to Manatee County’s value-added model. “Anything having to do with performance pay is yet to be negotiated.”
The new system faces many challenges and much criticism. No research has shown that the value-added approach to teacher evaluations improves student learning, but there is research to suggest that some models yield unreliable results.
The state teachers union is challenging the new law in court, arguing that it takes away teachers’ right to bargain for their pay and working conditions. The Manatee Education Association supports the lawsuit, said business agent Bruce Proud, who described the value-added model as having “lots of flaws.”
Still, Manatee is moving forward, forming a committee to figure out how the formula will be implemented.
“That’s a huge conversation that’s going to happen,” Superintendent Tim McGonegal said.
Even the biggest national supporters of value-added evaluations concede to caveats: Sufficient data exist for only about 20 percent of teachers nationwide to be given value-added scores. And questions abound about the accuracy and reliability of standardized tests like the FCAT.
Accordingly, even proponents urge states to recognize that whatever the promise of value-added models, they are still in their infancy.
“We don’t have evidence that this approach is going to improve teaching and learning,” Douglas Harris, an expert on value-added modeling at the University of Wisconsin-Madison and author of the 2011 book “Value Added Measures in Education: What Every Educator Needs to Know.”
Harris called Florida’s decision to use school-wide reading scores for individual teachers “backwards,” saying that it molds the education system to fit the teacher evaluation system instead of the other way around.
Officials with the state DOE say it will help teachers -- and in turn, their students -- improve.
“With value-added, we’re trying to give better information to teachers about their student performance so they can improve their instruction,” said Kathy Hebda, the state’s deputy chancellor for educator quality.
Crafting the formula
Using nearly $4 million from its Race to the Top Dollars, Florida contracted with the American Institutes for Research in Washington, D.C., to develop the complex formula.
“It’s not simply a matter about what they scored last year and did they improve,” said Juan Copa, director of research, evaluation and educator performance at the state’s Department of Education. “What you’re trying to do is isolate the impact of the teacher on the student’s learning.”
A committee of administrators, teachers, union leaders and parents from around the state met to decide which variables to include in the formula. Copa said they decided on a more complex model to gain more reliability and accuracy.
To try to isolate the teacher effect, the model weighs 10 factors, like student attendance and disability status.
Each factor has a different weight for each subject and grade level, explained Jon Cohen, executive vice president and program director for assessment at the American Institutes for Research.
The strongest factor in predicting a student’s future score -- explaining about 60 percent of the variance in achievement -- is that student’s prior test score. “Not because anyone said that’s the way it should be but that’s the way the world works,” Cohen said. “If I know how you did on the math test last year and you did well, we’d expect you to do well this year.”
Poverty left out
The formula does not take into account a student’s race, gender or socioeconomic status, despite evidence that such characteristics are linked to student achievement.
Florida legislators opted to leave poverty out of the equation.
“They said every child should be able to learn, regardless of their poverty level,” said Gisela Field, Miami-Dade’s administrative director for assessment, research and data analysis. Otherwise, it would have been good to include, Field said. “We all know that poverty is a big indicator.”
Cohen said that if poverty influences a student’s score one year, it likely impacted the previous year’s score, too. “It’s pretty much embedded in the child’s scores when you control for prior achievement.”
Still, Proud felt like not accounting for poverty neglected the fact that exposure to materials outside of the classroom -- something poor families may have difficulty providing -- will impact scores. “It’s kind of ignoring a real issue in the community,” he said.
Reliable and complex
In one study, Harris and other researchers asked principals to rate specific teachers. They found that the results generally corresponded with those teachers’ value-added scores. But in several cases, a teacher deemed wonderful by a principal fared poorly when measured by student test scores.
In another study, the error rate was calculated to be 25 percent, based on three years of data, according to a 2010 study by Mathematica Policy Research -- meaning the model would incorrectly rate 1 in 4 teachers. And with one year of data, the error rate rose to 35 percent.
Harris points out that in Florida’s system -- which will label teachers as “highly effective,” “effective,” “in need of improvement” or unsatisfactory” -- minor differences of just one or two points could separate the teachers classified as effective from their colleagues deemed in need of improvement. And with major personnel decisions -- such as how much people are paid and who is fired -- being tied to the new evaluations, a point or two could matter a lot.
William Sanders, the statistician considered to be the “grandfather” of the value-added model in education, said the approach must find a “delicate balance” between woefully ineffective teachers who harm students’ futures and people whose careers are at stake.
“It appears to me that the balance we’re looking for is to use that early information to try to identify those (teachers) who are going to need the most help, the most guidance,” Sanders said.
The value-added approach, Sanders said, can target that extra professional boost to teachers who are rated poorly early in their careers, but who have the potential to improve. A study by SAS Research, the software firm where Sanders now works, found that half of the teachers who scored poorly after two years improved after five more years.
In terms of the actual formula, Sanders maintains that complexity is a good thing. He said any formula must include multiple years of student data to be reliable.
“A lot of people are arguing the methodology used must be so transparent that a teacher can go to their dining room table with a $2 calculator and scratch pad and calculate her value-added,” Sanders said. “What I say to schools and districts is: To trade simplicity of calculation for reliability is a devil’s bargain.”
-- Angeline Taylor, Herald education report, contributed to this report.