The key to evaluating teachers: Ask kids what they think

Thomas Kane is professor of education and economics at the Harvard Graduate School of Education and faculty director of the Center for Education Policy Research, which works with states and municipalities to evaluate educational policies. He recently partnered with the Bill and Melinda Gates Foundation on the "Measures of Effective Teaching" (or MET) project, which was intended to develop metrics capable of determining which teachers are faring better than others, and to determine what factors help determine success.

He recently wrapped up a randomized study with MET that identified a number of factors associated with quality teaching. We spoke on the phone Feb. 1; a lightly edited transcript follows.

Dylan Matthews: Tell me a bit about how this study differs from the rest of the literature around standardized testing.

Thomas Kane: So for 40 years, we have known that when similar students enter different teachers’ classrooms, they come out with very different achievement. For 40 years we have designed our education policies as though that weren’t true. Very few of those differences had anything to do with teachers' paper credentials, yet that’s the only thing that state and local policies focused on. They only focused on paper credentials, and they didn’t systematically try to evaluate performance on the job for teachers.

The test scores, we knew, were just the most obvious manifestations of what is a difference in practice underneath, but nobody was systematically trying to find ways to measure those differences in practices. Quite the opposite. Most classroom observations were entirely perfunctory. Teachers, 98-plus percent of teachers, were given the same "satisfactory" rating, if their principal did an observation at all.

It was within that context that we said, "Let’s go out and try to identify some ways to identify effective teaching that help illuminate what’s going on with the difference in test scores." We want to know that these are at least related to the magnitude of gains that teachers provide. So let’s do that in a way where we could develop measures that could be implemented widely. That was one of the advantages of trying to start with such a large scale [3,000 teachers]. If we had tried to do it with 250 or 200 teachers, we’d have something you could do on a small scale, not a large scale.

You asked how was this different. Before 2007, there were two feuding camps in the teacher effects world. There were the outcome, or value-added only, group, that tended to focus just on outcomes and say, “Look, the effort to try to measure practice is just opinions, and subjective, and hopeless. So let’s focus on the outcome data.” And then there were the folks who tended to focus on practice. There was an organization called the National Board for Professional Teaching Standards, where a teacher could submit videos, but there was actually a hesitance to include student achievement in any of those measures, for many reasons, many of them ideological.

We tried to collect data from student surveys so that we might in the process bring together what had been very separate research. They were publishing in their own journals. To the extent that they were aware of what the others were doing, they were dismissive and critical of what others were doing. By creating this framework where we were using test score gains to validate practice-based measures, we were at least creating a common base for discussion.

Dylan Matthews: And your methodology was tailored in a lot of ways to address the concerns of potential critics in those camps.

Thomas Kane: So there were two things that we did specifically because we were taking the concerns of skeptics seriously, but seriously enough to test them. One thing that we did was a random assignment. That was explicitly set out because skeptics of value-added had correctly pointed out that you can control for students’ observable traits like their baseline test scores, but there are lots of other potential determinants of student achievement that are unobservable, and if students are being sorted to teachers based on those unobservable traits, you could be seeing teachers with exceptional students, not exceptional teachers. That’s why we did random assignment, because we knew that this was one of the major concerns of skeptics, that there were these unobservable traits.

We didn’t just look at student achievement gains on state tests, but on supplemental assessments as well. People could point out that teachers who are achieving gains on the state tests could just be teaching to the test, so you wouldn’t want to have an evaluation system that was being evaluated against teaching to the test, the practices associated with drill and kill as opposed to practices associated with student learning.

Rather than dismiss it, but also not to say, "You’re right, we shouldn’t do any of this assessment," we said, "Let’s find an assessment that measures some of these things that aren’t being captured in state tests." We used, for example, the Balanced Assessment in Mathematics. There’s very little time spent on whether kids understand math procedures. Give a kid a couple numbers, and ask them to add them — that’s just testing their procedural understanding. But instead, we give them open-ended word problems that test their understanding of math.

Dylan Matthews: So what big things did you learn?

Thomas Kane: I think what we showed was that if you combine data from three different sources, from classroom observation, a student survey and a teacher’s past history of achievement gains, controlling in the ways that school districts are now commonly controlling for them, by controlling for students’ baseline test scores, you can identify teachers who cause greater learning to happen, and I can use the word "cause" because we used random assignment.

The teachers who appear more effective will not only generate greater gains on the state tests that you’re measuring, but they’re also generating greater gains on the supplemental tests that we saw, that the state wouldn’t see. Their students are reporting subjectively that they enjoy being in those teachers’ classes. That’s relative to what we have now, which is nothing. That’s a big deal. Do I think these measures will get better? Yes, and we can talk about what I think will be the next round, where we need to get better. But relative to the information that we’re giving principals now for making personnel decisions now, and teachers who want to do better, it’d be a huge step forward to combine these three.

Dylan Matthews: How much effect did different amounts of time spent teaching have? A common criticism of successful charter schools like KIPP is that they work their teachers so hard that they're not tenable in the long-term.

Thomas Kane: That’s an interesting point. We did not measure just the number of hours in the day that a teacher devotes. We know that their in-school time is equivalent, and all of our comparisons are within school. We didn’t measure the amount of out-of-school time that teachers spend grading schoolwork, maybe working with students outside of class, and it could be that the differences are there. So we didn’t have a direct test of that.

But our main goal was just to say, "What kinds of data could school districts be collecting either through classroom observations or student surveys to identify teachers who were having big impacts on kids?" I think, practically, if we learned that part of it was just that these teachers were working longer hours, that begs the question, well how would you measure it at a teacher level? If you get teachers to self-report the number of hours that they’re spending and you build that into their evaluations -- I never ask my son how much time he spends doing his homework. I read the homework. So I think that could be really important for telling us something about the length of the school day.

Dylan Matthews: Back to the three measures you think are important. My understanding is that classroom observation and student surveys — the two subjective measures — don't add much once you're already using value-added metrics. So why add them? Why not just use value-added?

Thomas Kane: Clearly one of the objectives is: Let’s find teachers who are most likely to generate gains and achievement on the state tests. And actually, if that’s your only objective, you’d put a ton of weight just on the teacher’s past history of promoting achievement on the state tests. Once you have a teacher’s past history of promoting achievement gains on the state tests, the classroom observations add relatively little.

But, two things that are a little different. The incremental value of the observations is different from asking if the results are related to achievement gains. Part of why they add little is that the part they add is redundant with achievement gains. But predicting future achievement gains is not the only objective. One thing you’d want to do is identify teachers who are more likely to promote achievement on these unmeasured skills. Another measure is reliability, and giving teachers feedback on specific practices that they may try to improve. Those other measures do better.

You want a measure that doesn’t bounce around extraordinarily from year to year. The student surveys are the least volatile from year to year and class to class. The reason is just the number of kids and the number of classroom days that they observe. We saw in the classroom observations that even if you take a trained observer and show them the same class, you’ll get differences. I know in D.C. they use outside peers, and our research suggests that’s a good idea. And most places aren’t doing that. Because there’s a judgment measure that you’d never want to eliminate, but you need to average over judges to get reliability.

You're getting two, three adults observing a class, but you might have 25 kids in elementary school classrooms, so the power of averaging really lends to the reliability of the student survey. For that criterion, the value-added scores are really not very helpful at all. But the student surveys and the classroom observations do point to things that a teacher could track his or her progress on. So because each different component -- the classroom observation, the surveys and the value-added gains -- excel at different objectives, that’s why it makes sense to combine them.

Dylan Matthews: The findings on outside observers who watch videotapes versus observe in person are particularly interesting.

Thomas Kane: It’s interesting, we did not have people walk into the classrooms. We had video cameras, and then we had people watch the videos and score them. A couple of things, though, that we learned from that. There was this one study, one of the reports we released was a study focused on Hillsborough, and so teachers could pick which four videos their administrator would see, but they’d collected 25 videos last year in Hillsborough, so we could have other observers, other than administrators, watch their videos so we could say, "How did their performance differ on days they show their principal versus days they didn’t want to show their principal?"

While the mean score was higher on the days that the teachers chose to submit, once you corrected for measurement error, a teacher’s score on their chosen videos and on their unchosen videos were correlated at 1. They were perfectly correlated. The people who struggled on the lessons they’re willing to submit are also the people struggling on the lessons they didn’t submit. The best lesson from the best teacher is that much higher than the best lesson from the worst teacher. The order is preserved even if the mean rises.

That has huge implications, because it means that the element of surprise may not be that important. That contributes a huge degree of anxiety, to have observers be able to pop in whenever they like. Give them a camera and say, "Submit four to five lessons you’re particularly proud of." I think that would remove some of the anxiety that made this hard, and in the process would have all sorts of other benefits. It would allow principals to time-shift. It would make it easier to get people outside the school involved in education. D.C. spends a lot to get those master educators to drive around to schools. If you could do this video-based thing, and still have them sit down with a teacher one on one to discuss these three or four lessons they submitted, rather than go out there physically, I just think it’d be a more efficient way to do it.

Hundreds of thousands of observations, maybe not millions, will be done with digital video rather than in person. This is just anecdotal evidence, but many teachers told us that having the camera in the room was a lot less distracting than having an adult in the room. You’re trying to read their body language, see if they’re bored, but the camera quickly disappears from people's consciousness.

Dylan Matthews: Now, the three components you measure predict future performance on achievement tests. But a lot of people dismiss that, even though there's growing evidence that achievement test scores are correlated with all kinds of important real-life outcomes. Why do these scores matter?

Thomas Kane: So I’m really heartened — even asking that question means you’re aware of that Chetty/Friedman/Rockoff study. That’s the main one that gives me hope. Yeah, that was in a not experimental context, but they showed not only — some of the reporting on that has been misleading. In fact, in last year’s State of the Union speech, the president said, "We know teachers have impacts on earnings later."

Well, that’s true, but actually if that was all Raj and Jonah and John had found, that wouldn’t have been all that useful. Because gosh. If you have to wait 20 years to find out who's a good teacher, you could send the teachers flowers but it’s hard to do anything in terms of policy. But that’s not what they found. What they found was that the teachers who appeared to have high value-added while the students were in their classrooms, their value-added was predictive of students’ income later. I’m optimistic about it but based largely on Raj’s study.

Bill: Create year-round schools that mix classroom, online learning

This is an archived article that was published on sltrib.com in 2013, and information in the article may be outdated. It is provided only for personal research purposes and may not be reprinted.

Utah schools may get to try a new approach: teaching year-round while mixing online learning with time in class.

The Senate Education Committee has approved SB79, which would provide $275,000 for grants for such pilot programs.

Up to 10 selected schools would have to blend online and classroom learning, extend the school year and teacher pay accordingly; use data to make decisions about instruction; and use competency-based education, in which students move to the next level by showing proficiency rather than just by putting in time.

"We know that when children are born, from the moment they leave the womb they are learning creatures," said bill sponsor Sen. Howard Stephenson, R-Draper. "Then they enter kindergarten and we command them to stop that self-directed learning because it's disruptive to the order of the classroom."

Schools could apply to participate, as could parents and students. Up to $75,000 of the money could go toward a consultant to help develop and implement the pilot program.

Several other lawmakers, as well as a representative from the conservative Sutherland Institute, also spoke in support of the bill, which passed 4-1 and now moves to the Senate floor.

"SB79 taps into the power of digital learning, which if done right, can empower children to guide their educations," said Stan Rasmussen with the Sutherland Institute.

But Sen. Patricia Jones, D-Holladay, expressed concerns, including the cost of the program. "I'm all for pilots to make sure something works, but it seems like bill after bill, we're just chipping away at the education fund … and I'm concerned about that," Jones said.

Sharon Gallagher-Fishbaugh, president of the Utah Education Association, also said the bill, to her, seems unnecessary given that schools can already make such changes without a new, specific program.

Stephenson, responding to the criticism, said he doesn't believe it's the legislature's role to micromanage schools, but sometimes pilots and incentives can lead to great innovations.

The bill would also create an advisory committee of leaders of legislative education committees, including Stephenson, to offer suggestions to the State Office of Education on selection of applicants.

Education Bills and Background

Lots has been happening on the education front this year, though the most important thing is happening in closed rooms as the House and Senate work out their budget proposals. I sat on a school funding panel in front of the Washington State School Directors Association today with Sen. Bruce Dammeier (R- Puyallup), Rep. JT Wilcox (R-Yelm), and Sen. Andy Billig (D-Spokane.)

All of us were willing to make substantial investments in K12 education funding, though we all had weird conditions and our numbers were all over the map, ranging from $900 million for the House Republicans to $1.4 billion from me, the amount specified by the Joint Legislative Task Force on Education Funding this summer. I take this as a hopeful sign – they’ve all figured out that we have to invest in a quality system in order to get the results that we want.

Both Republicans wanted to ensure that there were “changes” to the education system that would make it more effective before they invested the money. I agree that there should be changes, but we should have some humility about our ability to change the system rapidly. There are 1 million children in our K12 system, and about 100,000 adults. This is not a ship that changes direction on a dime. There are a handful of long-term changes we’re engaged in that you will see bills about this year:

Graduation Requirements: We are in the process of upgrading our graduation requirements to something that looks more like Bellevue or Lake Washington’s than like the rest of the state. We’ll require 24 credits, not the 20 required now. Most students will take a course of study that will prepare them for education after high school, either college, community college, or some technical alternative that gets them a credential. Without this preparation it will be very difficult for our young people to make a living, and consequently to move out of our basements, something greatly to be desired. To read about this you can look at the State School Board’s document on it.

It’s important to not change these all the time or families can’t keep track of it, but we are working on phasing in stronger requirements. I unequivocally support this effort.

Testing: We are moving away from writing Washington-specific tests. Two national consortia have developed coordinated sets of tests based around the “Common Core” standards. Led by the National Governor’s Association the states got together and developed these standards. Washington signed on several years ago. We have adopted a set of tests from the “Smarter Balanced” group, one of two groups developing tests for this set of knowledge. Most first-world countries have a national curriculum, and this is a pretty reasonable attempt at it. There is lots of grumbling that we should have state-specific standards, as if algebra is different in Wisconsin than it is in Washington.

Being part of a consortia saves us tons of money (I’m expecting to save about $35 million on test development alone) and builds better tests that get affected less by individual strong personalities and fads sweeping through the education world. We have not worked out exactly what the changes will be, but you should expect to see less time spent testing, and tests that are more aligned with what college requirements are.

The tests will be delivered on computers, allowing faster scoring and tests that adjust to the level of the student. We’ll have work to do to ensure that each district has enough computers to do this, but I don’t expect this to be a big problem.

Teacher and Principal Evaluations: Washington has adopted a program to provide much stronger teacher and principal evaluations that are partly dependent on student learning data. We’re in the first few years of implementing this and should stay the course. There is some blow-back, as you might expect. (Seattle teachers are not administering a pretty routine test solely because (IMHO) it is a component of their evaluations.) We can’t depend totally on student learning data or everyone would want to teach in Medina and nobody in Yakima – that’s why it’s important to look at student growth instead of absolute values. This requires some subtle statistical work, and is one of the reasons we don’t turn teachers into salespeople on commission – the student learning data should be part of the eval, but not all.

There will be lots of push to change this system, but we should let it go through the adoption phase without screwing with it too much. It takes a long time to change the direction of large ships, and this is a really big one. It will require some money to train principals and teachers on how to work with the new system.

Bills in the System

The Senate passed out a block of education bills last week, and the House did about half their bills on Friday. Short list, cribbed liberally from the analysis of Stand for Children, theLeague of Education Voters, the Washington State PTA, and our internal staff.

SB 5243 - Academic Acceleration, automatically enrolls every student who is qualified into more rigorous advanced classes. The House passed a weaker version. There’s lots of data that shows that students who are in challenging courses do better afterwards, even if they don’t get a passing grade on the AP exam.  Kids that get college credit via running start, AP, IB or other rigorous systems do better in college, and save money for themselves and for the state. This can be critical for a middle-class family, and we want to encourage it as much as possible.

SB 5491 - State Education Goals, establishes statewide goals for students at key milestones, such as 4th grade reading, 8th grade math, and high school graduation. This is boring, but having some agreed-upon goals can really help folks get focused on what really matters.

SB 5329 – Transforming Persistently Failing Schools, allows the State Superintendent (OSPI) to bring in new leadership, innovation and resources to help persistently struggling schools improve. This is a very controversial bill that creates a requirement that the bottom ten lowest performing schools in the state must be actually changed or something will happen. The process is tortuous, but we don’t really want the state taking over very many schools as we are unlikely to be able to do it at scale. I would tend to agree that at some point if a school remains in the bottom ten state-wide for many years that something needs to be done. This seems like a reasonable process, but it still needs some work.

SB 5237 - 3rd Grade Reading, provides early intervention and support to make sure every student is reading at grade level. Originally this required that 3rd graders who couldn’t read be held back. The new bill has a lot of funding for intervention – required summer school, extra help during the school year, etc. It’s critical that kids learn how to read proficiently in the 3rd or 4th grade. They tend to fall behind catastrophically if they do not. Again, this will be a contentious bill.

SB 5242 - Mutual Consent, ends forced placements by requiring a teacher and a principal to agree before a teacher is assigned to work in a specific school. Again – very controversial. Teachers who aren’t hired go into a temporary position for a year, then can be let go. I do not yet have an opinion on the details of this bill, and the devil will be in the details.

SB 5328 – Letter Grades for Schools, provides parents with clear information by assigning schools a letter grade A-F for their performance based on the state Achievement Index. I’m not sure about the value of this, but allows parents to have a simpler way to evaluate how well their local school is doing. The key is getting the achievement index right – you don’t want to penalize schools that have a lot of at-risk students show up in the first years, but do great work with them, and conversely, you don’t want to reward schools with lots of well-prepared kids who do nothing to advance them.

SB 5244 - Student Suspensions, limits the length of time a student can be suspended from school and improves data collection on school discipline. There is a growing body of research that says that kids who are expelled don’t do well. This makes sense – they aren’t in school learning.

SB 5587 - State Assessments, establishes a transition schedule to move from existing state exams to the new Common Core college- and career-ready exams as graduation requirements for the class of 2018. The House bill is stuck in committee for reasons I don’t understand.

HB 1723 – Early Start, Helps build an integrated early learning system, including preschool, child care, and services for at-risk infants.  Details of the system would be left up to a taskforce with the overall goal of building a comprehensive system for children zero to 5 years old.

HB 1671 – Child Care. Improves the child care system for low-income children and increases the quality of child care these children receive by improving the Working Connections Child Care program, a program that provides child care to low-income parents who are working or in school.

HB 1872 –  Science, Technology, Engineering and Math. Gov. Inslee request bill improving outcomes for STEM. My guess is that this bill will have a lot of changes before we are done.

HB 1692 –  Career and College Ready Graduation Requirements. Rep. Pat Sullivan’s bill to implement the supports necessary for the career and college-ready graduation requirements I referenced above. This is profoundly expensive, adding additional instructional time called for in the bill the McCleary decision calls out as the framework for our constitutional requirements for funding.

HB 1680 – Strategies to Close the Opportunity Gap.  A collection of policies recommended by the educational opportunity gap oversight and accountability committee. This bill points out a fundamental discussion point in how we look at K12 funding. Do we fund specific programs that worked in one district statewide, or do we drive funding out to districts based on an allocation formula and make sure that they have opportunities available to them to select evidence-based programs that will work in their community? As you can tell from the language, I’m in the latter camp. What will work in Yakima may or may not work in Bellevue or Redmond.

HB 1424 – Dropout prevention and Retention. Similar to 1680, this lays out a set of programs that have worked and tries to create funding streams for them. I’m not a fan of all these little funding streams – I think we do better by providing funds to districts and letting them pick the programs that will work.

HB 1177 Accountability. This bill sets up a financial scenario that allows state funding to be used to assist schools that are not meeting their student learning goals, much as SB 5329 does, but it does not provide the intervention strategies needed to transform the schools.

The process of coordinating and reducing the number of bills is likely to take us a while this year. These bills all go through multiple steps and often change substantially before they are finally released. I would expect that we will create additional policy around assessments, around intervention in failing schools, and around early learning, the most powerful investment we can make.

Creating new possibilities in the classroom with iPad.

The inspiration

Located about 20 minutes outside Boston, Burlington High School is a public school in a middle-class suburban community. Its student body of about one thousand 9th-12th graders has performed well by traditional standards. With a 100 percent pass rate on the state examination and 95 percent of graduates going on to college, it appeared that Burlington had no need for change.

But the progressive-minded faculty felt they weren’t doing enough by just helping students perform well on tests. They wanted to adapt their teaching methods to an increasingly digital world, to help their students better prepare for real life.

The educators at Burlington recognized that students get most of their information from the Internet. “So students need teachers more than ever,” says former principal Patrick Larkin. “We need to guide them in determining what is a valid source and what is not.” By incorporating digital literacy standards into the curriculum, the goal was to create a teaching environment using the same technology that students were already using outside school.

It was important to the administration to move away from traditional textbooks. Burlington chose iPad as a learning tool because it gives students access to the world as it is today. “We saw this as a huge opportunity to democratize information,” says history teacher Todd Whitten. “With iPad, students have this little lightweight device that contains more knowledge than a traditional textbook could ever have, and it’s all current.”

The implementation

Administrators at Burlington were committed to providing each student and teacher with an iPad to use throughout the school year. To purchase the devices, the school found funds within its existing budget by eliminating costly computer labs and deciding to forgo printed textbooks.

Teachers were thoughtful about their roles and how they might change now that their students had a device giving them ready access to incredible amounts of information. They regarded themselves not only as providers of knowledge but also as guides, helping students navigate and analyze information. “I believe that part of the responsibility of the 21st-century educator is to take the time to help students think through the perspectives and points of view of the sites they’re visiting, and decide whether or not they wish to adopt these points of view as their own,” says Whitten. “It is important to have the discussion about what is ‘correct.’”

Using iBooks Author, teachers at Burlington create interactive course materials for iPad to keep their students engaged. English teacher Tim Calvin is able to present students with more than just text on a page. “I can create a book that has pictures and videos and hyperlinks in it. It’s great to be bound not by what’s out there, but only by what you can make.”

Students and teachers across disciplines are finding new ways to make learning more interactive. In Spanish class, students practice speaking by recording and playing back their voices. In economics class, students pull real-time stock market data to create and track a stock portfolio. And in math class, college-bound students research market costs to create a realistic housing budget.

Giving students access to iPad also allows teachers to better adapt to their students’ unique needs and abilities. Students are no longer limited to writing essays and answering multiple-choice questions in order to demonstrate what they’ve learned. “As educators, we still need to focus on learning outcomes,” says Larkin. “We have multiple ways for students to show their level of proficiency.” With iPad, students have more freedom to express themselves. They can make a video, design a digital poster, or create a website. If a student prefers a more analog approach, teachers can be equally flexible.

The results

In its first year, the iPad one-to-one program has already made a big impact at Burlington High School. According to an end-of-the-year survey, the majority of students indicated that they were more excited to attend class and more organized in their learning than in previous years.

Teachers are seeing a different level of engagement — including richer, deeper conversations — from students when they use iPad. “Rather than simply parrot back the same things, students can engage in a discussion about why sites present information differently, and come to their own understanding,” says Whitten. “In my view, that’s actual learning, and iPad makes that more possible.”

Larkin, the former principal at Burlington, has since been promoted to assistant superintendent of the Burlington School District. Because of the program’s success at Burlington High School, more than 50 schools from all over New England have come to observe the program in action. The school district has already expanded the iPad one-to-one program to its middle school, with the ultimate goal of expanding the program to grades K-12 across the district.

Should student feedback be used in a teacher's overall evaluation?

In the first installment of this exciting new blog series, Todd and Kate offer their perspectives on the controversial topic of the use of student feedback in 300 words or less. Many teachers survey their students during the year on their performance; for a variety of reasons, others feel the results of student feedback surveys are untrustworthy. In light of new research, we in Washington are again considering the idea of including results of these surveys as a part of a teacher's evaluation. Some still aren't buying in, though. Read on to hear what Todd and Kate think. Respectful comments are welcome. Enjoy.

Kate says:

Much is at stake for our students these days, and I pride myself on being the best teacher I can be every single day.  I only trust my students’ feedback implicitly.  I gratefully listen, but with caveats, to parents (who might be less likely to understand pedagogy), administrators (who are often too busy to really know the heart of our work), mentors (who many districts cannot afford), and colleagues (who have little time to observe and mentor.)  But the kids are there every single day.  They know what I did right.  And wrong.

At the end of 4^th grade, on my self-imposed, annual, year-end survey, I got dinged by Gillian.  My offense?  Asking her to wait until 5^th grade to test into reading level Z.  A bazillion times I told her: it’s just increased mature content.  The reading skills are identical.  I made a choice she didn’t like.  It’s an unintended consequence.

In considering my professional evaluation, I have questions about the global application of student evaluation surveys.  Like many new and exciting ideas in education, I suspect good intent will lose quality in execution.  I envision a morphed, soulless survey in a shaky, standard, blanketed application, applicable to all but valuable to none.  I question whether assessing me via bubbles captures the essence of my classroom; Do kids recognize what’s good for them; How will Sarah be assessed by her EBD kids; Juli’s medically fragile kids can’t comment; I wonder how primary kids’ thoughts will be collected; How about incarcerated youth; How would we capture the opinions of high school kids who’ve dropped out?

Some things, like this, belong in the “best practices” column, and not the “quantify & globally apply” column.  This is strictly a best practice.  Listening to our students is powerful.  Let’s keep it so.

Todd says:

Who spends the most time observing teachers in the classroom? Administrators? Principals? Peers? Coaches? The answer is none of the above. In reality, students spend far more time observing classroom teachers than any other group. While they may not be recording their observations on official-looking forms, they perceive far more than the majority of educators probably realize.

Student perception surveys have long been regarded as biased and unreliable. But that myth was recently dispelled by Dr. Ronald Ferguson, Director of The Achievement Gap Initiative at Harvard University. He led the development of the student surveys and analysis methods used in the Measures of Effective Teaching (MET) Project, a three-year study which included 3,000 teachers from seven urban school districts.

The MET project revealed student perceptions do predict learning outcomes and should be taken seriously by policy makers and educators. Not only are they consistent and reliable, they are a better predictor of student achievement than either classroom observations or test scores. Quality student surveys actually provide detailed insights about teaching and learning, including attributes not measured by standardized tests or principals.

For example, students were asked how strongly they agree with statements such as “My teacher wants us to share our thoughts,” and “l Iike the way we learn in this class.” Not surprisingly, there was a strong correlation between agreement with MET statements and higher student achievement levels. Of course there are obvious cautions, but there are cautions using classroom observations and test scores too.

Teacher evaluations should be based on multiple measures, given multiple times, over multiple years. We shouldn’t be satisfied with only two measures - classroom observations and test scores. When asked the right questions, students provide valuable feedback about teaching and learning. Student perception surveys would be a healthy addition to our current evaluation model.

Forget the future: Here's the textbook I want now

The old paper form of a textbook is certain to die. I'm sure of it.
The new form of a "textbook" has a feature list that turns the textbook from something people read to something people experience. Note that this feature list isn't fantasy, nearly all of these features already exist in some form.
Here are the features I think every textbook should have.
The textbook should be 100% searchable. No more wondering where eukaryotic appears in the text. You'll just be able to quickly type in a search term and find all of the places it appears.

 
Key words in the text should be linked to explanations of these key terms. Click on the word, find out what it means in this context and what other resources exist to understand it.
 
The readability of the text should be individually customizable. Want to challenge yourself and improve your vocabulary? There's a setting for that. Feel like taking it easy on the reading? There's a setting for that too.
 
Everything in the textbook should allow annotations which should appear as a user generated summary of the textbook itself in another location.
 
Users should be able to add bookmarks and tag parts of the textbook with terms so they can self-classify the information. These tags should optionally appear for other users of the same textbook.
 
You should be able to comment on any part of the textbook. This could be used to flag out-of-date content or just to ask questions. Each user of a textbook should optionally be able to see everyone else's comments on various sections of the text. These comments should happen in real time so that users can chat in real time about what they are examining.
 
Videos and other multimedia should be included in the textbook where appropriate. Want to talk about MLK's I have a dream speech? You can include the entire video of his speech as part of the book.
 
The textbook should be customizable. Users should be able to edit the content of the textbook and share the updated version of the textbook with other users. When a customization occurs, the original author(s) of the textbook could optionally be notified so they can either accept or reject the changes to the original work.
 
The textbook needs to be open source and free. No longer bound by restrictive and antiquated licenses, institutions can create their textbooks and share them with the world.
 
Textbooks need to be translatable if they are really going to be free to use for everyone. No longer would the language learners in your class be forced to struggle in your subject just because of a lack of knowledge of the language of instruction. Optionally you could have the textbook display in the language of instruction and have real-time translation services available for any section on demand.
 
For any section of the text, real time search of other resources or references needs to be available. Instead of relying on just the opinion of the author(s) of the text, now you can look at other (optionally screened) resources that could help understand some perspective on the subject of the textbook.
 
The textbook should be device agnostic and mobile-ready. It shouldn't matter if the person is reading it on an ereader, a netbook, an iPad, or a cell phone, the textbook should be available anytime, anywhere to anyone.
 
The textbook should be built with multiple models of pedagogy in mind. Instead of flatly stating the "facts" for the student reading the textbook, there should be opportunities for experiments, simulations, 3rd virtual worlds, or whatever other alternate forms of representation are available. Inquiry should be built into these textbooks.
 
Students should be able to click anywhere in the book and ask the question, "where is this used in the real world?" No more students asking why they are learning this stuff, because the entire learning process would be transparent.

You should be able to ask an expert on the topic from your textbook. Need more help with the topic than the textbook is providing, or have some more questions? You can call someone for help and ask for advice right through your textbook.
 
Your textbook could be a centre of a community of people who are all learning the same material. Not all of you need to be in exactly the same class, but as you work through the textbook and make comments, the textbook learns from you about your learning habits, strengths, and weaknesses, and connects you to the people and resources that you need to understand.
 
Any practice or other tasks that need to be done through the textbook should be included, if appropriate, and immediately assessed. No more waiting for feedback.
 
The textbook should be modular. This would allow for construction of textbooks from many different sources, potentially choosing the most effectively created resources for each section. Students could create their own textbooks for their personal study, selecting resources that they find to be the most effective for them. In fact, students could contribute modules to a textbook as part of a capstone project for their course.
 
The textbook content should include metatags, which should be searchable, so that over time related content can be found, and some of the connections between different content areas are made more clear.
 
Update: Thomas Baekdal reminded me of a couple of more important features: First that the textbook be non-linear so that the learner can access it in any order, and that the textbook should allow for embedding from sources anywhere on the web.
 
The most important feature I can think of in a textbook should be that it should be at most a place in the learning process, and help the learner develop further questions that they can explore for themselves. It should not be something that stops a learner from wondering.
What else would you like to see in a textbook?

Need a Job? Invent It

WHEN Tony Wagner, the Harvard education specialist, describes his job today, he says he’s “a translator between two hostile tribes” — the education world and the business world, the people who teach our kids and the people who give them jobs. Wagner’s argument in his book “Creating Innovators: The Making of Young People Who Will Change the World” is that our K-12 and college tracks are not consistently “adding the value and teaching the skills that matter most in the marketplace.”]

This is dangerous at a time when there is increasingly no such thing as a high-wage, middle-skilled job — the thing that sustained the middle class in the last generation. Now there is only a high-wage, high-skilled job. Every middle-class job today is being pulled up, out or down faster than ever. That is, it either requires more skill or can be done by more people around the world or is being buried — made obsolete — faster than ever. Which is why the goal of education today, argues Wagner, should not be to make every child “college ready” but “innovation ready” — ready to add value to whatever they do.

That is a tall task. I tracked Wagner down and asked him to elaborate. “Today,” he said via e-mail, “because knowledge is available on every Internet-connected device, what you know matters far less than what you can do with what you know. The capacity to innovate — the ability to solve problems creatively or bring new possibilities to life — and skills like critical thinking, communication and collaboration are far more important than academic knowledge. As one executive told me, ‘We can teach new hires the content, and we will have to because it continues to change, but we can’t teach them how to think — to ask the right questions — and to take initiative.’ ”

 My generation had it easy. We got to “find” a job. But, more than ever, our kids will have to “invent” a job. (Fortunately, in today’s world, that’s easier and cheaper than ever before.) Sure, the lucky ones will find their first job, but, given the pace of change today, even they will have to reinvent, re-engineer and reimagine that job much more often than their parents if they want to advance in it. If that’s true, I asked Wagner, what do young people need to know today?

“Every young person will continue to need basic knowledge, of course,” he said. “But they will need skills and motivation even more. Of these three education goals, motivation is the most critical. Young people who are intrinsically motivated — curious, persistent, and willing to take risks — will learn new knowledge and skills continuously. They will be able to find new opportunities or create their own — a disposition that will be increasingly important as many traditional careers disappear.”

So what should be the focus of education reform today?

“We teach and test things most students have no interest in and will never need, and facts that they can Google and will forget as soon as the test is over,” said Wagner. “Because of this, the longer kids are in school, the less motivated they become. Gallup’s recent survey showed student engagement going from 80 percent in fifth grade to 40 percent in high school. More than a century ago, we ‘reinvented’ the one-room schoolhouse and created factory schools for the industrial economy. Reimagining schools for the 21st-century must be our highest priority. We need to focus more on teaching the skill and will to learn and to make a difference and bring the three most powerful ingredients of intrinsic motivation into the classroom: play, passion and purpose.”

What does that mean for teachers and principals?

“Teachers,” he said, “need to coach students to performance excellence, and principals must be instructional leaders who create the culture of collaboration required to innovate. But what gets tested is what gets taught, and so we need ‘Accountability 2.0.’ All students should have digital portfolios to show evidence of mastery of skills like critical thinking and communication, which they build up right through K-12 and postsecondary. Selective use of high-quality tests, like the College and Work Readiness Assessment, is important. Finally, teachers should be judged on evidence of improvement in students’ work through the year — instead of a score on a bubble test in May. We need lab schools where students earn a high school diploma by completing a series of skill-based ‘merit badges’ in things like entrepreneurship. And schools of education where all new teachers have ‘residencies’ with master teachers and performance standards — not content standards — must become the new normal throughout the system.”

Who is doing it right?

“Finland is one of the most innovative economies in the world,” he said, “and it is the only country where students leave high school ‘innovation-ready.’  They learn concepts and creativity more than facts, and have a choice of many electives — all with a shorter school day, little homework, and almost no testing. In the U.S., 500 K-12 schools affiliated with Hewlett Foundation’s Deeper Learning Initiative and a consortium of 100 school districts called EdLeader21 are developing new approaches to teaching 21st-century skills. There are also a growing number of ‘reinvented’ colleges like the Olin College of Engineering, the M.I.T. Media Lab and the ‘D-school’ at Stanford where students learn to innovate.”

North Carolina pushing for digital devices to replace textbooks

Over the next four years, students in North Carolina public schools will trade in their traditional textbooks for digital devices as the state shifts how it funds educational materials.

Last week, Gov. Pat McCrory signed into law a bill that calls for public schools to allocate more money for devices such as digital textbooks and iPads. The legislation is one of four bills designed to address the changing landscape of public education because of the growing use of technology in schools.

“This is where we have been going the past couple of years,” said Cumberland County schools Superintendent Frank Till Jr. “Textbooks are becoming an antiquated way to learn. This puts a stake in the ground that says textbooks are passe.”

School systems have until 2017 to align curriculum with digital devices and make the transition to digital instruction, the law states. A similar mandate to move toward digital instruction by the 2015-16 school year exists in Florida, and Education Secretary Arne Duncan has called for all U.S. schools to move toward digital learning within five years.

The use of digital textbooks and devices can “raise the level of academic performance of the state’s students” and provides students with “high-quality, up-to-date information that can be customized for individual students throughout their educational experience,” according to North Carolina’s new law.

“The thinking is that there are many school districts moving to digital learning,” said Neill Kimrey, director of digital teaching and learning with the state Department of Public Instruction. “There are a few districts that are resistant to change. This bill draws the line in the sand.”

School systems will have flexibility to use state money designated for textbooks or local money for technology. The legislation suggests using additional funding sources, such as private grants and parental contributions. Schools can adopt a “bring-your-own-device” policy that allows students and teachers to use their own technology in the classrooms.

The bill does not allocate state funds for digital devices. However, legislation is pending that would allow lottery funds to be used for technology.

Cumberland County officials are not optimistic that the state will allocate additional funds for digital devices, and there is little local money to divert from textbooks.

The legislature drastically cut funding for textbooks to help balance the state’s budget. During the 2009-10 academic year, more than $59.6 million was allocated by the state for textbooks, according to the state Department of Public Instruction. Only $2.5 million was allocated the following year. This year’s allocation is about $21.2 million.

Cumberland County schools have received about $700,000 in state funding for textbooks for the past two years.

The school system is looking at federal grants, including Race to the Top, to help pay for some of the technology.

Even if legislation is passed to free up lottery revenue for textbooks, Till said officials won’t be able to use that pot of money. Cumberland County is using that money to pay for the system’s debt on recently constructed buildings.

Till said students and faculty likely will need to provide their own devices in the classroom, but that, too, offers complications.

“Some students don’t have their own cell phone,” he said.

Tech in the classroom

While leaders consider how to buy the devices, technology is being integrated into the curriculum at several Cumberland schools. At Pine Forest High School, a grant helped pay for iPads, laptops, and other digital devices.

Instructors in the school’s Academy of Emergency Medical Sciences use the tablets to illustrate medical procedures. During a recent class, students used an iPad application to simulate CPR.

“It brings it to life more than I ever could just by my words,” said Michelle Quinn, the director of the academy that trains students interested in pursuing careers in medicine and health care.

Several of Quinn’s students praised the legislation. Kayla Holliday, a junior, said it makes sense to shift to digital learning because today’s youth grew up using technology.

“It almost gives you a new perspective on the material when you can use the technology that you already use for the things that you enjoy,” she said.

Trevor Hendricks, also a junior, said the digital devices are more convenient than textbooks.

“It definitely helps us because you get more current information than what you can from textbooks,” he said.

Pine Forest Principal Jane Fields said she is glad school systems will have time to train teachers how to use the devices and software and properly integrate them in the classroom.

“It’s going to ask people to step outside their comfort zones and do things differently, and you need time to adjust,” Fields said. “The professional development that we provide is imperative so that teachers can use it as a tool to engage their students and expand their learning process, not cripple their thinking skills.”

Fields expressed concerns about the sustainability of the technology. Most digital devices are outdated within a year. School systems may not have the funding to keep up.

The change also would mean that textbook companies would have to find ways to provide more digital content. Cumberland County schools are not under a contract with a textbook company to provide digital content. Teachers are using online sources in the classroom.

Officials have met with several textbook companies and have reviewed their digital content, said Kevin Coleman, executive director of technology for Cumberland County schools.

“Converting textbooks to digital content is a move in the right direction,” Coleman said. “I am excited about the governor’s push toward digital learning.”

Moore County is in the first phase of a four-phase plan to move to digital learning and sustainability. One elementary school and an alternative school are using digital devices. Three middle schools will be equipped with digital devices in the next month, Moore Schools Superintendent Aaron Spence said.

Although the intentions of the legislators are good, Spence said, the lack of funding for the digital learning initiative may hinder the bill’s goal.

“If the state really intends to be a statewide digital learning system,” he said, “it will take a commitment of money and not shifting money that currently exists from already-stretched school budgets.”