Saturday, January 12, 2013

Concerns about Online Assessment? Yes! It's CAT.

imageWhen I forwarded my desperate email to Diane Ravitch to my good friends and kind listeners, my friend, Christine, always so observant and such a careful reader, wondered what my concerns were with the NWEA MAP assessment. I realized that I had about 100 concerns within that email, and that if I am to help grow the national conversation, I need to take my rant down a notch. In this first blog, I am going to try to clarify my concerns with online tests, such as the NWEA MAP test because of computer adaptive tests.

Advocates of computer adaptive tests (CAT) say that the program behind the assessment tailors the test to the student's ability. No longer are students frustrated by an exam with items that are too difficult. Teachers and students are given immediate results; no longer do we have to wait for months before test results are given. The tests have RIT reporting that allows all to see how student learning grows over time and over years. Educator evaluation is required by law for every teacher in every building in Michigan. These MAP tests are being used to evaluate teachers in some districts, especially as pre- and post-tests are incorporated into classrooms. The CA tests are being used across the nation with growing numbers. They are purported to be cost effective, efficient, and objective.

I have several concerns about the CAT, especially with regard to equity. For the sake of time, I will limit this rant to three.  First, the process of taking a computer adaptive assessment can easily become a psychological issue that has absolutely nothing to do with demonstrating knowledge. I have watched 30-40 students at a time take CA tests, up to 150 students. Students who are not easily compliant and are sick of assessments find out quickly that if they choose random answers that the questions get easier and easier and the program soon kicks them out of testing. For students who are anxious and want to a good job, they begin to know in short time that they are getting items wrong, because the items get easier and easier. For struggling students, this reinforces the fact that they will likely get a "bad grade", further digging the hole of hopelessness. The longer the student is taking the test, the better they are doing. So. students who want to get a "good grade" on the test will be anxiously anticipating each question, self-assessing how they are doing on the tests. This becomes less an opportunity to demonstrate knowledge, and more a psychological "man vs. test" scenario.

Second, most of these CA tests are being made by companies who exist to make a profit. The testing industry is a multi-billion-dollar industry. The largest of these companies is Pearson that now owns the national lion's share of textbooks, student information systems, assessments, and other hugely profitable products. While NWEA is a non-profit organization, it is partners with Pearson. I have learned that when you want to know who has the power and is driving the bus? Look for the money trail.

Third, and most concerning, students do not have the ability to review their answers on a CA test. When they have finished with an item, they go on to the next. This flies in the face of what we know about learning and demonstrating knowledge. How many times have you been allowed to go back and make revisions on high stakes assessment? Even with the GRE (paper and pencil still in 1996), I had an opportunity to go back and review my answers. For these types of assessments one corrected answer can make a huge difference in the overall score. For any paper I write, I go back and make revisions and edits until I feel that it is 'ready.' Even then, I find mistakes afterward. The only time I have been unable to go back and check my work is when I was subjected to yearly standardized tests as a child. The final product or performance is an end result of revision, editing, and reviewing.

Our children must understand that self-correction is an indication of learning. With the stakes placed on these tests, students' inability to review their answers, correct mistakes, and make revisions fails to give us information about how students' thinking has changed. Thoughtful and careful test-taking is difficult, if not impossible.

Next up? Standardized testing.

15 comments:

  1. I never thought about the reviewing of answers. The CATs promote the "do it and forget it" mentality that we teachers fight everyday. How many of our students are successful at "learning" some material for a test and the nforgetting it (which of course means that they never truly learned it at all). Also, we preach to the students to check your work, to go back and rethink an answer or try to work it another way. That is impossible with CATs.

    ReplyDelete
  2. Good introduction blog. I am looking forward to more. However, it's not CAT, it's educators' lemming approach to teaching and learning and yes, the acceptance of standardized testing as a measure of what students know and can do. Even if we overcame the psychological,and social issues to which you refer, even if we could print students answers and review them with the students would we know any more about what our students need from us as teachers to help them learn and to help us teach?

    ReplyDelete
  3. The CAT approach is totally unfair, and you have uncovered the tip of the iceberg when you talk about how it prevents reviewing of answers. By presenting only one question at a time with no choice of other questions to look at, students are under enormous pressure to be able to answer those first few questions correctly or face a bad score. On a pencil and paper test, or even a computer test that allows you to skip around, students have the opportunity to find a question they can answer in order to build confidence. I know that sometimes when I start a test, I suffer a panic and think "I can't answer any of these." But then I relax and look for a question I can answer. For questions that I"m stuck on, I find that if I come back to them later, with a more relaxed test-taker frame of mind, I'm able to address it. CAT does not allow for this. It's do or die. I predict there will be many bad scores on such tests, and as usual, students and teachers will be blamed for the poor performance.

    ReplyDelete
  4. It would be nice if you could add some facebook, twitter, pinterest and google plus buttons so we can share your thoughts with others.

    ReplyDelete
    Replies
    1. I learned to blog through Twitter followers. I am happy to add, only I'm not sure how to do so. :) Will look into this right now.

      I am a work in progress!

      Delete
  5. In the words of Yong Zhao, standardized tests reduce education to test preparation.

    ReplyDelete
  6. "While NWEA is a non-profit organization, it is partners with Pearson." Huh? Since when? And what are you basing that on? (that's not actually true at all)

    ReplyDelete
    Replies
    1. Pearson lists NWEA as a partner here: http://www.schoolnet.com/company/partners/

      Delete
  7. You might be interested in his report issued in 2010 by the Kingsbury Center, NWEA's research arm: http://media.charleston.net/2010/pdf/kingsburymemoccsd_102210.pdf It recommends that "student test data not receive more weight than the principal’s evaluation in importance" and also notes that "...according to our most recent norms, the top 10% of schools for growth only have 64% to 73% of their students (depending on grade level) meet the “one year of growth” target (set by the school board intending to use NWEA as a value added metric). CATs will be expanding across the nation in the near future as many States signed on to administer the Smarter Balanced Assessment, which will be administered in 2014-15. Here's a link to their website: http://www.smarterbalanced.org/smarter-balanced-assessments/...

    ReplyDelete
  8. Have you considered that CATs is an assessment technology with more than 30 years of life? It's has been a long, very long history of research about it and its effects on performance. Many of your concerns are not new and, if CATs are still alive, it is basically because testing and CATs are good.

    ReplyDelete
  9. I appreciate your willingness to challenge my thinking.

    Do you know the story of the Dvorak keyboard? Despite its being developed for the way we use language, we continue to use the QWERTY keyboard. Just beause it's alive, doesn't mean it's good.

    Also, "New" may be a relative term. Multiple choice tests (in education)are only a 100 or so years old, while performance assessment is around 2000 years old. Performance is still being used to measure student achievement in Finland, always at the top of the world in student achievement.

    About the history of research? Great - evidence-based practice is critical. That being said, I want to know if the studies were about psychometric properties or about accurate reflection of student knowledge. There's a big difference here.

    Something to think about. Again, I appreciate your pushing back on my thinking.

    ReplyDelete
  10. so what can concerned educators and parents do to prevent the imposition of CAT on our students? The more I learn, the more I am horrified.

    ReplyDelete
  11. Took me time to read all the comments, but I really enjoyed the article. It proved to be Very helpful to me and I am sure to all the commenters here! It’s always nice when you can not only be informed, but also entertained! assessment centre platform

    ReplyDelete

Cultivating Empathy - 3-Minute Video with Transcript, Discussion Questions, and Selected References

  Discussion Questions How would you describe your level of empathy right now?  How would you describe the level of empathy in your school? ...