Item Analysis
I really enjoyed the readings by Scott Howell. I discovered that in some respects he is my boss as I teach a night class and he is apparently the director of night classes. One of the readings that I particularly enjoyed was “Improving Student Assessment—One Item at a Time.” I guess I liked it because it is something that I’ve been working on right now. I teach a Book of Mormon class this semester and I taught the same class this last semester. So I took “exam #2” from last semester and just completed doing an item analysis and test revision on that exam prior to administering exam #2 this year. Some of the things I specifically found were the same as mentioned by Scott in this article. One of these items was “distracter analysis.” I noticed that several of my questions had distracters that were completely ineffective—nobody was distracted by them! In addition I had several questions that had an item difficulty of 1.0, meaning that nobody got them wrong. I made some adjustments to my test to strengthen some distracters and eliminated some of the questions that were apparently too easy.
I have a couple of questions that I would like to ask Scott in our class. First, it seems to me that this article is mostly focused on norm-referenced tests. I’m wondering if I am trying to create a criterion test if he believes that I should still strive for item difficulty levels between .4 and .7, as well as a minimum discriminating power of .3.
I also want to learn more about what he terms a “test blueprint.” I am sure that I could benefit from such a product. I’m looking forward to hearing from Scott in class this next week.
Subscribe to:
Post Comments (Atom)
Posts
4 comments:
Personally, I am happy to hear that someone out there has actually adjusted and bettered a test based on item analysis. Good assessment practice is completely missed in K-12 education and most of higher education. I think that assessment should drive instruction and not the other way round, but it will be very difficult to get educatinal professionals to make that switch.
It is easy to be critical of educational professionals. We should turn the lens on ourselves. What are each of us doing in our teaching to effectively assess our students learning in whatever teaching opportunities we have whether it be formal education or even teaching a class at church.
Item analysis is an important tool to make a certain type of assessment better. I guess sometimes I have a gut level reaction to tests that are valued because of their ability to have good distractors. If distractors are indicators of commonly made mistakes, it seems like we should be addressing those specifically in the instruction. If we do a good job of that, they won't be much of a distractor any more - and then the ability of the test to sort students will be diminished and new distractors may need to be found:-(
I guess overall, I am a big believer in more authentic forms of assessment. There is a huge tension though because of the time it takes to provide feedback on those kinds of assessment and the subjectivity often involved.
So for example, in a religious education class, multiple choice tests can be used to assess a certain surface knowledge level. But it can't really get at the things that we most care about - which are the reasons why we have the students take religious education courses in the first place. I really like Dave Williams' assessment ideas that he has incorporated into his class. It seems that the overarching goal is to give students choice and to have them begin establishing patterns of reading, studying, and connecting their study of the scriptures to other aspects of their lives.
Thanks Mike for the comments--I also believe item analysis can be helpful.
@ Charles, I've carefully read Dr. Williams' syllabus, and I agree with your analysis. As you also point out, it is very difficult to do that kind of assessment when the number of students increases. The average religion teacher has five classes with a total of 350 students--so the brutal reality of teaching load conflicts with what might believe to be best assessment practice. By comparison, Dr. Williams teaches two classes with approximately 45 total students (advanced evaluation, and religion). I guess what I'm saying is it is one thing to say, "I like what Dr. Williams is doing" it's another thing to see how this could practically happen with scaled numbers.
I wonder though . . . I think that we can shape to a large degree what experiences we want our students to get out of a class. With limited resources you always have to make trade-offs. There are an infinite different number of creative ways to make those trade-offs.
I'm confident that if anyone can - you can figure out a way to make it happen - so that you feel less dissonance (like you mentioned) between your ideal and your practice.
Another thought that I had on this that might be valuable is the importance of helping people to figure out how to "self assess." Also, giving individuals opportunities for meaningful experiences without feeling like all of the experiences have to be assessed by the instructor.
There may also be ways that you can leverage learner-learner interactions in a meaningful way. An example that I saw a while ago was reported in Diane Robison's thesis. She looked at a science classroom (large lecture) where the instructor actually believed that students would learn best by trying to explain and teach difficult concepts to others rather than just listen to him teach the concepts. There was no way logistically that he would be able to personally have conversations with his hundreds of students - or even monitor those conversations. But he set up a system where students were required to teach someone else a principle each week and then self-report on that. The results were great. Were there some that cheated the system - almost certainly - but there were also lots of students that benefited from it.
Post a Comment