Steve Smith's Assessment Journey

I started teaching in a volunteer ESL Program for a local church that was next to the University of Washington which I was attending as an undergraduate. I spent the first week doing a lot of exercises I had found in an ESL instructional book of which many were written. I was working with a group of Hmong. It wasn't until the second week that I realized that many of the students were illiterate in their own language. Much of what I had been doing had been worthless. I realized there was more to this teaching and learning business then I had realized.

After graduating, I moved to Ecuador, South America. I taught English and eventually became the director of a language and cultural center. I was responsible for 30 Spanish and English instructors many of whom had no teaching experience. I began to research teaching and learning strategies. I started to wonder if there were a systematic way to approach teaching and learning.

After 5 years we returned to the US. I started teaching computers. I also started a master’s program in adult education. Instructional design, Knowles’ principles of Andragogy, Gagne’s 9 Instructional Events, Constructionist theories of teaching and learning opened my eyes to a whole new vision of teaching and learning. I also was introduced to assessment through Kilpatrick's 4 levels of evaluation. While now outdated, it transformed how I viewed evaluation. Can the student do the learning task in class was only the beginning. Can they perform it outside the class without classroom support and finally the most important and hardest to assess; did the learning solve the original problem? I realized that what I did inside the classroom needed to be assessed at least in part on what the student could do outside the classroom. This was a monumental shift in how I viewed teaching and learning. I started to believe that if it could be measured, it could be learned and that instead of a Bell curve of grading my expectations were that everyone could succeed if I applied the appropriate instructional design principles.

I taught off and on in various formats including distance learning for the next 10 years. Recently, I finished the coursework for a PhD in Community College Leadership. In this program I was introduced to the concepts of Chaos Theory, Freire’s Transformational and Social Critical theories of learning, Qualitative vs. Quantitative research, living systems and Wheatley’s application to organizations. I realized that I had become too reductionist in my teaching and learning. I needed a more holistic approach. Some things are hard to measure and when you try to measure them, they change.

These experiences have shaped my view of assessment. I believe in the concept of assessing learning based on what the students can do “out there”. I believe that we need to measure not only what goes on in our classroom but also the larger core outcomes. This process is messy and we may not always be able to cleanly assess some of the critical learning components which happen in our classrooms such as the student is more confident, more engaged in their own learning, open to new ideas, more excited about continuing with their education and more willing to take emotional and intellectual risks. These are the mana from heaven that we seek out as teachers but may not be able to necessarily assess. I believe we owe it to our students to keep struggling to find the right balance of assessing with realizing that learning is not necessarily the sum total of its parts. Assessment is not an either or but rather an and/and focus. We need to assess to continue to grow and improve as an institution while allowing room and time for those things which are difficult to assess to flourish.

Steve Smith is the director of Curriculum Support Services

What does PCC get from being accredited, anyway?

From Laura Massey, Director of Institutional Effectiveness

I don’t know about you, but I am not motivated by scare tactics. A sentence beginning “Without accreditation” followed by frightening statistics does nothing more than bring forth my inner twelve year-old who mentally shuts down while rebelling in full voice. Instead, here are a few facts.

Because PCC is an accredited college . . .

• 
  Almost 40% of our students are able to receive Federal Financial Aid dollars.
• 
  The credits earned by 5,100 students who transfer to another college or university (each year) are accepted at the transfer institution.
• 
  Last year’s 3,400+ graduates have credentials that are valued by other institutions, employers and licensing agencies.
• 
  Almost $14 million in recently awarded grants will provide improved student services, new equipment, expanded curriculum and so much more.

But wait. This is supposed to be an assessment blog. Why the emphasis on accreditation?

Because to maintain our accreditation, PCC is required to “hasten its progress in demonstrating, through regular and systematic assessment, that students who complete their programs have achieved the intended learning outcomes of degrees and certificates. Further, the college must begin to demonstrate, in a regular and systematic fashion, how the assessment of student learning leads to the improvement of teaching and learning” from the visiting team response on behalf of the Northwest Commission on Colleges and Universities, Spring 2010.

Scare tactic? Hardly - although it is certainly an important statement requiring action. Let’s first put this in context at PCC.

We care deeply that our students are learning. Period. It is our professional responsibility to understand what is working well and make improvements when needed. This is nothing new or different from what hundreds and hundreds of PCC faculty do in classrooms each day.
However, Northwest is also asking we demonstrate (in their language that means document) that we regularly and systematically assess graduates to have achieved the College defined outcomes. Furthermore, that we link how we used what is learned through assessment to make changes where needed to improve teaching and learning.

The faculty-driven Learning Assessment Council developed an assessment model which was initially implemented in 2009-10. As support staff to that work, I could see how institutional learning is both energized and realized through faculty creativity, insight and on-going commitment to excellence - the same qualities that support student learning.

As we ‘”hasten” our progress and fully implement the model this year, I believe we are on the path to fulfilling Northwest’s recommendation.

From Cynthia Killingsworth, Accounting Instructor

I enrolled in Sylvia and Shirlee’s assessments class last spring with a narrow objective of improving testing in my accounting courses. This class ended up being a major teaching style alignment and I learned to appreciate the unavoidable bottom-line of the assessment movement. We talk a lot about measuring student performance, but it is really a matter of facing our own teaching effectiveness head-on. The harsh line “It’s the teaching, stupid!” comes to mind from a 2006 article about retooling math education. Here are some of my thoughts from this experience:

Do I understand what should be assessed? I discovered that I was assessing concepts easiest to measure, such as preparing a financial statement in proper format, but may have failed to fully consider those concepts harder to measure, but equally relevant for student success, such as interpreting financial information. This led to a rapid detour away from testing as my focus and towards reconsidering what I was teaching.

Do I understand what is most relevant for course content? In this detour I had to invest some time finding actual research in my discipline regarding the concepts or skills important for accounting student success. Ironically, the PCC assessment focus last year was critical thinking and problem solving, and when I researched my discipline, I rediscovered that the Accounting Education Change Commission had completed an extensive study in the 1990’s determining that critical thinking skills were being ignored in college accounting education. Accounting educators’ subsequent response was to include one mildly unstructured critical thinking problem at the end of each accounting textbook chapter, still preceded by dozens of highly structured procedural questions. Guided by this inference of relevance, it is not surprising that most accounting instructors still lean towards procedural content and not much has changed.

I learned that it pays to reexamine our discipline to determine if our focus is supported by research. I found that this might not be true in accounting education. Another interesting irony of the Accounting Education Change Commission study is that many of the commission’s recommendations aligned perfectly with the PCC core outcomes. I now have even more confidence in the wearability of PCC’s overall vision!

Do I understand the level of my students’ academic development? I reviewed the research subsequent to the Accounting Education Change Commission’s findings and learned more about student cognitive development. After this report was released, some college professors started introducing complex financial analysis projects at all levels of accounting courses. The problem with this quick fix was that most students cannot handle complex financial analysis until their senior year or graduate level studies. A tiered approach of first introducing students to concepts involving uncertainty, followed by increasing analytical tasks, was found to be more effective.

Do students need to know about all this? Students are the most significant stakeholders in the assessment movement, so it makes sense that they should be aware of what is happening. I was teaching a non-accounting general education (Introduction to Nonprofits) class at the same time I was taking the assessments class and spent some time discussing these concepts with my students. I was pleased and relieved to find that students were very interested in PCC’s core outcomes and wanted to understand meaningful ways to measure the quality and progress of their education. This desire confirmed a key assessments’ concept discussed in Sylvia and Shirlee’s class. Letter grades may stay around, but more of our effort needs to be focused on an assessment system that gives feedback and advice to students throughout the term. It should be just as much of a learning tool as their textbooks.

I have to admit that the above experiences unsettled my relatively peaceful teaching bubble, but my perspective has been broadened along with my courage to face the inevitable changes on the horizon. I remember when the Accounting Education Change Commission’s report was released about the poor quality of accounting education. I had just graduated from college and was facing the reality of paying my student loans. The thought occurred to me that I deserved a refund! As I face the future ramifications of “value-added” education, I hope I never forget my perspective as a primary stakeholder in education from 20 years ago.

Assessment: a blog post

by Andy Simon
September 2010



I haven’t been involved in the discussion of assessment for more than a year. I apologize if the issues I raise have already been heard and discussed. I confess that I am skeptical about the whole enterprise, and yet I know that many intelligent, aware, hard-working people–some of them my friends–have put in a great deal of time and effort examining proposals and strategies. I am not completely comfortable saying, in effect, to these folks, “your efforts are for naught, the project is inherently flawed.” For one thing, I may be wrong. Just on the other side of the next committee meeting a beautiful, effective, and efficient approach may be discovered or created. Yet, I was asked to write about my objections–qualms might be a better term. And so I shall.

The point of the project, I remember being told, is that taxpayers, politicians, and business leaders want to know that they are getting a good return for the money they invest in higher education. What they want to know–and what we must be able to show them–is that we are indeed enhancing students’ lives–adding value to them–by educating them.

My first observation about the project so described (and perhaps inaccurately so) is that we should be very careful about how we think about the relationship between those who sign the paychecks of teachers and those who receive the checks. John Dewey, a century ago, in defending the concept of academic freedom, offered a similar warning. He said, in effect, that although the trustees of a university are in an economic sense the employers of professors, the job of teaching is primarily to serve the public.

Nowadays, I see the idea of “the public” as a bit more problematic than Dewey did, and besides it might seem plausible (though I would say mistaken) to equate “the public” with the taxpayers. I would say that though we are employed by the taxpayers (in public education) we are responsible primarily to the generation we are educating and beyond that to future generations as well. For if we do not educate the current generation, who will there be to educate the generation that follows, and the generation after that?

This observation points to one of the problems with the very model the concept of adding value to students lives depends on. It is, after all, a rather industrial model–students’ lives are the raw material we work our magic on. They pass through our educational factories and, like raw steel turned into automobiles, their value is enhanced. But processing students’ lives can at best be only part of the function of our institutions of higher education. Another function–and a vital one–is to preserve some important things: ways of thinking, intellectual skills, bodies of knowledge, bodies of literature and the keys that unlock their meaning, bodies of art and the keys that unlock their meaning, historically significant works that inform our own culture and literature, and much, much more. How much value is added to our civilization because we can understand that the piles of stones that are scattered across Europe are the remains of a once vast empire, and because we can study the ideas that enabled that empire to rise and that caused it eventually to fall? Having that kind of knowledge adds at best a pittance of value to any one student’s life, but does that exhaust its value to our civilization?

It is important to note that higher education is one of the few extant institutions that predate the industrial revolution. It would be a grave mistake to force it into a industrial model merely because that is the only model we can think of. And yet, that is precisely what assessing education in terms of processing and credentializing students does. The success of the industrial model in many spheres of our lives has made us forget that there were–and hence are–other ways of thinking about education. One of them involves seeing education as a process of preparing and inducting students into a community–a community of scholars.

Clearly, there are problems with this model, too. One of them is that a closed community can perpetuate an unfair exclusivity. I was tempted to say that a pre-industrial approach to education saw the point as inducting students into a brotherhood, which would have been historically accurate as it reflects the way the “community of scholars” model has in the past excluded people who had a legitimate claim to be admitted. But the solution to such problems is not to throw out the model altogether but to try to ensure that there is open access to the community of the educated.

I can say quite explicitly what I think we must protect our institutions of higher education from: market forces. The threat to education that has existed at least as long as the advent of Capitalism has been enormously amplified by the computerization of our culture. Education has become a commodity–something to be bought and sold. That might not be so bad, but in our society, the only commodities that can be successfully bought and sold are ones that can be mass produced. In order to be mass produced, education must be standardized and homogenized. When I go to the store for, say, a pair of socks, the characteristics that make the socks I come home with unique an individual entities in the world are unimportant. What is important is that the socks I come home with are interchangeable with any of several dozen in the store I shopped in, any of tens of thousands in the stores the sock manufacturer supplies. That’s the way
it is with mass-produced commodities.

We are already well on our way to commodifying higher education. When we discuss the question whether college credits earned on-line are equivalent to credit earned in face-to-face classrooms we are addressing the interchangeability of our product. Of course, the answer to our question is obvious: if earning the credits in the two venues add equivalent value to our students’ lives, then they are clearly equivalent.

But if that is the approach to education that we want, why stop there? Why have professional teachers at all, with all their quirks and personalities? Why not have Harvard or Stanford develop the most effective on-line curricula (since massifying the classroom is impractical, though not impossible) as measured by value added to students’ lives, and then hire educational technicians to administer them? I’m being a bit facetious, but I really do fear that something like what I’ve described is the future for higher education: most students will enroll in institutions (note I didn’t say “attend”) where they will be offered the best on-line curricula the school system can afford. Only the elite, that is, the wealthy, will be able to afford colleges that offer face-to-face instruction from actual professors.

That poses a serious question: is there any real value in the uniqueness of face-to-face instruction by professional instructors? Maybe the answer is no. Maybe the individuality of instructors, their quirkiness, their personalities are irrelevant, or worse, impediments to the educational process. Obviously, I don’t think so. And I don’t think so because what I remember most about my undergraduate education is not the content of the classes but the inspiration I took away from many professors–and some of the quirkiest were the most inspiring. It may be just me, but I think inspiration is best transmitted face-to-face. I don’t think it comes across all that well through a computer screen. (I’m pretty old fashioned, but I believe that the immediate experience of presence is qualitatively different from any technology-mediated experience, no matter how life-like.)

Now that I’ve struck a personal note, let me pursue it a bit longer and then bring this long post to a close. I have no doubt that my education added a great deal of value to my life. Probably the most valuable contribution was never explicitly stated but implicit in everything I studied. My professors revealed to me the vast world of ideas–not specifically philosophical ideas, but the world of thought and knowledge. At some point I caught onto the notion that, if I played my cards right, I could spend the rest of my life studying anything at all that interested me, whether it be the sex lives of the Greeks (the revelation came to me while I was barely post-adolescent) or the musical traditions of the former Portuguese colonies, or anything else.

What a valuable lesson–to be shown to the vast continent of human knowledge and to be led to some of the points of entry to it. We have so many opportunities to enhance and enrich our students’ lives. We give students literature to read and insist on discussing it, and by so doing reveal to them the inner lives of other people, and by implication revealing their own inner lives, too. I feel some regret that throughout my teaching career I didn’t emphasize nearly enough to my students the importance of finding and creating beauty in their lives. I don’t think the intangible enhancements we have to offer are any less real because they are intangible, nor are they any less real because they are unmeasurable. I’m probably just tilting at windmills, but I think we have a responsibility to the future to do our best to protect education from the market forces that would industrialize it, and by so doing would destroy its humaneness and its ineffable value.