NWCCU pressures us.... and who pressures NWCCU?

by Shirlee

Assessing student learning outcomes through PCC's SACs was put in place in response to pressure from our accrediting agency, The Northwest Commission on Colleges and Universities. The NWCCU upped the pressure on us in 2010 by saying that PCC was not in compliance with their standard on assessment for on-going improvement of teaching and learning. Many people here at PCC then stepped up to the plate in 2010-11 to create and implement assessment plans and to file reports on what they learned. For many faculty members, it was the first time accreditation had ever been directly considered.

Even though the pressure on us has lessened a bit, you still might be interested in knowing that the six regional accrediting agencies are feeling some pressure of their own. There are three separate bodies that each take a chunk of oversight of higher ed -- (1) the Federal government, through the Dept of Education, (2) state governments, who write and enforce a wide variety of requirements and standards, often directed to technical or vocational education and (3) the five regional accrediting agencies, including NWCCU,who require a routine of self-study and then provide peer review. These three bodies have intertwining connections. For example, eligibility for financial aid from the Feds is tied to attendance at a college or university accredited by one of the accrediting agencies.

It is this financial connection that is one prominent focus of a new advisory committee to the Dept of Education. In their report, the committee explores the reasons for and against severing this link between accreditation and financial aid. One possible future route they outline is to allow states to monitor educational quality -- reducing the power and role of both the federal Dept of Education and the accrediting agencies. Another possible future route puts more power into the Dept of Ed, and weakens the state role and the importance of the accrediting bodies.

The report is only 11 pages long, so you might just give in a glance. To get to the report, please navigate to insidehighered.com and then click on the hot line labeled "discussion draft." What gets decided is ultimately going to effect what we do here at PCC, so it might be of interest to you for that reason. But you might also find some sort of emotional resonance in seeing the agency that pressured us to hasten our assessment process now experiencing it's very own pressure from this advisory committee.

If you are a good and compassionate person, you may feel some sympathy for them. Or, if you are another kind of person, you may have another kind of reaction.....

Occupy and Higher Ed: The Cost Accountibility Movement

by Shirlee

The Occupy Wall Street movement has drawn attention to many problems that have been with us for a long time, but without achieving headline status until now. Wealth inequality and its effect on democracy, for example. The distribution of profits in the financial sector, pointing to the question about what is contributed to our collective well-being to justify such profits. And the increasing likelihood that the next generation of adults will not be able to achieve the same level of prosperity and security as their parents....

In connection with that last concern, Occupiers have been drawing attention to the major shift in the funding sources for higher ed over the past three decades -- from grants and scholarships for students, supplementing relatively generous public funding of the institutions themselves to increasing reliance on student debt. The accumulation of student debt is being likened to indentured servitude, even a kind of debt-slavery, by Occupiers. Average debt for students achieving a bachelor's degree is cited as between $23,000 and $25,000.-- and this at a time when unemployment is running high.

So why does higher education cost so much? Where does the money go? These are questions that belong in the Cost Accountability wing of the Accountability Movement. PCC's Learning Assessment Council exists in response to the accountability movement, and the effort to respond to the spiraling costs of Higher Ed has been part of our program from the start. As we here at PCC are busily assessing student learning outcomes, with an eye to making sure our students are learning what we promise to be teaching them, the plan is to locate and share around the instructional practices that are most effective. In this way we will be able to serve more students, more effectively, with less cost per student -- the 3-sided demand on Community Colleges set down by the Obama Administration. The hope is for the quality of our instruction to go up (as measured by achievement of learning outcomes) even as the cost of providing that quality goes down. This is our indirect response to the problem of student debt. Is it enough?

Inside Higher Ed has a piece on the Occupiers' anger about college debt. And in response to student agitation about college loan debt, the Obama Administration has posted a description of its plans. See: https://wwws.whitehouse.gov/

But if you are just plain interested in why Higher Ed costs so much these days, here is the source I recommend:
Delta Cost Project.

Here is a juicy tidbit from page 17:

Among public institutions, spending per student for instruction declined between 2002 and 2005, most dramatically in public community colleges. When state funds increased in 2006, instructional spending increased as well, but not enough to make up for losses in prior years.

Delta Cost Project has charts and graphs comparing costs and funding sources of community colleges to other sectors of Higher Ed. Student debt is increasing partly due to increased costs of education, but the larger factor is what they call "cost shifting" -- shifting from taxpayer funding to individual funding. The increased costs attributable to instructional versus non-instructional institutional costs are also charted. I think it is interesting reading.

We work all day, long and hard, on behalf of our students. It is important to me that we are not just saddling them with debt that will decrease the quality of their future lives -- instead of increasing the quality through all those intangibles associated with being an "educated person." For example, I try to imagine my life without my college experiences, and I cannot do it. College set me on my life's path, and made me into the person I think of as myself. But I graduated with no debt at all. I paid my way by working, with the help of grants and scholarships. That is near impossible these days... Would I be so pleased with my education if it had taken me 30 years to pay it off?

I fear that higher ed looks different with a monthly debt payment that appears to go on forever, and with no good job prospect in sight. I think as educators, we need to know where the money goes -- the money that is at least in part being spent today on our paychecks, but will be paid back by our students long into their futures. And we need to know that it is not just knowledge and skill we are providing our students.... it is years of a repayment plan, too.

Grading Gets Outsourced to India

by Shirlee

The way PCC is approaching the new demand for assessment in Higher Education is only one of several different models that have been adopted across the landscape of colleges and universities in the U.S. These differences became apparent as members of the Learning Assessment Council started to scout around in our first year of existence (2008-9), which was provided to us as a "year of inquiry." Here are short descriptions of some of the ways colleges and universities have decided to go:

1) Many institutions have adopted a single high-stakes standardized test, designed to measure core learning outcomes like communication and critical thinking. These are usually "value added" measurements, intended to show how much development there has been in student competency from the first term of entrance to an exit with a degree. These tests allow comparison of one institution with another, which is part of the call for "accountability" mentioned in last week's blog.

This first approach is based on an assumption that assessment of student learning is a separate kind of activity from instruction. Teachers may be skilled at teaching, according to this thinking, but they are not experts in measuring learning. Measurement expertise is called psychometrics. Experts provide the design and on-going re-design of the major competing high-stakes tests used by colleges and universities in the first model.

(2) In the second model, the idea continues to be that we should leave assessing of learning to the assessment experts, in order to disrupt teachers' lives the least amount possible. In this case, though, assessments are customized to different SACs or departments via the work of a team of psychometricians who are called in to (i) interview faculty about their specific student learning outcomes, and then (ii) design customized assessments to be used by all instructors in that subject area. In this way, for example, experts might come and consult with the PCC history SAC to determine what "critical thinking" means in the area of history, across PCC's history curriculum. Then all the instructors of a given section of history class would be required to administer the test the psychometricians came up with, and the results would be examined to see what they say about the effectiveness of history instruction at PCC. This model gives assessment results that can be used for continual program improvement (the other main purpose of assessment, as mentioned in last weeks blog). The major company that has emerged to do business in this second model is EduMetry.

(3) Many institutions created a new administrative office, and put someone in charge of organizing faculty assessment work. Often, this office oversees the adoption of an expensive assessment software system, and then trains faculty (usually department chairs or supervisors) in how to use it. The software system ensures consistency of reporting, and eases bundling of assessment results for display to the accrediting bodies. For this approach to work, the administrator has to have the power to compel reluctant faculty to both do assessments and then learn how to report results using the system. Faculty are involved to a greater extent in assessing than in either of the first two models, but they tend to be viewed by administrators as reluctant participants likely to drag their feet....

(4) Some colleges and universities have decided that assessing is a critical component of the instructional process, and must be kept as part of the bundle of teaching tasks. The idea here is that faculty are deeply invested in successful student learning, and when they see the connection between assessment and improved learning outcomes, they will embrace assessment as a new and useful tool for doing their important jobs even better. This model then leaves assessment in the hands of faculty, in the form of an assessment committee or council. PCC was set on this path through the recommendation to the college made by the faculty Learning Assessment Council that program/discipline assessment be the responsibility of SACs, and implemented as an ongoing component of Program Review. This last model is the only one that is fully respectful of faculty professionalism and expertise....

The national body for the union that represents PCC's instructors and APs has endorsed this last model, coming up with an interesting slogan that in higher ed we should count what counts. I remain deeply convinced that this last model is the best both for students and for teachers, in the long term. But I am also aware that some faculty at PCC would have picked one of the other models, had they had the choice. And I often call to mind a participant in one of our first assessment classes who voiced a very strong positive response to the second model above, and the company that is most successful in that endeavor, EduMetry.

EduMetry has a varied approach to assessment activities in Higher Ed, and I recently came across another aspect of their business plan in the Chronicle of Higher Education. EduMetry has started outsourcing grading to India through their program called Virtual TA. (See http://chronicle.com/article/Outsourced-Grading-With/64954/) In this part of their business, they devise rubrics for assignments, train and norm a group of assessors on use of the rubric, and then ask their assessors to provide detailed, rich feedback on student papers -- feedback of the sort we all might dream of providing, but are often too busy to actually do. One sociology instructor at a community college, is quoted in the article.

And although Ms. Suarez initially was wary of Virtual-TA—"I thought I was being replaced"—she can now see its advantages, she says. "Students are getting expert advice on how to write better, and I get the chance to really focus on instruction."

It is a new world of assessment in Higher Education. With so many things changing so rapidly, and with many different kinds of responses to the changes being pioneered at different institutions, tuning in to assessment news provides lots of surprises. I used to think that education was a service that couldn't be outsourced. But EduMetry has surprised me. The logic of it is just an extension of the thinking that leads to the first two assessment models I described above -- if teachers are experts at teaching and psychometricians are experts at assessing, and we should each do what we are experts in, then assessing should be peeled off from the work of instructors and handed over to someone else.....

I heard an instructor say the other day that grading was the least satisfying part of his job, and he wished he could teach without having to grade. I wonder if he would really be so happy if EduMetry granted his wish.... Instead of doing more assessing, like we have asked instructors to do at PCC, the day may come when we will do no assessing at all. In the back of my mind, I can hear David Rives (president of Oregon AFT) talk about the de-skilling of the instructor's job...

I say that perhaps we should be careful what we wish for....

Two theories of Assessment

by Shirlee

As assessment has become ever-more-prominent in education and the non-profit world -- a trend that has been building for the past 25 years, according to my reading -- two distinct sets of reasons for assessing have been given. Often they live side-by-side in uneasy alliance. But they are very different, and that difference could make a BIG difference for faculty lives.

Assessment Theory #1 -- ACCOUNTABILITY

The problem assessment is supposed to solve:
Lots of money goes into education and non-profits, and it is hard to tell if it is well-spent or wasted.

The thinking: Lots of money, for example, is given to feed the hungry. Is it being spent well? Are there fewer hungry people than there would have been without the spending? Could it have been spent more effectively? It is hard to say.... Education is in this same boat -- lots of money gets spent, but how can we tell if it is being spent effectively? In this thinking, this aspect of the non-profit/education world contrasts sharply with commercial enterprises. That is, in a market environment, there is a straightforward way to tell if an investment was a good one from a business point of view -- did it lead to the creation and selling of a good or service that enough people wanted to buy so as to make a profit? In a market, it is possible to compare investments over time, using the metric of return on investment, to judge whether or not money was well-spent. But in education, as in most of the not-for-profit world, it is harder to tell what is "working" and nearly impossible to tell if one kind of investment in education -- for example, college scholarships to kids from poorer families -- yields a better or worse return on investment than another strategy -- for example, full-day pre-kindergarten for kids based on income eligibility.

In this context, assessment is intended to provide metrics that enable valid comparison of one possible use of funds with other possible uses. In the world of higher education, this leads to the call for standardized exams of basic skills -- usually writing and critical thinking -- to be given to all students (or a representative sample of all students), across all schools. The scores on these standardized tests would allow easy and clear comparisons of one institution with another. Indeed, there are several exams that are competing to fulfill this dream, such as the CLA (or for community colleges, CCLA) or the ACT CAPP. Lots of colleges and universities have responded to the call for assessment using this theory of assessment by mandating the use of one of these tests.... After the admistration gets the results, they then inform faculty of how well (or badly) they are doing. Poor results lead to lots of administration pressure on under-performing faculty.

Assessment Theory #2 -- CONTINUAL IMPROVEMENT

The problem assessment is supposed to solve:
Our world is in dire need of the skills and competencies characteristic of educated people -- primarily communication, collaboration, and critical thinking skills -- and this mission is so important, we must devise ways to identify and quickly roll-out best practices.

The thinking: teachers tend to be intrinsically motivated, curious innovators. However, they have historically worked in great isolation. One teacher might informally track what works and doesn't work in his/her classroom -- and make changes based on observations -- but this is primarily a matter of independent practice. Education is so important in our changed world, though, we have to take these kinds of individual practices and formalize them. In this way, assessments allow us to formally track the results of innovations, and more quickly and easily isolate components of the most effective education practices, so they can be shared more widely.

In this context, assessments must be in the control of faculty members, as it is their familiarity with their student population, curriculum, content, and current practices that leads to the determination of what to assess. Standardized assessments for a general population of students will not help (for example) history faculty to discern best practices for critical thinking in history. Faculty must control the assessment process, because they have the detailed familiarity to know how to focus an assessment.


At PCC, we have been firmly within the fold of Assessment Theory #2. It is, after all, what our accreditors have asked of us -- evidence that assessment of student learning is being used to improve both teaching and learning. It is also the process that is most respectful of faculty, so it is not suprising that a faculty Council would come up with this sort of a recommendation. Additionally, it is the only approach that could lead to results that would actually be useful to instructor practice.

Notice, however, that Theory #2 leads to ever more customized and distinct assessments, while Theory #1 leads to ever more generalized and standardized assessments. These two theories lead to incompatible pictures of what GOOD ASSESSMENT looks like. The more locally useful a particular assessment is to a given SAC, the less useful it will be to compare one college to another....

I mention all this because one of my heroines in the Assessment World is Trudy W. Banta, editor of Assessment Update: Progress, Trends, and Practices in Higher Education. In the most recent edition (Sept-Oct 2011) she has written a piece warning that "... the promise of assessment for improvement might be diminished by increased focus on assessment for accountability."

She writes:

...[A] significant portion of US colleges and universities may be moving in the direction of providing to the public information based on scores of standardized tests of generic skills that inevitably will be used to compare quality of institutions. It just seems to be human nature to hone in on those easy numbers when we seek a standard for making comparisons."

I offer two thoughts in this context:
(1) We, here at PCC, are lucky to have both an administration AND an accrediting agency operating from Assessment Theory #2. This is the approach that fully respects faculty as THE key players in driving program improvement. Although assessment takes up hours and energy, when done by faculty (and done well) it leads to results that make a difference in student lives -- it leads to continual program improvement and better learning outcomes.

(2) While we here at PCC are getting better and more proficent at assessing our programs and disciplines -- witness the amazing variety and innovation of assessment approaches across our SACs, with ever-better strategies and instruments -- it simply may not be enough to stave off the rush to standardization. At this point, there is still a national push for assessments that fit the picture from Theory #1. Assessing to compare one institution with another is very, very, very different from assessing to be able to do our important job ever better. While both are part of the rise of assessment, they lead to very,very,very different pictures of what good assessment looks like. A one-size-fits-all test looks ridiculous on one model, and the only thing that will work on the other....


Wait. I really meant:
very, very, very, very different.....

I'll keep an eye out and let you know what I see on the standardization horizon. Until then, we will continue with our PCC plan of asking for a splendid locally-controlled profusion of SAC-specific assessment!!