INTRODUCTION From Spring Semester 2005 to Spring Semester 2006 the LMC Social Sciences Department conducted a three-part pilot research project assessing critical thinking skills in social science classes. Our first project activity focused on the first assessment criterion for the general education outcome of critical/creative thinking: 1. Students will analyze, synthesize and evaluate information. All four full-time instructors in the department participated in the project: Dr. Shalini Lugani in economics, Dr. Donald Kaiper in history, and Prof. Milton Clarke and Dr. David Zimny in political science. We were ably assisted and advised by TLP Co-Director Prof. Nancy Ybarra. The goals of the project were: 1. to identify a mode of critical thinking common to all three social science disciplines; 2. to develop a holistic rubric for assessing skills of critical thinking in this mode; 3. to formulate class assignments incorporating these skills; 4. to assess student competence in critical thinking by applying our rubric to student work in all three disciplines; and 5. to facilitate the introduction of critical thinking assessment into our classroom teaching.

A COMMON MODE OF CRITICAL THINKING Identifying a mode of critical thinking common to economics, history and political science proved more difficult than we had anticipated. After considerable (sometimes heated) discussion, we focused on public policy argument as a common element of all three disciplines. If the LMC general education curriculum in social sciences should promote any critical thinking skills, the ability to analyze and judge social and economic policy should surely be among them. This type of critical thinking may be distinguished from the strict scientific method. Problem solving in the "hard sciences" requires weighing evidence and reaching conclusions within an agreed set of assumptions forming a single theoretical model. Policy analysis in the social sciences requires reasoning within multiple, partially overlapping and partially contradictory systems of value-laden assumptions. The hard sciences produce knowledge in the form of "correct" answers; social sciences produce reasoned judgments that cannot meet the higher standard of objective "correctness." Nevertheless, we can distinguish better and worse policy arguments - arguments that exhibit lesser or greater competence in critical thinking.

A HOLISTIC RUBRIC FOR POLICY ARGUMENT In developing a holistic rubric for assessing critical skills in policy argument, we leaned heavily on the work of Richard Paul and Linda Elder at the Foundation for Critical Thinking. Adapting their list of the eight "elements of thought" to the special focus of public policy analysis, we identified three criteria for an effective policy argument. 1. An effective policy argument should clearly explain the reasons for accepting or rejecting a particular policy. A clear explanation of reasons should specify the assumptions on which the argument rests and state the thinker's point of view. An effective argument should analyze the policy's effects and focus on the most important features and values of the situation as determined by those assumptions and that point of view. The argument’s conclusions should be reached through logical inference and interpretation. 2. An effective policy argument should support its conclusions with adequate evidence. Evidence represents empirical information of many different types: numerical data, facts, observations, experiences and narratives. The thinker must select this evidence, describe it in sufficient detail, and demonstrate its relevance to the policy under consideration. 3. An effective policy argument should refute opposing arguments. Since policy arguments depend on conflicting and often incommensurable systems of assumptions, there is no such thing as an objectively "correct" or "incorrect" policy. It is therefore not enough to state only the reasons and evidence supporting the thinker's conclusions; an effective policy argument will deal explicitly with the most common arguments built on opposing assumptions and viewpoints, and incorporating different types of evidence. This refutation should identify flaws in the opposing arguments' assumptions, evidence and/or inference. Without refutation, a policy argument cannot be effective. These criteria are, of course, extremely general; they will generate different specific rubrics for different policy arguments.

SPRING SEMESTER 2005: THE SOCIAL SECURITY ASSIGNMENT After extensive discussion and debate, the four participants agreed that the best vehicle for cross-disciplinary assessment of the critical thinking skills required for policy argument was a common assignment for students in general-education classes in all three disciplines. The participants' diverse textbooks, course structures, and teaching techniques made it especially difficult to identify a single assignment for all our students, and several alternatives were discussed and rejected. Fortunately for the project and our sanity, fate intervened on February 2, 2005, in the form of President Bush's State of the Union Address. The portion of the Address devoted to his proposal for the radical reform of Social Security presented a concise but thorough policy argument, complete with explicit assumptions and values, supporting evidence and attempted refutation of opposing arguments. Since extensive analysis and criticism of the proposal were available in the mass media, we chose to concentrate on the third criterion for an effective policy argument: the refutation of opposing arguments. Our research question was: Given an excerpt from President Bush's State of the Union Address and several articles of argument and background information about Social Security reform, could students accurately state the President's policy argument and offer a refutation of that argument based on relevant evidence? Rather than add the assignment to existing course outlines, we "embedded" it in each instructor's existing course requirements as a replacement for other activities or as an extra-credit activity. Seventy-eight students in several sections of ECON 10 and POLSC 5 were supplied with a transcript of the relevant section of Bush's Address and a selection of readings on Social Security reform from the Washington Post. Each student completed a "take-home exam" paper of approximately 500 words, consisting of a one-paragraph summary of the president's argument and a hypothetical letter to the editor of a local paper disagreeing with Bush's proposal. The assignment required students to oppose the suggested reform of Social Security rather than allowing them to choose their own position on the issue; we believed that this more structured exercise would make the task of formulating and applying our first critical thinking rubric much easier than it might have been. The papers were assigned and collected at the end of February 2005, roughly halfway through the Spring Semester. Each student received a letter grade and course credit for the assignment in his or her course section; ungraded, anonymous copies of the essays were used for a holistic scoring session. The specific rubric we developed for scoring the papers differentiated among low, medium and high competency essays. Proficiency in refutation was defined as a holistic score of medium or high. Papers were assessed on their explanation of Bush's argument, and their analysis and critique of his supporting evidence, assumptions, values and proposed policy. The holistic assessment process began with a norming exercise to assure that all participants were applying the rubric consistently in scoring the papers. We carefully read and discussed "anchor" papers that exemplified arguments rated "low," "medium" and "high" according to the rubric. Each paper was then read and scored by two workshop participants. If they disagreed on the score, the paper was read and scored by a third reader. The results of our first pilot study served to establish a rough baseline estimate of the percentage of Social Science students proficient in writing a refutation of a public policy argument based on a supplied sample of data and argumentation. Final scoring of the papers placed 38% (30 students) in the "low" category, 54% (42 students) in the "medium" category, and 8% (6 students) in the "high" category. Since proficiency was defined to include students who scored at the medium or high levels, 62% of students were assessed as proficient. This first assessment was extremely limited in several ways. It measured only one aspect of policy argumentation: refuting opposing arguments. It did not give students the option of choosing their position on the president's proposal. Most important, it gave us no way to estimate the impact of a general education social science course on critical thinking, since it measured students' competency at only one point in time. We attempted to correct these problems the following semester in the next stage of our pilot study. Social Security Assignment Sheet and Information Packet This packet included several background and opinion articles from the WASHINGTON POST. Social Security Assignment Scoring Rubric

FALL SEMESTER 2005: THE MARIJUANA LEGALIZATION ASSIGNMENT Our second study once again focused on critical thinking skills in policy argument, but we broadened the focus of assessment from refutation to the construction of a complete argument, and from support for a specified conclusion to argumentation on both sides of an issue. We picked the legalization of marijuana as the policy to be discussed because of its topicality and interest to our students, as well as the availability of convincing arguments and relevant evidence on both sides of the issue. Once again we asked our students to limit the evidence they used in their papers, this time to the data available on a select set of web sites. Each student would be required to write two complete policy arguments of approximately 500 words each: the first in or near the third week of the semester either in favor of or opposed to the legalization of marijuana, as the student chose; the second in or near the thirteenth week of the semester on the opposite side of the issue. Our research question thus became: Given background information about the legalization of marijuana and relevant arguments by proponents and opponents of legalization, could students construct effective and complete policy arguments on each side of the issue? This "pre/post" research design would ideally give us a rough estimate of the impact of a single social science course on critical thinking skills. We could compare each student's first essay and second essay scores; higher competency on the second essay would indicate improvement that might be due to participation in the course. Of course, our research design rendered all such conclusions extremely tentative: we did not match students according to their relevant characteristics or control for confounding variables. In particular, students might choose to write their first essays on the side of the issue they personally agreed with, leaving the more difficult task of constructing an argument for the unappealing alternative until the end of the semester. This could lower scores for the second paper and mask any improvement in critical thinking over the ten weeks between essays. Even though we understood these weaknesses of the research design, we believed that a comparison of pre- and post-scores could still be heuristically useful. Our second holistic rubric was understandably more general than the first. It included the three criteria for a persuasive policy argument: the presentation of reasons for a position based on explicit assumptions, an articulated point of view, and logical inference; the selection and description of relevant evidence; and the refutation of opponent's assumptions, logic, and/or evidence. Once again, the rubric differentiated among low, medium and high competency essays. Proficiency in refutation was again defined as a holistic score of medium or high. Due to schedule constraints, only one instructor administered the assessment exercise in multiple sections of a course. Because we felt that a disproportionate number of essays from one instructor's sections would bias the results, a random sample of papers from this instructor's sections was chosen to approximate the number of papers submitted by the other three participants. This produced a final sample of twenty-eight pairs of papers - 9 pairs from one section of HIST 36, 10 pairs from five sections of POLSC 5, and 9 pairs from two sections of ECON 10 and ECON 5. Norming and scoring proceeded as for the first set of papers. As might be expected, it was difficult to apply a more general rubric consistently, and almost all the 58 papers required a third reader to resolve scoring discrepancies. As we had feared, 28.6% (8) of the 28 students were "backsliders": they scored lower on the second essay than on the first. 24.9% (7) of the students improved their rating on the second essay, and 46.5% (13) received the same rating on both essays. Twenty-two students, or 78.6%, were judged "proficient" on the first essay, and twenty students, or 71.4%, were judged proficient on the second. (See link below.) In retrospect, we believe that the second stage of our project was over-ambitious. The results were almost certainly skewed by our decision to allow students to choose the order in which they wrote the essays, and the sample size was too small for any detailed analysis. Furthermore, student opinion on the issue of legalizing marijuana is itself strongly skewed: eighteen of our twenty-eight sample students argued in favor of legalization in their first essay. We attempted to correct for these shortcomings in the third stage of our research. Marijuana Legalization Assignment Sheet Marijuana Legalization Assignment Information Sheet This sheet specified Web resources with background information and arguments on both sides of the issue. Marijuana Legalization Assignment Scoring Rubric Detailed Results of Marijuana Legalization Scoring

SPRING SEMESTER 2006: THE WAL-MART ASSIGNMENT Our third study, like our second, asked Social Sciences students to construct a complete political argument, this time on a practical policy issue somewhat closer to home. The Antioch Planning Commission and City Council had recently considered a proposal from Wal-Mart to build a Supercenter store on vacant land within the city limits. We posed the issue in hypothetical form: If the student had been hired as an economic development consultant to assist the City of Antioch in its decision making, would she recommend that the Wal-Mart proposal be accepted or rejected? The assignment was to write a 500-word consultant's report, culminating in a recommendation. The report needed to include the reasons for the author's position, evidence supporting that position, and a refutation of the other side of the issue. Our research question was: Given background information about the economic and social effects of Wal-Mart expansion and relevant arguments for and against building a Wal-Mart Supercenter in Antioch, could students construct an effective and complete policy argument for or against building the new store? The rubric for scoring this assignment was very similar to the rubric for the marijuana legalization assignment, with adjustments made for the particulars of the new issue. Once again we asked the student to draw his evidence from a limited base of data, this time a hardcopy packet of magazine and newspaper articles on both sides of the issue. Our sample included a total of 59 papers drawn from seven sections of POLSC 5 and one section each of POLSC 6, ECON 10, HIST 37 and HIST 38. All the papers were embedded in course requirements as regular credit activities with the exception of the essays from Prof. Clarke's two sections of POLSC 5, which were assigned as extra credit. Norming and scoring proceeded as usual, with one exception. Instead of assigning each essay to one of three categories, participants used a six-point scale, with scores of 1 and 2 corresponding to a "low" ranking, 3 and 4 to a "medium" ranking, and 5 and 6 to a "high" ranking. The new scoring system and the practice we had gained in ranking the previous essays greatly increased inter-rater reliability. Very few essays required a third reading. The results: 31% of our sample (18 students) were rated in the "low" category; 54% (32 students) were rated "medium"; and 15% (9 students) were rated "high." This means that 69% of the students in our sample were rated as "proficient." Below are links to a "high-rated" paper, a "medium-rated" paper and a "low-related" paper. Wal-Mart Assignment Sheet Wal-Mart Assignment Scoring Rubric

DIRECTIONS FOR FURTHER RESEARCH AND ACTION Since the Spring 2005 and Spring 2006 assessments were comparable in design, sample size and results, it seems reasonable to conclude that roughly 2/3 (66% of the students in our Social Sciences general education courses are able to construct a proficient argument on public policy issues; approximately 1/3 cannot. There may be some question as to what level of competency the College should aim for, but there is no question that we must improve the efficacy of our general education courses in instilling skills of critical/creative thinking. The next step for further research should be a well designed multivariate analysis of growth in critical thinking skills, incorporating a more sophisticated version of the pre/post assessment that we attempted in Fall Semester 2005. There are many exciting possibilities for experimental and quasi-experimental research worthy of publication in professional journals of postsecondary education. Our pilot research project has already produced a change in the participants' attitudes toward interdisciplinary assessment and professional collaboration. We are now planning to incorporate common critical thinking exercises in all our general education courses. In Fall Semester 2006, Social Sciences faculty will meet to begin work on program review and the development of program-level student learning outcomes and associated assessment instruments. Our pilot research will inform those efforts, and serve as a baseline for future action plans and ongoing instructional improvement.