Project Summary To successfully extend the outcomes of an engineering education beyond "how to do" engineering to include "how to be" a good engineer, professional skills must be linked with being successful (career advancement). Creating and using industry-modeled performance reviews in team-based capstone projects enhances how students value and develop professional skills.

Context: Background and reasons for this inquiry This study started out as an exploration of the impact of the team context on integrative learning (and it will probably return to that eventually), but guided by my project group I refined my 2005-2006 Carnegie Project to more closely address my true educational passion. I want to craft a capstone design experience that is truly an "education of the whole person" in order to encourage engineering students to appreciate the importance of integrity and wholeness in life. I want them to see the profession of engineering as an opportunity to serve the world and support social justice, and I want them as individuals and as groups to "care as well as know". To support this goal my CASTL integrative learning project deals with integrating the non-technical but critically important "professional skills" into student conceptions of what it means to be both a good engineer as well as a successful engineer. There is support for this in the engineering profession. For example, the keynote address for the 2005 ASEE International Engineering Education Conference stated "Engineers need to not only solve the direct problem but also to understand its context. The awareness of the social responsibilities related to and required by the solution adds a complexity to the practice of engineering that was not there before." A simple way to state the essence of my project is to expand the student's developmental goals from "How 2 Do" engineering to "How 2 Be" a good engineer. Over the course of the year my Carnegie project had to be significantly refined as I realized there was no accepted method of defining what "professional skills" look like at different developmental levels and how they could be authentically assessed. These questions have to be answered before proceeding with an evaluation of activities that attempt to improve student achievement of those outcomes. Therefore, my immediate Carnegie project research tasks are: (1) Develop a taxonomy of professional skills outcomes for engineers that are understandable to undergraduate engineering students and that are linked to the ABET professional skills outcomes. (2) Develop an authentic way to get undergraduate engineering students to self-evaluate relative to professional skills and willingly engage in professional skills development. (3) Use the skills taxonomy and other tools to begin assessing the effectiveness of activities designed to help students develop an integral engineering identity (and achieve the professional skills outcomes). (4) Begin to investigate the impact of the team context and team identity on each individual's development of an engineering identity. Inspired by my project group, I have focused my effort on developing an industry-modeled performance review (PR) for capstone design courses as an authentic way to fulfill the first two research tasks. Tasks 3 and 4 will follow once the performance review process is refined and shown to be effective for this purpose. Professional Skills Outcomes List of ABET outcomes related to professional skills.

Questions What is? What is the current level (in terms of understanding and development) of professional skills within the Mechanical Engineering seniors at Ohio University, as evaluated through assessing their participation in the development of a professional skills performance review, as well as their use of the performance review to evaluate both themselves and their teammates. What Works? Does active participation in the development of a professional skills performance review and use of it for self-evaluation and team member evaluation lead to increased awareness of the importance of professional skills in being a good engineer?

Evidence of student learning The Project Methodology Overview (see link) identifies the various types of data collection methods used throughout the different project stages, but the main sources of evidence are: Early in the project > Student reflections on the characteristics of a "good engineer" used for formative evaluation (see link) Throughout the project > Observations and direct assessments of student project work, teamwork, leadership, communication, decision making, etc. > Performance Reviews (PR): * Input for developing the PR form from students (see links) * Student evaluations of the PR and ratings of importance of professional skills Middle of project > Written self and peer PR reports, and face-to-face performance review meetings. End of project > Survey responses, activity assessments, and agreements for participation in academic research. > Follow up on developmental needs identified in mid-project PR and reflections on professional skills. Project Methodology Overview Overview of project, especially with respect to the "What works" study of course activities. Good Engineer - Student input Summary of initial student reflections. Professional Skills Taxonomy - Performance Review Student generated professional skills list for use by them in evaluating their teammates and themselves Performance review Instructions The PR coversheet, use along with the Professional skills taxonomy for the PR.

Team Hopper showing the product of their integrative learning

Characteristics of a Professional

Findings: Results and interpretation of evidence Some conclusions that can be supported by the evidence already collected are: > The student input for developing the PR forms showed that most students seem to understand and appreciate at least some of the professional skills, and the skills they value appear to be strongly situationally dependent (those dealing with communication issues focused on those skills, those dealing with conflict mentioned those skills, etc.) 34/43 students suggested that each year students should have to modify or re-create the PR form rather than just using it, indicating that they valued the experience of creating the form. > The collection of professional skills proposed by the students, each described with 4 developmental levels, produced an impressive taxonomy that will be of interest to other engineering educators. > A complex project embedded in a social context is an important condition for successfully developing professional skills. The emphasis of the capstone project on the context of the energy situation and the need to be aware of societal impact was successful in raising awareness 88 percent of students reported increased understanding of the current energy situation 88 percent of students reported increased appreciation for the potential social and environmental impacts of engineering decisions > According to student feedback, instructor and outside expert observations, and direct assessment of student work, methods that worked well in linking professional skills to what it means to be a successful engineer include: a)having the ME advisory board share personal stories on "what it means to be a good engineer", b) continuously returning the focus of the design decisions to the overall context (including social and environmental impacts), c) continuous assessments and reflections on teamwork, d) continuous formal and informal technical communication (with prompt feedback), e) linking all project activities (including the PRs) to best practices in industry and showing how professional skills help engineers get jobs and advance in their careers. > Students viewed the capstone experience as an overall growth experience as evidenced by statements like: "More than an engineering project, this class has been a journey of personal growth that I have not taken for granted". > Although it is difficult to assess actual developmental levels for professional skills like "ethical perspective" (especially since behaviors are often situationally dependent) , it is possible to make a meaningful assessment relative to these skills by assessing student performance and engagement in performance reviews focused on these skills, by assessing: Ability to provide an example that demonstrates their use of a professional skill Ability to complete an honest and valid rating of themselves with respect to a professional skill (in comparison to the expectations of the profession Ability to describe and explain the reason for the expectations of the profession with respect to the professional skill Ability to design and implement a plan to bring individual performance in specific areas up to an acceptable professional level > Using this "process assessment" to assess this year's class indicates that student engagement in the process led to a high level of awareness of the importance of professional skills 39/48 students ranked good to excellent in their ability to provide a good example demonstrating the professional skills 75 percent of students were able to develop a plan to improve performance in an area where they self-evaluated the need for improvement A high percentage of students (>60 percent) appeared to have over-rated themselves with respect to at least some of the skills, likely due to their perception of how the ratings might affect their overall grade. Student Voice: "Coming into the course I thought the most important characteristic of an engineer was technical knowledge...I've learned that there are many skills that a "good engineer" must master. Also, I've learned the importance of self-assessment to midentify which of these skills I need to improve to become a "good engineer"

Challenges and pitfalls encountered The biggest challenge I faced in project development was the need to make the professional skills assessment authentic for engineering students. The PR-based approach that came out of the January residency works as an authentic method because it is a standard in industry. In the 360 degree PR process, students evaluate their peers and it is sometimes difficult to distinguish true ratings of professional skills from effort/ability/personality issues. Although the PR-based methods appear to be very successful within the capstone design course, I must explore ways to extend its influence throughout the ME curriculum - ideally professional skills will become a part of the ME culture as they are intentionally developed in context throughout the college experience. The face-to-face PR meetings have many benefits, but the time needed to do PR meetings with ~50 students is significant.

Summary The methodology of using industry-modeled performance reviews as an authentic way to develop and assess professional skills in undergraduate engineering programs seems to work well and has the potential to advance achievement and assessment of professional skills outcomes. Modeling the process used in industry to determine merit raises and career advancement seems to significantly increase the value students place on the process and the skills (since they see they are valued by potential employers), motivating the students to develop professional skills and see them as part of being a good engineer Involving the students in developing the PR and deciding how to implement it increased their level of commitment to the process. Also, strongly linking these activities to course grades significantly increased the effort put into these activities. Next Steps Many lessons were learned in developing the performance review skills list and implementing the performance review process with students, and I have generated a list of ideas and tasks that I will be working on to improve the effectiveness of this approach for next year.