Bridging diverse institutions, multiple engineering departments, and industry: A case study in assessment planning

McMartin, Flora

ABSTRACT

As ABET and regional accreditation agencies focus their attention on student learning, effective planning processes and practices become critical to the departments and colleges that must initiate them. The lessons learned from the Synthesis Coalition's experience can be applied at the departmental level to meet the requirements of these agencies. The planning strategy focused on a collaborative process that involved all stakeholders: faculty, administrators, students, and industry representatives. The stakeholders validated and prioritized the Coalition's original goals and identified a specific set of abilities, criteria, and activities associated with them. (The goals included, to facilitate: communication, teamwork, hands-on facility with hardware, awareness of the social implications of engineering, modern industry practices, multi-disciplinary design, and open-ended problem solving.) The Coalition's industrial board provided written scenarios of "day in the life" challenges engineers face to help ensure that the values and goals the Coalition planned to assess matched those that industry felt were important. Next, course documents were analyzed to identify a set of assignments to use for developing authentic, performance-based assessment tools. This approach assured that by participating in the Synthesis Coalition assessment project, each campus prepared a core group of faculty to assess (and train other faculty to assess) student learning and ultimately document evidence of student learning for ABET accreditation.

I. INTRODUCTION

What is assessment?

Why do we need to assess?

How much time will I have to take to participate in this?

What are student learning outcomes?

What will I get from doing this?

How are we going to convince faculty to take assessment seriously?

How do we assess process rather than content based leaming?

I know what I am doing works; why should I do assessment?

These are the kinds of questions being asked at engineering departments across the United States as educators try to plan and implement assessment programs. These questions are real, they are thorny, they are messy. They challenge us to create programs that respond to the particular needs of our faculty and our schools.

Multiply these questions by seven diverse campuses (California Polytechnic State University, Hampton University, Iowa State University, Southern University, Stanford University, Tuskegee University, and the University of California at Berkeley) and throw in faculty from several engineering disciplines, and then scatter the participating universities across the country from coast to coast, and you get the complexity facing the NSF-supported Synthesis Coalition* as they developed their assessment process. This article will describe the lessons learned as a result of devising an assessment plan at the coalition level, and how departmental or college level assessment planners might apply them.

While the task of reaching agreement among faculty on curriculum matters at the department level is daunting, for the Synthesis Coalition, even the basic question of what abilities to assess required agreement among 40 faculty from six different engineering disciplines at universities with very different missions. Given that there are no common courses taught among these schools, consensus about what learning outcomes are important to assess had to be developed through a dialog generating a common, negotiated understanding among the participants.l In addition to generating consensus, creating faculty commitment to an ongoing collaborative assessment process was also the major challenge facing the Synthesis Coalition as it developed an assessment plan to evaluate the effects of seven years of the Coalition's diffuse reforms. The process the Synthesis Coalition created to address these challenges and ultimately to craft a process of ongoing and performance-based assessment,23 was guided by principles of teamwork, collaboration, and faculty/industry interaction-all of which are important to devising an assessment plan regardless of the level (course, major, or department) or complexity (one college or coalitions of colleges). The focus of this article will be to detail this process at the coalition level and draw connections to the major and department levels. II. CHALLENGE: TO CREATE FACULTY AND INSTITUTIONAL BUY-IN AND COMMITMENT TO ASSESSMENT

In both program evaluation and assessment, collaboration of stakeholders in the process is critical to success.45 For the Synthesis Coalition this meant collaboration between students, faculty and administrators, and representatives of the industries who ultimately employ engineering graduates. In order to ensure the stakeholders' commitment to the assessment process they were given clearly defined roles that kept them actively involved from the initial planning stages of the project to the final analysis of the data.

A. Faculty

Many of the Synthesis Coalition's faculty were already instituting small scale classroom research projects,?' and some had been involved with in-depth course level assessment projects involving, for example, video analysis,8 in-depth questionnaires,9 Internet kiosk projects,lo and evaluating spatial reasoning skills.11 However, because assessment at the broader coalition level was a new step for a majority of the participants, uncertainty remained regarding how committed faculty would be to the assessment planning process. This was a critical concern since research on faculty involvement in assessment planning 12 offers strong evidence that the role of faculty, in this case, the Synthesis project leaders, is essential to developing a successful assessment program.

Based on their comments, it was clear that the faculty project leaders' perceptions about the level of work involved, the validity of the assessment instruments and tools, and the degree of consensus on the plan were factors that would govern their commitment to the process in ways that could either kill the plan or energize it.

In addition to project leaders who were experienced in assessment, several of the faculty held dual roles as campus administrators, i.e., they were associate or assistant deans. These faculty were critical to the planning process for they were most familiar with the new ABET requirements. It was crucial to the process that these faculty administrators know how Synthesis assessment efforts could parallel and, in some cases, model effective assessment processes that could meet ABET criteria. Being new to the criteria and to implementing outcomes assessment, it was essential that their input be included in the planning process.

In an effort to build a process of involvement that met faculty needs, the assessment team (a small group of assessment specialists and representatives of the stakeholders) began to collect project leaders' questions and concerns about assessment using Angelo and Cross's"3 classroom assessment technique "the muddiest point"* at each project leader meeting. This is an example of a technique that allowed faculty to provide input in their areas of expertise but did not create an undue burden on their workload. Faculty responses to the muddiest point provided valuable insights which both framed the planning process and the way it was communicated to the participants.

B. Students

In addition to faculty, students have a critical role in identifying what is important to assess in engineering education" and how to assess it. A student advisory board was instituted, and for the first time, undergraduates and graduate students met at project leader meetings. After being oriented to the goals of the Synthesis Coalition, students organized the Synthesis Student Advisory Committee (SAC). This committee gave students a more formal voice and role in decision making.

In addition to their role on the SAC, students met with and worked alongside faculty and industry representatives at brainstorming, negotiation, and feedback sessions at project leader meetings. The students also worked with the task force charged with designing and recommending the formal assessment plan. Their participation in these sessions grounded the assessment plan in the daily reality of student life and guaranteed that students' concerns were incorporated into the plan.

C. Industry

Industry involvement required a different set of tactics. In 1995 the Synthesis Coalition formed its Mechatronics Industrial Board (MIB). The primary purpose of this board was to provide industry input to the shape of curriculum reforms promoted by the Synthesis Coalition. The members of this board quickly volunteered to go beyond this initial advisory role, and developed scenarios that described events in the "day in the life" of an engineer. Initially these scenarios were to support design projects by providing a window into the realities of engineering practice. However, after reviewing several of the scenarios, it was clear that they were so "rich" in describing real life engineering situations that they could form the basis of assessment tools. This ushered in a more active role for the MIB in assessment and quickly created a meaningful role for the industry representatives in the assessment process - a role they energetically embraced.

Clearly established roles for faculty, students, and industry were essential to gaining buy-in to the planning process. While slightly more complex to arrange at a coalition level, they are equally important at the departmental leveL Only by the participation of the primary stakeholders can assessment planners ensure that what is valued in engineering education is assessed.15

III. CHALLENGE: CREATING CONSENSUS ON WHAT TO ASSESS

In January of 1995 the Synthesis Coalition hosted a meeting of its project leaders, students, and industrial board members to kickoff the assessment planning process. This was the first time these three groups had met at the same place and time. Together, they began to negotiate common understandings, explore their different perceptions about engineering education, and recognize how and where their worlds intersected in this setting. For instance, one MIB member argued that engineering education needed to better prepare students for the ambiguity of working without clear standards or rewards. Faculty responded that if they tried to put students into such an ambiguous situation, students would not react well because they would not know what was expected of them or "how to get an A." With a little prompting, this kind of informal discussion focused around the seven qualities associated with Synthesis reform efforts: open ended problem solving, communication, multi-disciplinary design, teamwork, hands-on facility with hardware, integrating industry practices in the classroom, and attention to the social, technical, and economic impact of engineering.

At this point, the participants were engaged in a modified version of the assessment planning process recommended by Rogers and Sando.16 In order to articulate the seven general Synthesis Coalitions qualities more explicitly, the participants brainstormed responses to statements based on the qualities such as: "An engineer who can communicate effectively has the ability to

." Once the list of these kinds of abilities was generated, the group brainstormed the characteristics (criteria) that are examined in industry and in the classroom to determine if an engineer or student possesses that ability. Finally, the group identified classroom practices that allow students to demonstrate those characteristics and abilities. During the course of these meetings over 500 abilities, characteristics, and activities were identified.

To begin narrowing the list towards a manageable number of outcomes to assess, the faculty listed the top three qualities on which they wished to have their students evaluated. The vote showed clearly a consensus around: communication, open ended problem solving, multi-disciplinary design, hands-on facility with hardware, and teamwork. After more discussion and negotiation with students and the industry representatives, it was agreed to focus student learning assessment around these five learning goals.

An assessment task force was formed to work on the (raw) brainstormed material. In addition to the Assessment Director, the task force was made up of two faculty members, an industry representative, and two students. This task force reviewed the ideas generated by the larger group and refined them into a smaller, focused set of approximately 30 assessable learning outcomes. First, the task force took a straw vote to identify the five to ten most important abilities identified at the project leader meeting (e.g., teamwork or open ended problem solving). This accomplished, more negotiation and some additional brainstorming were necessary to finalize the list.

The result of the task force's negotiations was such a rich product for two reasons: it was based on the insights generated by the larger group and it included representatives of all the Synthesis Coalition's stakeholders. The task force refined the list of qualities into measurable goals, identified a list of abilities associated with those goals, and selected criteria by which a student could demonstrate competence of those abilities-this list became known as the Synthesis Assessment Framework. Last, the task force generated possible measures for each criterion to serve as the foundation for the rubrics to be used to evaluate a student's ability level given the classroom activities and criteria associated with it.

In parallel with the task force's activities, the approximately 40 scenarios written by members of the MIB were analyzed by a second team of assessment staff to extract a list of abilities and actions that an engineer would employ to resolve these "real-life" situations. This list of industry activities/abilities was then compared to the Synthesis Assessment Framework for validation. A rough cross comparison between the two lists showed a high (approximately 80%) match between activities/abilities required by the industry scenarios and those identified for assessment in Synthesis engineering courses.

The Synthesis Assessment Framework, a comprehensive list (see Table 1.) of what would be assessed in terms of student learning,* in draft form, was sent to each stakeholder group for review and comment. Once the review period was completed, the list was revised based on the comments received. Finally, after nearly five months of negotiation and refinement, the Synthesis Coalition Board of Directors approved the final Synthesis Assessment Framework, setting the stage for the next phase of the planning period: deciding how to assess these abilities.

Coming to consensus around what to assess is a time consuming but critical first step in initiating assessment. Once the faculty and students at a particular level (e.g., major or department) agree on the most important abilities to assess they should be compared with those of the other stakeholders (e.g., industry, other integral college departments) to make sure that what is being assessed is valued by all. Including stakeholders in defining what to assess strengthens each participant's accountability for what is assessed, taught, and learned, and builds ownership in and responsibility for the assessment process.

IV. CHALLENGE: CREATING CONSENSUS ON HOW TO ASSESS

The process for coming to consensus on how to assess the abilities identified in the first phase of the planning process employed a different set of strategies including new roles for faculty, students, and the MIB. Phase I depended upon the participants to provide information and expertise via a brainstorming and negotiation process. The participants generated information, which the task force organized and focused. Phase II required that the generative work be done by the Synthesis assessment team and that the stakeholders be the arbiters and editors of that work.

A primary goal of Phase I was to create trust between the assessment team and the participants, in addition to building common agreement around the abilities to be measured. Throughout the process, the assessment team communicated regularly with project leaders and the Synthesis Coalition's assessment coordinator on each campus to ensure that the participants stayed involved in the next stages of the process and that they saw their input included in the changes and revisions to the final product. The team's milestones and completed deliverables were communicated to all participants. This attention to detail built trust in the ability of the assessment team to accomplish what they had promised, and was a critical accomplishment in setting the stage for the team to move into the lead in the planning process for Phase II.

Phase II began with a search for the exact assessment questions the team would try to answer. The overall assessment question was deceptively simple: What are the effects of Synthesis reforms? Three sub-questions emerged from that central question and framed the approach for assessing the Synthesis Coalition's effects on the abilities defined in Phase I:

What are the institutional effects of Synthesis reforms?

What effect has Synthesis had on pedagogy and student learning in individual courses?

What are the cumulative effects of Synthesis on student outcomes at graduation?

The process outlined in this naser touches on elements of questions one and three but is primarily concerned with question two regarding pedagogy and student learning.

The learning outcomes identified in Phase I were predominantly higher learning, or process-based" outcomes, factors that were extremely important in identifying or developing assessment tools. For example, one set of abilities focused on the process of solving open-ended problems which require significant skills in evaluation, analysis, and the synthesis of information. These were not specific content-based outcomes, (e.g., they met ABET criteria for teamwork and communication skills but did not meet the criteria for knowledge of mathematics and science.) For this reason, the assessment team rejected the use of standardized tests and instead focused on developing authentic or performancebased assessment tools-tools that would allow students to demonstrate higher learning and process-based learning most effectively.18

An analysis of course syllabi of Synthesis Coalition courses produced a set of classroom assignments (e.g., project reports, project presentations, and demonstrations) that along with the ability framework became criteria for identifying possible assessment tools. A range of tools that related to existing classroom activities and covered a broad range of abilities were identified. Next the team negotiated a coverage grid associating tools with the particular abilities they could assess. (See Table 2. Map of Assessment Tools to Student Learning Abilities.)

Four tools: scenario-based assignments, design project reports, self and peer reports, and taped observations were selected to assess learning at both the course and Synthesis Coalition levels.

Student portfolios, focus groups, and interviews were also identified as important tools, but were judged too time intensive to be used as broadly as the core tools.* Once the four core assessment tools were selected, the assessment team researched various formats to maximize their validity and limit any disruption to classroom activities.

Scenario based assignments associated with learning outcomes for open-ended problem solving, multi-disciplinarity, and hands-on learning, were designed to be short openended problems based on the scenarios provided by the MIB. These assignments were designed as homework problems to be administered in the first and last week of a semester or quarter. (Several versions of the writing prompts for the scenario assignments are currently being pilot tested to determine the optimal format.)

Design project reports provided the broadest coverage of the abilities, and required the least additional effort by faculty. These reports were based on sample assignments provided by project leaders. (The faculty participated in finalizing a common format for these assignments at the project leader meeting in the summer of 1997, and it is being pilot tested this fall.)

Self-peer reports ask students to assess their own performance and that of their peers. These assessments focus primarily on teamwork skills, but instruments were developed to other abilities, e.g., hands on learning, which could be used alone or (preferably) in combination with design project reports or taped observations.

Taped observations encompass both audio and video taping but focus on two separate student abilities. Video taped observations were designed to capture student oral presentation abilities. This requires a straightforward process of taping and evaluating student presentations of (primarily) final projects. The analysis of teamwork by way of audio tape required the development of a grading rubric to capture the essence of student abilities through the linguistic content of team discussions. The utility of this approach -including the type of analysis and sampling needed to draw reliable conclusions from the tapes -is currently being tested.

To date, the results of the limited pilot testing is inconclusive. Scenario assignments and audio taped team meetings have been tested at two different schools on freshman and sophomores. Industry representatives have completed several of the assignments, and their responses will be used to validate the scoring rubric for the tool and provide baseline data against which to assess student abilities. Final revisions to the tools will be made in the spring of 1998 based on the results of the fall pilot tests for each tool.

The core assessment tools will tell us what happened in terms of student ability development, but not why. To uncover the elements of these courses that are most effective in facilitating student development and how they influence student attitudes, a small sample of students will be recruited on each campus to create student portfolios and participate in focus groups, interviews and attitudinal surveys.

While the process for creating consensus on how to assess was different from that used in Phase I, some of the principles remained the same. First, faculty and administrators, students, and the MIB were involved on a regular basis. Each participant was contacted individually in order to get his or her input regarding course information. All work by the team was public; during this phase, a web site was created (www.synthesis.org/assessment/ Assessment.html) and the working drafts were posted for review and comment. Where possible, each campus was visited by a member of the assessment team to review the proposed tools and processes. When such a visit was not possible, the campus assessment coordinators met and reviewed the proposed tools and processes. They passed on the information to the participating faculty and administrators on their campuses.

The culmination of the work for Phase II came at a project leader meeting where the tools and process were formally presented to the Coalition. This meeting provided the participants an opportunity to review the process, discuss the plan, experience the assessment tools, and make recommendations for change. As always, this input was incorporated in the tools before initiating formal pilot testing.

The selection or development of appropriate assessment tools and processes, regardless of whether assessment is implemented at the course, department, or coalition level, must first be based on well defined and agreed upon student learning outcomes. Once articulated, faculty can more readily identify existing classroom practices that might be useful for assessment in addition to, or in lieu of, standardized testing. Assessment must be matched to the level of learning, e.g., multiple choice tests are unlikely to be good measures of complex learning processes.

Assessment tools and processes need to be tested in the circumstances where they will be used. In the case of the Synthesis Coalition, the results from each campus will be examined to check for bias that might result from manner of administration, self selection based on school population, and so on. Pilot testing serves multiple purposes: first, testing insures the assessment tools appropriately measure student learning, next they provide actual data and they help train faculty how to administer tools and analyze assessment results; and last, pilot tests are another means for collecting feedback about the process as a whole and how it might be improved or streamlined.

V. CHALLENGE: DETERMINING WHO AND WHICH COURSES TO ASSESS

In any one year nearly 5,000 students are enrolled in over 150 Synthesis-related courses at the seven campuses. A critical aspect of the planning process was developing a sample that would be both manageable and sufficient to answer our assessment questions. This process began by putting all of the degree programs into commonly formatted "Degree Maps." These maps arrange a program's required courses by grade (i.e., freshmen, sophomore, etc.) and identify the technical electives which satisfy a major's requirements. Within the maps, Synthesis courses were highlighted to provide the path of Synthesis related experience for each major. This map of courses was used to identify the level to which Synthesis courses had been integrated into the curriculum, types of electives, and grade level of courses that could be included in the Synthesis assessment plan. Maps also graphically illustrated the impact of Synthesis courses on a major based on a student's course selection.

For each course identified in the degree map, project leaders were asked to rate its emphasis on each assessed student learning ability and provide a description of classroom activities that supported that learning. The resulting course profiles produced a picture of the courses as they related to the student learning outcomes identified in Phase I.

In addition to the course profiles, a student survey provided another perspective on the level of emphasis of each course on the Synthesis abilities. Student surveys have long been used by the Synthesis Coalition-originally, they attempted to gauge student learning. During the 1996-1997 academic year the survey was converted to ask both faculty and students to describe their experience in a course (e.g., how much teamwork was included in the course?).

The results from this survey were related to the course profiles in a comprehensive matrix created for each degree program. This matrix identified each of the Synthesis Coalition's courses and listed the student/faculty emphasis ratings for each of the learning goals. Teams of project leaders from each campus used these matrices to identify the level of emphasis on particular abilities in each course. Faculty then decided which of the courses would be assessed, and assigned specific core learning tools that best fit those courses and abilities. By the time the campus teams had completed this exercise, they had developed a plan for assessment in their degree programs. (See Table 3.)

Campus control of this aspect of the process was critical to gaining consensus on the assessment plan. The faculty identified in these plans have committed to the ongoing use of the tools. This commitment came only as the participants began to see what they would gain from the assessment process.

At a departmental level, degree mapping is critical to deciding which courses provide good opportunities for assessment. Degree maps are tools that clearly illustrate how a curriculum works together (or doesn't work together) to accomplish the goals of a major or department. They provide departments with an opportunity to adjust courses that are not aligned with over-all department or college goals prior to assessing student learning. The mapping process also helps to guide the final selection of tools, timing of specific assessments, and the total assessment experience for each student. The last is particularly important in protecting students from being over-assessed.

VI. CHALLENGE: MAKING THE PLAN REALIMPLEMENTATION

Three factors are critical to the successful implementation of the assessment plans: provide faculty with sufficient planning time as well as time to learn about assessment, link the plan to other users (in this case ABET), and involve key campus administrators in the assessment process.

For the Synthesis Coalition, the most time consuming aspect of the process was developing consensus on the specific abilities it should assess. The assessment team's strategy was to focus participants' efforts around specific decision points: the who, what, how, and why of the plan. Early in the process these decision points were clearly outlined and communicated to the participating stakeholders. The assessment team carefully outlined the work of the participants around these points and scheduled meetings to coincide with academic "down-time" in order to reduce competing teaching and research demands on the participating faculty.

The planning process begun in the spring of 1996 was completed and ready to implement by the following fall semester (see Figure 1. for a timeline of the process.) The time between deciding what, how, and whom to assess was very quickly followed by the implementation of pilot testing. As a result, the relationship between the tools and the abilities they were supposed to assess was still fresh and clear to the faculty. The vitality of the process and the feeling that something concrete was being accomplished reinforced faculty commitment and participation to the overall effort.

As the plan developed it became clear that the abilities faculty wished to assess related closely to ABET learning outcomes criteria 3 (e.g., ability to function on multi-disciplinary teams; ability to communicate effectively, and ability to identify, formulate, and solve engineering problems). The assessment team reviewed the plan, tools and processes to make sure that all Synthesis assessment efforts were designed to further campus efforts to address the new ABET 2000 requirements. The dual focus of the assessment efforts thus served to illustrate for the participants how assessment can be linked to assessment and continuous improvement efforts of their individual campuses.

Since the associate deans for the schools of engineering on each participating campus were key to implementing the new ABET study requirements, they were identified and sought out to be key partners in implementing the Synthesis plan. As ABET moves to include faculty in the accreditation process through the assessment of student learning outcomes, the associate deans and the Synthesis Coalition will collaborate to reinforce the role of assessment as a responsibility of the teaching profession.

Departments must identify the kinds of assessment activities required by their institution as well as by their regional and professional accreditation associations. Assessment processes and procedures can be designed to meet the needs of multiple constituencies. As the Synthesis Coalition's assessment planning process has modeled, collaboration between faculty, administration, and their major stakeholders will lay the groundwork for successful implementation of assessment at the department level.

VII. LESSONS LEARNED

Building commitment to the assessment process began by negotiating a common understanding among the participants. This effort evolved into a process of continuous involvement by the faculty, students, and MIB members. The relevance and utility of the stakeholders' work, in combination with a well defined and understandable process, led to their strong level of commitment to assessment. By creating tools which fit neatly into existing course work and limiting the demands on faculty, the core assessment tools promoted cooperation. Finally, by providing the campuses with a comprehensive picture of their courses and curriculum so that they could assign specific assessments to appropriate courses, the Synthesis Coalition insured that the process was sensitive to campus differences and linked assessment to the faculty most interested in pursuing it.

Key to these efforts was the decision by the Synthesis Coalition to hire an assessment director with a background in program evaluation and assessment. This decision expanded the commitment to collaboration to include engineering and education schools. The assessment staff focused their expertise and knowledge about assessment planning, methodologies, techniques, and implementation to craft the Coalition's commitment to the "who," "what," and "how" of assessment. At the same time, the other collaborators focused their expertise and knowledge on the "what" and "how" of the assessment process.

Despite all of this effort to reduce the demands on faculty, assessment does take time, particularly if it is used effectively to improve courses. As we launch into the implementation phase, we have been careful to design the process to satisfy more than our specific Synthesis Coalition research questions. Once the process is in place, the results can be used to satisfy many of the ABET learning criteria as well as provide valuable feedback and validation for individual faculty classroom practices.

Assessment cannot, however, be left to the faculty alone to implement. Campus administrators who are responsible to implement the new ABET criteria for accreditation have been essential in bringing assessment activities to the campus and ensuring that the results they produce are used to improve student learning, teaching, and the curriculum. While assessment works best if faculty driven, the results must be used for improvement purposes.19

Faculty, administrators, indeed whole offices on campuses are devoted to collecting information about our students, faculty, courses, and alumni. Only recently has information about learning and teaching been added to that list. Much of the information collected is used more (ironically it seems) by those outside of the institution, e.g., trustees, regents, state legislators. Assessment to be useful or meaningful must be feedback to its primary users faculty and students. Administrative support and structures are necessary to ensure that this occurs.

Helping faculty develop the expertise in assessment began a year ago when we asked them to be explicit in the goals they wanted to measure. The planning process will continue until a core of expertise exists on each campus. (In the final year of the assessment plan, analysis will shift from a central team composed of stakeholders from every group, to the campuses themselves.) Once the awareness is raised, the results used, and the faculty prepared, institutionalization of assessment as an integral part of academic life will have moved considerably forward.

VIII. CONCLUSION

Assessment planning at all levels (course, department, college, or coalition) requires attention to detail, clear roles for participants, timely follow through, and active listening. In short, it requires all the characteristics of good teamwork. Faculty and administrators, students, industry representatives, and assessment practitioners all share the responsibility to determine what our students are learning (and what they aren't learning) to improve our courses, curricula, and institutions so that that learning continues to take place and grow. By working towards these goals, the Synthesis Coalition crafted a plan that measures the effects of their reforms and gives faculty new ways to use existing classroom activities to assess the kinds of student learning required by modem engineering. Ultimately, the assessment tools and processes devised by the Synthesis Coalition are readily adaptable to the course, major and departmental levels.

ACKNOWLEDGMENTS

The Synthesis Coalition's assessment planning was financially supported by the National Science Foundation Engineering Education Coalitions Program (Award No. EEC 9625456). The authors wish to thank all the Synthesis Coalition project leaders, students, and industrial board members who contributed their energy, ideas, and thoughts to the assessment planning process and the tools developed with their advice and counsel.

*"The Synthesis Coalition has been funded by the NSF Engineering Education Coalition program since 1990. The primary goal of the Synthesis Coalition is to improve the effectiveness of engineering education by emphasizing embedded computing and mechatronics.

*Participants are asked to write on 3x5 notecard any questions about what remained "muddy" or unclear about a presentation. The cards were collected and analyzed by the presenter in order the identify areas which required more explanation or clarification.

*Clearly, with the practical limitations of assessment, no list could cover every possible ability that engineers need. However, given the importance of developing commitment and a common foundation for assessment across such diverse institutions, the critical issue was gaining consensus around a sub-set of abilities the Synthesis Coalition's participants could agree on as important.

REFERENCES

1. Guba, E., and Y. Lincoln, Fourth-Generation Evaluation, Sage, Newbury Park, CA, 1989.

2. Herman, J. L., P. R. Aschbacher, and L. Winters, A Practical Guide to Alternative Assessment, Association for Supervision and Curriculum Development, Alexandria, VA, 1992.

3. Pavelich, M. J., B. M. Olds, and R. L. Miller, "Real-World Problem Solving in Freshman-Sophomore Engineering," New Directions for Teaching and Learning, no. 61, 1995.

4. Davis, B. G., "Demystifying Assessment Learning from the Field of Evaluation," New Directions for Higher Education, no. 67,1989, pp. 5-21.

5. North Central Association of Colleges and Schools, "Assessment: Comments from the Assessment Plan Review Process," Assessment Update, vol. 5, no. 5,1993, pp. 8-9.

6. Angelo, T., and K. P. Cross, Classroom Assessment Techniques, Jossey-Bass, San Francisco, 1993.

7. Cross, K. P., "On College Teaching," Journal of Engineering Education, vol. 82, no. 1, 1993, pp. 9-15.

8. Brereton, M. F., J. Greeno, L. Leifer, J. Lewis, and C. Linde, "Innovative Assessment of Innovative Curricula: Video Interaction Analysis of Synalysis Exercises, " Proceedings, 1993 ASEE Annual Conference, ASEE, 1993, pp. 1286-1291.

9. Aglan, H., and S. Ali, "Hands-On Experiences: An Integral Part of Engineering Curriculum Reform," Journal of Engineering Education, vol. 85, no. 4,1996, pp. 327-330.

10. Hsi, S., C. M. Hoadly, and M. C. Linn, "Lessons for the Future of Electronic Collaboration from the Multimedia Forum Kiosk," Speculations in Science and Technology, Special Issue on Education, vol. 18, no. 4, 1994, pp. 265-277.

11. Hsi, S., M. C. Linn, and J. Bell, The Role of Spatial Reasoning in Engineering and the Design of Spatial Instruction," Journal of Engineering Education, vol. 86, no. 2,1997, pp. 151-158.

12. Riggs, M. L., and J. S. Worthley, "Evaluation of Student Outcomes Assessment Pilot Projects in the California State University," in Student Outcomes Assessment: What Makes it Work? California State University Institute for Teaching and Learning, Long Beach, CA 1992, pp. 1-22.

13. Ref. 6, p. 2.

14. Astin, A., et al., Principles of Good Practice for Assessing Student Learning, AAHE, 1992. lS. Ref.1 and 4.

16. Rogers, G. M., and J. K Sando, Stepping Ahead: An Assessment Plan Development Guide, Rose-Hulman Institute of Technology, Terre Haute, IN, 1996.

17. Bloom, B. S., et al., Taxonomy of Educational Objectives: The Classification of Educational Goals, David McKay Company Inc., New York, 1956. 18. Ref 2, p. 2.

19. Banta, T. W., et al.,Assessment in Practice: Putting Principles to Work on College Campuses, Jossey-Bass Publishers, San Francisco, 1996.

FLORA MCMARTIN

Synthesis Coalition

University of California at Berkeley

ERIC VAN DUZER

Synthesis Coalition

University of California at Berkeley

ALICE AGOGINO

Mechanical Engineering

University of California at Berkeley

Copyright American Society for Engineering Education Apr 1998
Provided by ProQuest Information and Learning Company. All rights Reserved