The Essential Components of Institutional Effectiveness in Academic Departments

In order to conduct a meaningful and systematic program of institutional effectiveness, three essential tasks must be completed for all academic majors and programs:

(1) The formulation of an Assessment Plan, including a mission statement, objectives, and assessment mechanisms

(2) The conduct of an Annual Assessment Program

(3) The integration of assessment activities within the College's planning and budget processes

This brief document describes the contents of the Assessment Plan and the Annual Assessment Program and recommends means for their completion. A related document, Institutional Effectiveness, Assessment, and Strategic Planning at Roanoke College, describes the integration of these activities into the planning and budgeting processes.

The Primary Justifications for Academic Outcomes Assessment

Conducting assessment of academic programs requires us to conceptually distinguish between "teaching" and "learning." Higher education has rarely used direct indicators of the quality of outcomes achieved. Many academicians have traditionally taken the position that it is simply not possible to evaluate a college education in the same way that we can evaluate a new automobile, a new television, or a gourmet meal that we purchase.

So, in higher education, we have placed much greater emphasis on the "processes" and "input" of higher education as a means for assessing outcome and quality. The education that we provide has been judged to be of high quality, for example, if the instructor knows his or her subject, has a carefully prepared syllabus, incorporates new technologies into the classroom, and stimulates meaningful discussion. Yet, if challenged on the point, all of us would probably acknowledge that none of these "inputs" are necessarily good indicators for the amount of "learning" that has occurred in our classes. Today, it is more widely recognized that teaching and learning are two sides of the same process, but that they are not the same thing. The "means" of higher education (i.e., instruction) are not the same thing as the "ends" or "outcomes" of higher education (i.e., changes in student knowledge or capabilities or attitudes). Academic outcomes assessment forces us to make this conceptual distinction and to focus on both the teaching process and the learning outcome.

Properly done assessment of academic majors and programs contributes to improved teaching and learning in a variety of ways:

(1) Academic departments are led to engage in serious discussion about what they really want to achieve or to have happen with their students. The processes of considering the wide variety of possible learning objectives and reaching consensus on a limited number of them leads faculty to think more purposefully about their mission and their hopes for students.

(2) The design and administration of good assessment mechanisms offers a meaningful way to rigorously evaluate the extent to which a major or program is reaching its specified objectives.

(3) Reflection on and discussion of assessment results among faculty offers a forum for genuine consideration of possible changes that will enable learning objectives to be more fully realized. The ultimate purpose of academic assessment is to stimulate improvements in program structure, course content, and pedagogy.

The Main External Agents Stimulating the Academic Assessment Movement

Advocacy for conscientious assessment stems from four primary agents:

(1) The regional accreditation agencies. In its Criteria for Accreditation, the Southern Association of Colleges and Schools identifies "Institutional Effectiveness" as one of the key review areas for reaccreditation. Included in this section of the 1998 Criteria (p. 19) are the following passages:

"The concept of institutional effectiveness is at the heart of the commission's philosophy of accreditation and is central to institutional programs and operations. It pervades the Criteria for Accreditation. This concept presumes that each member institution is engaged in an ongoing quest for quality and can demonstrate how well it fulfills its stated purpose...The Commission advocates no single interpretation of the concept of institutional effectiveness. It does, however, expect each member institution to develop a broad-based system to determine institutional effectiveness appropriate to its own context and purpose, to use the purpose statement as the foundation of planning and evaluation, to employ a variety of assessment methods, and to demonstrate use of the results of the planning and evaluation process for the improvement of both educational programs and support services. Educational quality will be judged finally by how effectively the institution achieves its established goals."

(2) State governments. State governments around the country are applying significant pressure on public higher education institutions to engage in a systematic process of assessment of their programs and to document the outcomes and value of the education provided. While institutions have responded in a variety of ways, many state governments are insisting on adherence to carefully delineated requirements.

(3) Various academic groups, academic foundations, and disciplinary associations. Encouragement to engage in systematic assessment also emanates from within higher education. Groups such as the American Association for Higher Education and the professional associations in several disciplines have acknowledged the value of assessment and have encouraged institutions to shape their own programs so as to be maximally valuable.

(4) As more and more institutions of higher education have committed to the concept of institutional effectiveness and have engaged in systematic assessment--and see the benefits of doing so, they have become advocates for the process and models of developing good programs.

Writing an Assessment Plan

Each academic major and program must have an Assessment Plan that includes articulation of a mission statement for the major/program, outcomes objectives for the major/program, and a brief description of assessment mechanisms that will be used to determine progress in meeting the objectives. A carefully written Assessment Plan serves as the foundation for all assessment activity. Nevertheless, it can be modified at any time as decisions are made to sharpen or extend the mission and/or as program objectives and assessment mechanisms are modified. A copy of the Assessment Plan for each academic major and program is kept on file in the appropriate Vice President's office and in the Office of Assessment.

The Mission Statement

Each academic major and academic program must have a mission statement. The statement should follow these guidelines:

(1) The mission statement should specify the purpose of the major/program within the overall college context; it should make clear the contribution of the major/program to the institution.

(2) The mission statement should be an extension of the College Statement of Purpose, Institutional Standing Goals, and Curriculum Goals; it should make clear how the major/program contributes to the mission and goals of the College.

(3) The mission statement should be succinct but should be sure to include the essential mission of the major or program (rather than the mission of any individual faculty member). Most statements are about 1 - 3 paragraphs in length.

---

Example of a Mission Statement (from the General Education Program):

As an embodiment of the central values of a liberal arts education, the General Education program seeks to provide students with a broad base of knowledge and skills that will prepare them for productive lives as individuals and as members of communities. This includes a broad and diverse basis of knowledge, an understanding of the fundamental features of human life and culture, and the ability to reason and communicate effectively.

In order to fulfill this mission, the General Studies portion of General Education aims to develop in students a sound, broad intellectual foundation, including the ability to think logically, analytically, and creatively, and to make judgments and evaluations as a result of that process. It also provides an understanding of our world, including a sound historical perspective, an appreciation for diverse cultures, an aesthetic awareness and appreciation, and the ability to communicate effectively. The program helps students to develop an integrated, mature perspective, including an understanding of the interrelationships among various disciplines, a set of ethical values, and a commitment to lifelong learning.

The distribution portion of General Education provides students with effective mathematical skills, knowledge of the potential capabilities of computer technology, and knowledge of scientific and social scientific methodology and contributions. It also promotes an awareness of the importance of mental and physical well-being, the ability to assume new tasks and to adapt to changing realities, and the ability to function effectively, both independently and in groups.

---

Does this statement identify the contribution of the General Education program (the General Studies portion) to the institution? Yes. Does this statement relate to the College Statement of Purpose, Institutional Standing Goals, and Curriculum Goals? Yes. Is this statement succinct? Yes.

The Program Objectives

Outcomes objectives must be in place for each academic major/program. The objectives should follow these guidelines:

(1) The objectives should state the key standing (continuing) aims for the major/ program. They should include the knowledge, skills, attitudes, behaviors, and achievements expected of students in the program. Appendix A, "A Classification of Outcomes Dimensions," should be helpful in consideration of possible outcomes.

(2) The objectives should do the following:

(a) Include an action verb and a statement of ability. Use meaningful verbs: better objectives use action verbs like "paraphrase," "compute," "describe," and "construct"; poorer objectives use verbs that are too general like "understand" and "appreciate."

(b) Involve objectives that can be operationalized and are empirically verifiable/directly observable.

(c) Are attainable or feasible given the resources of the major/program.

(d) Clarify or establish a link between what students accomplish in the program and what they do after they graduate.

(e) Permit multiple paths of demonstrating mastery of program objectives.

(3) The objectives should be outcome- or result-oriented rather than statements of process, and they should be specific. This is contrary to the tradition of both administrative and academic departments which are more accustomed to identifying "what we will do." This model is based on identification of "what will occur" as a result of what we do. Thus, "offering sound courses in core areas of the discipline" is too process-focused. We should instead be thinking about what the objectives are (in other words, why do we offer core courses? What are we trying to accomplish? How will students who take these courses be different?)

The presentation of objectives can follow either of the following two formats:

(1) Majors and programs can articulate broad goals each of which is followed by two or more specific outcomes objectives. This format is used by the General Education program.

---

Example of Outcomes Objectives Being Related to Broad Goals:

The General Education program has identified four broad goals. The third goal is that "Students should develop the ability to communicate effectively." Related to this broad goal are three specific outcomes objectives:

(a) An ability to write clearly using proper grammar, syntax, and punctuation

(b) An ability to identify and use effective structure and devices for organizing written communication

(c) An ability to give an effective oral presentation

---

(2) Majors and programs can articulate specific outcomes objectives that are consistent with the institutional and major/program goals but without a specific reference to them in the list of objectives. Had the General Education program simply listed all of its outcomes objectives without grouping them under specified goals, it would be an illustration of this second format.

Do statements (a), (b), and (c) in (1) express key standing objectives of the General Education program? Yes. Are these objectives outcomes- or results-oriented and are they specific? Yes, they do express an outcome, and they are specific.

The Assessment Mechanisms

Each academic major and program must identify a roster of assessment mechanisms that are used to assess the extent to which the objectives are met. Appendix B, "A Partial List of Potential Assessment Mechanisms," should be helpful in identifying appropriate mechanisms. The following guidelines should be used in identifying and shaping the assessment mechanisms:

(1) Assessment mechanisms should provide answers to genuine questions that we have. If we do not care about the results of questions that we ask, then we have designed a poor mechanism. This is an opportunity to obtain answers to questions that we have about the effects of our educational program on students.

(2) Care should be taken that the assessment mechanisms used actually measure what we intend for them to measure. Assessment mechanisms should enable us to draw correct conclusions about the extent to which our objectives are being met.

(3) Both quantitative (i.e., numerical data such as scores on comprehensive exams and number of students doing independent studies) and qualitative (i.e., more subjective information such as that which might be obtained by assessing student portfolios) should be used. (Often, they nicely complement each other.)

(4) Assessment information should be collected from a variety of constituencies. For example, departments might focus on current majors and minors, students in the introductory class, non-majors taking electives in the department, recent or older alums, faculty in other departments, student services staff, etc.

(5) While there is no magic number of assessment mechanisms that should be included, a sufficient number should be used to ably evaluate the variety of objectives that have been formulated. The number should not be so great as to be burdensome, but having only a couple of mechanisms is clearly not enough.

(6) It is permissible and desirable to incorporate mechanisms that have been used in the past. We should build these into our programs.

(7) Make the assessment program your own. Make it fit our college and our departments and our programs. Make it an expression of what we value and what we, as a quality liberal arts college, are able to provide for our students. Creativity is a plus. Assessment mechanisms such as the Business Policy course and the Education and Chemistry portfolios are excellent assessment mechanisms that reflect the "specialness" of Roanoke College.

---

Example of Assessment Measure (as Used in General Education):

The General Education program includes the following types of assessment mechanisms:

(1) From Current Students: Evaluation of Students on specific examinations, exercises, and assignments in courses; course evaluation forms; focus groups; and essays on ethical values (that are written in the first and third year at the college).

(2) From Graduating Seniors: The Senior Exit Survey.

(3) From Alumni: Alumni Surveys.

(4) From External Reviewers and Institutional Data: Program Evaluation (the PEP process) and the General Education Annual Report.

---

Preparing the Annual Assessment Program

An Annual Assessment Program must be conducted for each academic major and program. The Annual Assessment Program used at Roanoke is based on a model constructed by James Nichols of the University of Mississippi. The Program involves annual development of a five-column grid that includes the following components:

(1) Column 1: Identification of an important goal derived from the College Statement of Purpose, the College Institutional Standing Goals, the Curriculum Goals, or the major/program's own mission statement. The first three columns are completed in late summer-early fall.

A Single Example from General Education:

(2) Column 2: Identification of approximately three objectives that are selected from the longer list of objectives included in the second part of the Assessment Plan. These are the objectives on which assessment activity will be focused for the remainder of the year. The first three columns are completed in late summer-early fall. The objectives selected typically change from year to year.

Continuing Example for One Objective:

(3) Column 3: Identification of the assessment mechanisms that will be used to examine the selected objectives. These mechanisms are drawn from those listed in the third part of the Assessment Plan. Each mechanism should contain a criterion or threshold of success; this is the level at which faculty would conclude that the objective has been reasonably met. In the following example, "" and "" establish this threshold. The criterion should be set at whatever level is appropriate and meaningful. Institutional effectiveness seeks to encourage honest assessment and creative effort to make program improvements without fear of failure or personal retribution. Each major/program is evaluated on the quality of its mission statement, objectives, assessment activity, and effort to make program improvements.

Continuing Example for One Objective:

(4) Column 4: Identification of the results of the application of the assessment mechanism. This column describes what has been learned from each assessment mechanism about performance relative to the corresponding objective. It must contain information that allows determination about whether or not the criterion level has been reached. This column is completed at whenever the results from the assessment mechanism are available or at the end of the year.

Continuing Example (With Hypothetical Data in Column 4):

(5) Column 5: ("Closing the Loop") Identification of efforts to make program/service improvements based on what has been learned and reported in column 4. If assessment shows that an objective is being met, and no one has ideas for doing better, the fifth column simply reports that. When assessment shows that an objective is not being met and/or ideas are suggested for doing better, then the fifth column reports the changes designed to enhance program performance. This column is completed in late spring-early summer.

Continuing Example (With Hypothetical Data in Columns 4 and 5):

Essentially, the whole purpose of a program of institutional effectiveness and of all assessment activity is contained in the fifth column. Institutional effectiveness is a systematic process for thinking about, studying, and making efforts to improve programs and services.

Integrating Assessment in Planning and Budgeting

The Annual Assessment Program will typically produce plans and activities designed to improve the program or service and to increase the likelihood of meeting the designated objectives. The work involved in making these changes should logically be incorporated into the department's goals for the ensuing year. Program/service changes that require financial support should be incorporated into the department's budget request. These items receive prompt attention. The integration of College assessment, planning, and budgeting processes are described in detail in the document, Institutional Effectiveness, Assessment, and Strategic Planning at Roanoke College.

APPENDIX A: A Classification of Outcomes Dimensions

(Source: Peter Ewell, The Self-Regarding Institution:
Information for Excellence, National Center for Higher
Education Management Systems, Boulder Colorado, 1984)

In constructing learning objectives for students, it is helpful to think of the variety of dimensions in which growth or change in students might be desired. The following classification system is helpful in thinking through these various dimensions. It should be considered as a menu; departments may choose to formulate objectives relative to as many or few of these dimensions as they think important.

I. Knowledge Outcomes--the purely cognitive aspect of educational outcomes

A. General knowledge (breadth of knowledge)--familiarity with and analysis and comprehension of facts and principles inherent in broad areas of study. It may include:

1. recall and comprehension of facts in a broad area of study

2. recall and comprehension of theories and terminology in a broad area of study

3. recall and comprehension of investigative principles and methods in a broad area of study

4. recall and comprehension of the history and development of a broad area of study

5. ability to relate/integrate approaches and concepts drawn from more than one broad area of study

B. Knowledge of specific fields (depth of knowledge)--requires mastery of the accepted body of facts, theories, language, and techniques of a particular field of study. It may include:

1. recall and comprehension of facts in a particular discipline or specialized field

2. recall and comprehension of theories and terminology in a particular discipline or specialized field

3. recall and comprehension of investigative principles and methods in a particular discipline or specialized field

4. recall and comprehension of the history and development of a particular discipline or specialized field

5. ability to effectively manipulate appropriate concepts, theories, and investigative methods to create new knowledge in a particular discipline or specialized field

II. Skills Outcomes--the relative abilities of students to perform explicit tasks

A. General competence (social functioning) skills--encompass the abilities to use knowledge, organize information, define problems, and discover and implement solutions to problems. It may include:

1. verbal skills, including reading, reading comprehension, writing, and oral communication

2. quantitative skills, including mathematics, statistics, and computing

3. leadership/organizational/human-relations skills

4. analytical skills, including skills of problem definition, problem solving, and critical thinking

5. invention/innovation/creative-thinking skills

6. aesthetic-appreciation/creative-expression skills

7. physical/motor skills

B. Professional/occupational skills--consist of the particular, specialized skills needed for effective performance in an identified profession or occupation

III. Attitude/Value Outcomes--the affective impacts of higher education

A. Personal goals and aspirations--are defined in terms of levels, patterns, and directions of personal interests, desires, drives, and ambitions. It may include:

1. general goals and aspirations (lifestyle, social mobility, family goals, personal goals)

2. occupational and career goals

3. educational goals

4. motivation and drive levels for each of the above

B. General attitudes, values, and satisfactions--in terms of their content, extent, and internal consistency. It may include:

1. beliefs (including religious beliefs), belief systems, value commitments, and philosophies of life

2. mores, customs, and standards of conduct

3. patterns of feelings and emotions, including particular satisfactions and dissatisfactions with individuals, groups, institutions, and social situations

C. Attitudes toward self--development of identity. It may include:

1. perception of self, general self-concept, self-discovery

2. self-reliance, self-confidence, including adventurousness and initiative, autonomy, and independence

3. satisfaction with self, psychological well-being

4. personality/personal coping characteristics, including flexibility and adaptability, dogmatism/authoritarianism, tolerance and persistence, and so forth

D. Attitudes toward others--related to and part of individual identity. It may include:

1. specific perceptions of other individuals/groups in a society

2. tolerance for cultural and intellectual diversity, including a willingness to accept different points of view

3. general human understanding, including empathy, sensitivity, and cooperation

IV. Relationships with Society and with Particular Constituencies--an often longer- term measure of the effects of higher education on students

A. Relationships with educational institutions

1. individual educational development goals

2. patterns of enrollment, placement, and participation

3. patterns of retention, attrition, and program completion

4. patterns of program/institutional change or transfer

5. levels of achievement in subsequent educational experiences

6. quality of student effort

7. assessed relevance and contribution of past education to subsequent educational experiences

B. Relationships with employers

1. individual employment/career choice goals, including change, stability, and intensity of goals

2. first job obtained after education, including the relevance of this job to the education received

3. long-term employment history

4. income/earnings history

5. promotion and job performance

6. job satisfaction

C. Relationships with professions/professional associations

1. individual professional development goals, including change, stability, and intensity of goals

2. patterns of professional certification/recognition/award

3. patterns of subsequent professional development activities

4. assessed contributions of past education to professional success

5. professional satisfaction

D. Relationships with family/community/society

1. family roles, relationships, and child-rearing practices

2. patterns of social affiliation, group membership, and participation

3. patterns of voluntary contribution, including contributions of time, money, or other support

4. patterns of citizenship activities/political participation