PREFACE TO THE FIRST EDITION
The first edition of the MAC Handbook has been the result of three years of work including the development of the initial design, writing of sections, adding a section on Assessment Design and Plans, and finalizing partially finished sections. Information varies as to its level of current accuracy; changes were received periodically and information has been collected over a long period of time. Some sections that were in the initial outline were omitted as a result of no contributions for a particular topic. All sections were designed with brevity as a requirement. No contribution was intended to be longer than one page.
The original intent for this handbook as established by the Missouri Assessment Consortium was to fulfill the need for a reference resource of assessment practices and experiences of Assessment Coordinators at public four-year institutions in Missouri. This handbook is designed as a loose-leaf notebook in order to easily insert and update changes without reprinting the complete text. Updates should be submitted once a year in June.
This handbook would not exist if it were not for the generosity and willingness to help others that is continually exhibited by MAC members as they interact throughout the year. I appreciate the work of all contributors for their well-meaning contributions, attempts to meet deadlines, their skill in writing, and their cooperation in submitting items in a set format on disk. I would also like to acknowledge the painstaking and cheerful efforts made by Laura Richardson, my secretary, who has worked on this project periodically over the past two years.
- Contributed by Ellen Lenninger
PREFACE SECOND EDITION
The second edition of the MAC Handbook has been the result of nearly a decade of growth and development by one of the most productive ad-hoc organizations in the Missouri Higher Education Community. While a loose-leaf, paper-and-pencil edition of the Handbook will still be available, modern technology makes it possible to download a copy of the handbook from a web page. It is also possible to continuously revise materials in order to keep up with the constantly changing needs of assessment people throughout the State.
Special recognition should be accorded to Bertha Doar, Harris-Stowe State College Assessment Coordinator, who devoted many hours to gathering material and laying out the format for the new electronic version of the MAC Handbook. Without her dedicated service none of this work would have been available.
- Contributed by Wilbur Simon, December 1999
I. HISTORICAL FOUNDATIONS OF MAC
ASSESSMENT IN MISSOURI - AN HISTORICAL OVERVIEW
The current emphasis upon assessment in Missouri higher education reflects a national trend that gained momentum in the 1980s. During that decade the public and government officials increasingly asked whether colleges and universities were accomplishing what many considered to be their primary purpose; namely, to educate students. There were well publicized reports critical of American higher education and also a growing and widespread perception that graduates lacked the skills necessary for the "job" market. These criticisms came at a time when the leaders of both higher education and of other state agencies sought more state funds, but when there was also a "taxpayers revolt." Given the demands for state funds and the rising tide of criticism of higher education, many state political leaders came to view assessment as a means to make higher education more accountable.
In point of fact, already in the 1970s a few colleges and universities had begun to develop assessment programs. These included Northeast Missouri State University (NMSU), which by the mid-1980s had in place a "value-added" model. The national recognition NMSU received for its efforts, a report calling for assessment which the National Governors' Conference Task Force on College Quality issued in 1985, and the widespread criticism of higher education led Governor John Ashcroft of Missouri to become a strong advocate for assessment.
In 1986 Governor Ashcroft, at his Governor's Conference on Higher Education, challenged the administrators, faculty, and governing boards of the state's institutions to develop assessment programs that would show that their students were learning and that their graduates were competent. Earlier, he had directed the state's Coordinating Board for Higher Education (CBHE) and had urged its Council on Public Higher Education (COPHE), a body composed of the presidents and chancellors of the state's four-year public institutions, to develop plans to implement assessment programs on all campuses. The members of COPHE formed a Faculty Committee on Assessment composed of a representative from each of Missouri's four-year public institutions and the committee, working closely with the MDHE staff, developed a statement promoting assessment.
Thus in the 1990s assessment programs are at various stages of development at all of Missouri's four-year public institutions, and the CBHE regularly requests reports and data on the progress being made. In 1991, the coordinators and directors of assessment at each institution formed the Missouri Assessment Consortium (MAC) for the purpose of exchanging ideas among themselves and with members of the MDHE staff. In 1992, MAC issued a statement listing eight "Principles for Assessment in Missouri's Public Institutions of Higher Education." These principles emphasized the need to develop assessment programs which would lead to increased student learning and improved programs, and which also would assure the public that their moneys are being spent wisely.
- Contributed by Jim Sylvester
THE ESTABLISHMENT OF THE MISSOURI ASSESSMENT CONSORTIUM
In July 1991, Pauline Fox, Director of Academic Assessment, wrote to assessment directors and coordinators at four-year public colleges and universities in the state, as well as members of the CBHE, inviting them to attend a one day meeting on assessment. The group met for the first time on August 16, 1991, in Columbia, Missouri.
Each of those attending the first meeting provided an overview of assessment on his or her campus. Several assessment issues were raised and discussed. The consensus of the group was that it should continue to meet, on an informal, grass roots basis. No officers are elected or appointed. Instead, at each meeting someone volunteers to be in charge of the next meeting. This individual makes arrangements for the meeting site, sends out announcements of the meeting, and prepares an agenda. He or she serves as moderator for that meeting, and then takes minutes at the following meeting. The name "Missouri Assessment Consortium" was agreed upon at the second meeting.
The group has chosen to remain very informal. In October 1991, the Vice Presidents for Academic Affairs were asked to recognize the group in order to provide it with some legitimacy. The Vice Presidents for Academic Affairs chose to grant the Missouri Assessment consortium its recognition.
Two members of the MDHE staff, Alan Contreras and Robert Stein, joined the group at the first meeting. At subsequent meetings, one or two MDHE staff members have attended, for at least a portion of the meeting, to listen to the discussions and to offer insights into the accountability dimensions of assessment. In addition, the MDHE staff have used the meetings as a vehicle for sharing information and getting the point of view of assessment personnel.
- Contributed by Pauline Fox<
THE MISSOURI ASSESSMENT CONSORTIUM
Summary of Activities 1994-1999
During the years following the establishment and initial development of MAC, its structure and purpose have remained unchanged. It continues as the forum for information exchange on issues important to the stability and advancement of higher education assessment in Missouri. Other groups have joined with MAC members on occasion. They include assessment representatives of two-year public institutions as well as many private institutions of Missouri. Institutional Researchers and members of the Colloquia for Writing Assessment have also joined with MAC in important discussions on topics of mutual benefit.
There is an important link between this group and the MDHE staff. The mutual exchange of information and advice has helped to shape discussions that will insure and enhance assessment in Missouri for years to come. Topics included CBHE initiatives, statewide surveys, and information on national trends.
Bi-monthly meetings have provided a forum to share information from national, regional, and special interest conferences on assessment. Colleagues have made presentations on procedures and projects unique to their campuses and shared information on North Central accreditation issues involving assessment plans. Additional presentations from vendors of assessment instruments have kept MAC members abreast of developing options and testing trends in assessment. Special guests have explained new initiatives on state and national scenes; some involving competitive grant programs.
Collaboration has, in the past, been helpful to secure price discounts when purchasing certain assessment instruments as a statewide group. Development and refinement of "Assessment Guidelines and Principles" as an official document has helped shape new and emerging program policies. As a contributing "Affiliated Group" MAC gave important council to the State's development of policy for a distance learning delivery system. MAC members were also active on many levels in Missouri's "Performance Funding" and "Funding For Results" initiatives. As Missouri higher education continues to develop its transfer and articulation efforts, several MAC members are contributing in important ways to the debate, particularly in the area of general education curriculum.
Even as some members of MAC have changed, it continues to provide an important resource to those new to Missouri's assessment culture as well as established members. Much of this culture was built on the expertise of several founding members. The new members in turn have provided ideas and perspectives which strengthen the entire group. MAC's informal flexibility and the camaraderie of its members are the strengths that have allowed it to evolve as a viable voice for higher education assessment in this state and beyond.
II. PRINCIPLES OF ASSESSMENT
DEVELOPMENT OF PRINCIPLES BY MAC WITH INPUT FROM CBHE & CAOs
From the beginning, one of the main purposes for MAC was refinement and distribution of a common statement of philosophy. Such a philosophy, we reasoned, would help each of us individually as we sought to develop successful, locally sensitive, comprehensive assessment programs. It would provide the philosophical underpinnings that could direct all of our assessment activities. Practically speaking, discussing theory and practices with other practitioners could sharpen our own programming and provide a frame of reference as we each addressed accountability and improvement of instruction. In addition, we felt that a clean and well-reasoned philosophy to which we had all contributed would do much to present a united front on assessment to policy makers, legislators, and other constituencies. Not only was the articulation seen as a politically expedient thing to do but also as part of the education process we all knew would be necessary. We knew that the assessment dialogue was critical if institutions of higher education in the state of Missouri were to be given the understanding and support necessary for they to react positively to increased accountability and increasing demands for global competitiveness through educational excellence.
After exploring philosophical statements from other states and institutions, notably Kean College, MAC members began working to articulate common principles. These principles were discussed at length; they were reviewed and clarified at each meeting for well over a year. After that time, a version was given to the liaison from the CBHE, who had been both a participant in and an observer of the discussions. The staff at the CBHE made some suggestions for modification, which were discussed at length. Another version of the philosophy, incorporating several of the CBHE suggestions, was then presented to the vice presidents for academic affairs at one of their meetings for the purposes of information exchange, review, and refinement of the philosophy. Finally, the philosophy was shared with the presidents of the institutions of higher education (COPHE) for the purposes of information exchange and review.
- Contributed by Pat Van Dyke
PRINCIPLES FOR ASSESSMENT BY MISSOURI'S PUBLIC INSTITUTIONS OF HIGHER EDUCATION
The Coordinators and Directors of Assessment in Missouri's public institutions of higher education are supportive of assessment when it is used:
- to improve instruction and learning,
- to facilitate the achievement of each institution's mission, and
- to demonstrate the educational, economic, and social roles of higher education.
To these ends, institutions will collect evidence which includes data verifying that the institution is making progress in meeting institutional goals based on state, national and/or international academic standards.
The effectiveness of Missouri's institutions of higher education can be improved by assessing the attainment of the desired outcomes in:
- general education,
- individual courses and majors, and
- the higher education experiences which transcend individual courses and majors
Assessment in Missouri serves two purposes: first, the improvement of instruction and student learning; second, accountability. Assessment should focus on student learning and instruction and should be approached as a multi-dimensional exploration of curricular and co-curricular issues and the learning processes associated with them. In addition, the public institutions of higher education in Missouri recognize a variety of state-wide constituencies to which they are appropriately accountable for the effectiveness of their educational programs, including but not limited to students and parents, employers, taxpayers, the respective governing boards, the CBHE, and the state legislature.
Assessment must be fully integrated into the entire campus environment--including academic administration, student affairs, and support services--and linked meaningfully with institutional planning processes and the allocation of resources.
An institution should have flexibility in selecting the assessment procedures that, in the professional judgment of the faculty, administration, and staff, are consistent with the institution's specific mission, the students being assessed, and the purposes for the assessment.
Assessment should be based on multiple measures appropriate to the institution using them. The data collected should be longitudinal and should include both quantitative and qualitative elements. Assessment programs should be based on reliable research and proven practices. In addition, assessment instruments and methods should be continually evaluated to determine their utility in the assessment process.
Each institution shall assume responsibility for ensuring that it conducts assessment, analyzes and interprets the data collected, and disseminates the results in a professional and ethical manner.
Given the refinement of missions and the recognition of the need for multiple means of assessment, comparisons among public institutions should be based on multiple measures. The institutions collectively with the support of the CBHE, will work to help the public understand the complexities involved in assessing students given the mix of traditional, non -traditional, and transfer students of varying interest, backgrounds, and abilities served by higher education in Missouri.
- Contributed by MAC Coordinators
III. ASSESSMENT INSTRUMENTS
Capstone courses in the major or in general education provide campuses with a curricular vehicle to encourage students to integrate many concepts and skills from previous course work. However, the course might also provide an excellent opportunity for faculty to design local assessments of intended program outcomes. As faculty review student work, there needs to be an attempt to identify common strengths and weaknesses as demonstrated by actual student work.
One of the challenges to using capstone courses as assessment is establishing expectations and identifying methods for faculty to go beyond assigning students a course grade. Conversation among faculty regarding observation of student strengths and weaknesses is the heart of the assessment activity. Thus, capstone courses as a curricular component do not equate with capstone courses as assessment.
The advantages of this form of assessment are many. Faculty themselves create the assessment tools. Faculty evaluation of their own assessment initiatives requires high levels of involvement and thus may be more likely than some other assessment methods to lead to change in the classroom and in the curriculum. The probability of this is even greater if capstone assessment evidence is reinforced by institutional survey data, interview data, and standardized test scores.
- Contributed by Candy Young
ACADEMIC PROFILE TEST
What is the Purpose?
The Academic Profile was developed to test general education skills and knowledge, and thereby assist in assessing the outcomes of general education programs to improve the quality of instruction. This test focuses on academic skills rather than the knowledge gained in general education courses. College level reading, college level writing, critical thinking, and math skills are tested within the context of the humanities, social sciences and natural sciences. Test takers need not have specific subject area knowledge because all information needed is contained in the question. All that is needed to answer the question correctly is the skill. Institutions whose definition of general education is consistent with the skill approach will find a match between their general education priorities and the intended purpose of the Academic Profile.
Two forms of the test are available: short and long forms:
Short Form. The short form allows a college/university to assess its general education program to the extent that the goals and curriculum of the institution's general education program reflect the skill areas measured by the test. A total score for the individual student and mean score for the group of students is provided. The short form is useful when time is limited, the institution is interested in group data only, or the focus is on an initial screening device to determine overall strengths and weaknesses of the student body. Individual student results are too limited to be used for advising purposes.
Long Form. The long form provides individual student sub-scores as well as total individual scores and group and subgroup scores. The long form which provides considerable information is appropriate for an institution which wishes to look closer at subgroups, or use individual or subgroups scores for tracking or advising purposes.
What does the Test Measure?
Rather than measure specific content in college level general education courses, the test focuses on college level reading, college level writing, critical thinking, and math skills in the context of three subject areas: the humanities, social sciences and natural sciences. The humanities selections include poetry, fiction, and non-fiction prose from classical literature through American and British literature. An example of a task measuring critical thinking in context of humanities is recognizing elements in the selection that strengthen or weaken the argument presented. A writing task would require the ability to recognize and reword figurative language in a selection. The social sciences material is selected from areas such as history, economics, political science, psychology, anthropology, and sociology. An example of a reading task using social science content is one that requires students to interpret the meaning of key terms used in a social science selection. Natural science selection tend to be multidisciplinary in that they focus on topics and issues common to basic courses in biology, chemistry, or physics. Of the three areas, biology tends to receive more emphasis than chemistry or physics. As an example of a math skill using natural science content area students may be asked to interpret ratios, proportions or percentages presented in experimental findings. The long form, which provides individual student sub-scores, has 144 multiple choice questions to be answered in two and one-half hours, while the short form consists of thirty six multiple choice questions to be answered in forty minutes. There is also room for 50 locally written multiple choice questions, to further strengthen the match between the local program and the standard test items.
A content-related essay is available. Students select a question from one of the three content areas and write for 45 minutes, using information from course work to support the position they take. The focus is on providing supporting evidence for the position taken. The essays are scored locally, and a detailed scoring guide along with sample papers are provided for this purpose.
How was the test developed?
In 1986 as a response to the growing national interest in general education testing, Educational Testing Service (ETS) convened a series of regional groups to discuss what would be most useful. As a result of those discussions, two products were developed: the Academic Profile for general education testing, and the Major Field Achievement Test for assessment in the major. A multiple matrix design was selected, for the Academic Profile, which would measure skills attained through three broad general education content areas. Improvement of the test occurred after feedback from Pilot Years One, Two, and Three, which resulted, e.g., in shortening the short form so it could be administered in a class period, adding criterion referenced scores, and collapsing the reading/critical thinking proficiency scores.
What are the Characteristics of the APT?
This multiple-choice test is machine scored and provides two types of score information: norm referenced and criterion referenced.
Norm Reference Scores. Eight norm referenced scores are provided: Humanities, Social Sciences, Natural Sciences, College Level Reading, College Level Writing, Critical Thinking, Mathematics, and the Total Score. Scores are reported for a group as mean scores within a 95 percent confidence band.
Criterion Referenced Scores. Criterion referenced scores for writing, mathematics, and reading/critical thinking (treated as a single dimension) are reported for each student on the long form as an institutional summary of the percent of students at each level for both the long and short forms. An example of specific definition of writing ability at level 3 follows: In addition to performing Level I and Level 2 skills successfully, a student at Level 3 also can recognize logical statements and comparisons and is able to solve difficult or subtle writing problems such as appropriate use of parallelism. These students can make fine distinctions among closely related root words and grammatical structures characteristic of mature writing styles.Comparative Data. Three types of comparative data area are available. First, the Standard Reference Group consists of percentile ranks of institutions norm referenced mean scores broken out by the Carnegie classifications (Research & Doctorate, Comprehensive, Liberal Arts and Two Year institutions). Second, Self Selected Reference group scores are available. An institution selects at least eight schools from a list of all institutions that meet the participation requirements of testing at least 30 students with the long form or 40 students with the short form during an academic year. Institutions can select as many as 10 Self-Selected reports each year. The third type of comparative data is criterion referenced or Proficiency Level Data. The percentage of students performing at each of three criterion defined levels is presented for the group of students tested.
Subgroup and Demographic Data. Total score means are available for subgroups based on age, gender, race, number of credit hours completed, native English speaking, enrollment status (full or part time), transfer status, number of hours working, type of program, portion of general education completed, intended or current major, courses taken. This type of information allows an institution to determine variations in general education skill development by subgroups, and thus target problem areas. The fact that as the number of general education hours increases, the mean score increases, provides some validation that the test is measuring college level general education constructs.
Validity. Content validity was addressed during the development of the test. ETS worked with committees of college and university faculty members to address the concerns expressed in the Association of American Colleges 1985 report Integrity in the College Curriculum. ETS staff developed items sampling the subject area. Refinement of the test content continued during the first pilot year when institutions provided additional content information. Construct validity gathered during the 1989-90 year provided additional information regarding explanatory relationships involving class level, grade point average, and the amount of core curriculum completed. Detailed information is available in the Academic Profile User's Guide Part 1. A discussion of Discriminate Validity of the Proficiency Scores also appears in the same document, and supports reasonable differentiation with respect to the level differences among students having flat profiles.
Reliability. Reliability coefficients for the subject area and skill scale norm referenced scores range from .74 (critical thinking) to .85 (reading). The reliability coefficient for the total scale score is .94. The average reliability coefficient for the total score across the four short forms is .80. The standard error of measurement, the extent to which scores vary when the true ability does not, is also a measure of reliability. Approximately 95 percent have true scores within 9.15 points of their reported scores.
Still another approach to reliability is to equivocate forms across two or more alternate or parallel forms. Correlations, adjusted for length, range for .72 to .90 among the various content, skill and total scores.
Item Difficulty and Discrimination. In total, the test is slightly harder than middle difficulty level for junior and senior examinees, which is expected since freshmen are not expected to have experience with the core curriculum, and therefore find the test difficult. Many users find a wide spread of scores which facilitates discrimination of student skills within the mid-ranges as well as at the extremes.
How has the Academic Profile been Used?
The Academic Profile can be used to study the profile of the student body from a general education skills focus. The test can assist in the study of program effectiveness and student strengths and weaknesses. The user norm referenced results can contribute to the accountability requests at the state level. Both norm and criterion referenced results lend themselves to institutional self studies and the accreditation process. Base line data can be established and used to relate performance of future groups or subgroups, or to establish minimum performance expectation ranges. Comparisons can be made with other institutions, using peer institution groups, as well as aspiration institution groups.
The test can be administered at different points in time to the same group longitudinally, and provide a measure of growth. Changes in criterion referenced score information is particularly useful for this purpose since changes often appear when there are no norm referenced changes.
The test can be used as a performance standard. An example involves the rising junior test, where a certain minimum proficiency level is needed before eligibility to take junior level courses is granted. Still another use is an exit exam for graduating seniors, with specific standard tied to graduation. This use requires the greatest care in establishing institutional content and standard validity.
A final use involves counseling students. The long form can provide information which advisors can share with students. The criterion or proficiency scores are most useful in giving students at a particular level such as the end of the junior year, feedback which they still have time to act upon. As an example, students at level 1 in writing ability at the end of the junior year will be handicapped in the job market when they graduate if skills are not improved in the interim.
The many facets of the Academic Profile offer various options for assessment of general education when the focus of the institution's general education mission matches the skill based definition of the test. To further enhance the use of this test, data for faculty and administrators is available on diskette in various user friendly forms (ASCII file or menu driven), so additional local research inquiries can be made.
- Contributed by Ellen Lenninger
College BASE is a criterion-referenced achievement test of post-secondary general education developed at the University of Missouri-Columbia. The "Guide to Test Content" delineates the domains measured, and scores reflect student achievement in each of those areas: 23 discrete skills, which are grouped into 9 clusters within 4 broadly defined subjects. In addition, students who complete the entire test receive ratings in three ranges of reasoning competencies.
Within Missouri, College BASE is used for two purposes: as an admission requirement for undergraduate students entering the "professional" phase of their teacher education programs (a requirement established by the State Board of Education in 1988) as a campus-wide indicator of general education achievement at some colleges. (The Curators of the University of Missouri required the use of College BASE on UM campuses from the fall of 1991 through the spring of 1994.)
Outside of Missouri, the Riverside Publishing Company markets and distributes the test to about 60 colleges and universities nationwide, ranging from Carnegie Research I institutions through Doctoral and Comprehensive universities and four-year liberal arts colleges to two-year institutions that serve a variety of missions.
For further information, contact the Assessment Resource Center (ARC) at the University of Missouri-Columbia -- formerly the Center for Educational Assessment (CEA) 1-800-366-8232.
- Contributed by Tim Parshall
IV. PURPOSES OF ASSESSMENT
STUDENT SELF ASSESSMENT
Assessment efforts provide an important opportunity for encouraging students to reflect on their progress toward knowledge, skills, and attitudes objectives. These opportunities exist at the university, program, and individual faculty level. The university can request self-assessment in numerous ways.
Surveys can include questions asking students to evaluate their learning, ranging from knowledge of specific subject matter, to the values of citizenship, to the level of library skill. Symbolically, surveys also convey messages to students about the types of learning the university considers important. In the process of structuring self-assessment programs, the university or its programs can ascertain self-reports of student attitudes, student maturity, and student learning successes. For example, students might be asked to submit cover letters to portfolio collections describing their best learning experiences at the university or evaluating their educational growth as represented in their portfolio entries. Students might be asked to analyze their personal strengths and weaknesses in student interviews. Writing assessments might include a process for students to analyze their writing habits as well as their writing results.
Within the classroom, faculty can use numerous strategies to get students to reflect on their papers, projects, and other assignments. The faculty member might have students use a criteria sheet to evaluate their own papers just prior to the faculty member returning the criteria-based grade to each student. Another self-assessment project with tremendous potential to provide insight for program assessment as well as individual student assessment is a seven day time-log project. Students can analyze their time spent studying, socializing, attending school events, participating in extra-curricular activities, and working on and off campus. As reflective skills are claimed to be an essential leadership and life skill, more attention should be given to structuring opportunities for students to engage in this type of thinking. Assessment initiatives provide us with an ideal opportunity to do so.
- Contributed by Pat Van Dyke
CHANGE AND IMPROVEMENT
The purpose most campuses identify as the ideal type of assessment is assessment for improvement. To begin the campus identifies the goals it has agreed upon for student learning. Just having a relative consensus on goals gives many faculty a clearer sense of purpose. For many institutions, this phase of assessment is the most dramatic in bringing about changes in the classroom and in the curriculum. University communities frequently feel uncertainty regarding the university's mission. Even if nothing else were to come from assessment, having a more precise view of the university's purpose would be a significant improvement.
As faculty implement methods for assessing student learning and begin to analyze the patterns of evidence relative to student learning goals, a recursive process of goal identification, assessment, and program development can occur. Similarly, faculty who interact with the data frequently find themselves continually adapting program curricula and classroom techniques. The more faculty are directly involved in the development of assessment and the analysis of assessment data, the more dynamic these recursive processes of continuous improvement are likely to be.
When using multiple assessment measures, evidence from many perspectives and values can be examined to identify points of convergence. When quantitative and qualitative evidence converge, momentum for change accelerates. Standardized tests provide information about student performance relative to local objectives and skill levels in such areas as writing, speaking, synthesis, and critical thinking. Examining student performance relative to national standards can be very helpful in identifying program strengths and weaknesses and may be one of the best ways to shock the university community out of its complacency. Instead of assuming that its quality is self-evident, faculty and staff might conclude that its programs could be improved.
Once a data baseline is established, changes in program can be evaluated based on trends apparent in the data collected. The administration can significantly assist campus use of assessment for improvement by emphasizing the importance of using the evidence to identify difficult questions regarding university performance. To be successful there must be a foundation of trust between faculty and administration. Faculty need to be supported in efforts to improve the quality of education on the campus and need to be reassured that assessment results will not be used punitively. Faculty are unlikely to identify weaknesses if there isn't trust in higher administration along with support for suggested efforts to improve.
- Contributed by Candy Young
Accountability is a concept that tends to be received by faculty with tremendous skepticism. For years educators have relied on faculty credentials, student GPA and credit hour measures, and anecdotal success stories to support their claims of quality. However, budget pressures, grade inflation, rising costs of tuition, and governmental efforts to evaluate program effectiveness (that began in the 1960s and escalated as budget constraints escalated) have all contributed to increased calls for accountability. The factors mentioned above as contributors to public calls for proof of educational effectiveness are not likely to diminish in the near future. Furthermore, the pressures for producing a well-educated work force in order to compete in a global economy continue to increase.
The United States has provided access to higher education at a level far beyond our economic rivals, yet the evidence of students' knowledge and skills achievements seems to lag far behind what U.S. businesses need especially in areas of communication, foreign languages, mathematics, and technology. While GPAs have escalated, student knowledge and skills seem inadequate.
Assessment on the one hand might be a way to demonstrate that students really do possess knowledge and skills. However, the data is likely to identify areas in need of improvement. To gain additional resources from the increasingly burdened public, higher education must improve its evidence of a job well done. As the rest of the public sector seeks methods of performance measurement, higher education cannot expect to be exempt from the expectation.
Another type of accountability is framed in the context of total quality management (TQM) wherein organizations are asked to address quality issues from the perspectives of its customers." In higher education, we have many. Society, taxpayers, parents, students, and employers come to mind immediately as consumers of higher education. However, faculty are also consumers of colleague's courses and graduate schools and professional/academic discipline associations are customers of our students. If higher education can't satisfy these entities, criticism of the educational enterprise will only grow stronger.
- Contributed by Candy Young
PLACEMENT & DIAGNOSIS
In its broadest sense, "diagnosis" refers to the systematic and reasoned interpretation of a result or a cluster or results that leads to a particular statement of cause. In this context, diagnosis refers to the meaningful association of a test score with a local education experience. In other words, after careful study the institution has determined that students earning a score below a given point will not be successful in a particular course or pattern of courses without institutional intervention and individual scheduling decisions. To prepare for the individual diagnosis, then, an institution should have completed a self-assessment of its curricular experiences and the backgrounds of students who are/are not successful in them. Initially, the institution may administer a placement test or look to a particular element in the student's academic record (e.g. ACT scores or high school background) and study the relationship between those elements and academic success in specific courses. When the institution has defined what goals its students need to meet and what information seems to indicate the presence of the ability to achieve those goals, it can establish "cut scores" for placement in enrichment or remedial/developmental sections. In cases where more discriminating assessment is required (learning disabilities or other special cases), the institution may work with professionals in departments of psychology or licensed professionals in external agencies in order to develop an individual educational plan.
- Contributed by Pat Van Dyke
ESTABLISHMENT OF MINIMUM COMPETENCIES
One of the most rewarding of the early stages of assessment is the process of establishing minimum competencies for entry-level experiences. Out of the dialog and holistic training come strengthen departmental and institutional perceptions about the skills and the sequencing of material that helps to develop increasing competence in the students. Variations in grading standards often diminish because the faculty have articulated their basic assumptions and affirmed their commitment to intentional development of basic skills; they have collaboratively declared their standards, and, with periodic reinforcement, maintain more commonality in assessing student performance than they had before commonality in assessing student performance than they had before attempting to establish minimum competencies.
One approach to establishing minimum competencies is readily accessible to the department or institution. The department seeking to establish minimum competencies for placement into a course might study the entering performances of students who were agreed to be successful in the course for which the entry level is being established. By establishing those competencies which were present at entry in the successful students, the department would be able to describe those competencies which were basic to success in the course. The minimum competencies, then, would reflect specifically what the professional faculty recognized as competencies expected in all students in a particular course or activity.
A more complex process might deal with minimum competencies after a student has taken a course or series of courses. This establishment of competencies oftentimes touches on elements of program review and curricular responsibility for the achievement of maintenance of acquired skills. As a consequence, the process of establishing minimum exit competencies can become a hot issue, detracting from the central concern of a clear articulation of those skills felt to be essential for a college education person. Moving the debate to the level of student learning, systematic reinforcement of that learning, and appropriate student demonstrations of it is a labor-intensive task. Some institutions have had very good results by stating their minim competencies at placement, after the initial course, and at exit and matching the students' development of competencies with course experiences and specific tasks in a comprehensive assessment program, this, in effect, is a longitudinal audit of student experiences and competencies and can provide a larger picture of what the institution is addressing across the curriculum.
It should be noted that one of the chief dangers of establishing a minimum competency approach at any level--placement, after the initial course, and at exit--is that students may not feel sufficiently pressed to go beyond the minimum standards. An institution setting off to establish and publish minimum competencies as a basis for "passing someone through" or "waiving a requirement" should, at the same time, establish, publish, and strive for extraordinary results from those who easily meet the minimum standards.
- Contributed by Pat Van Dyke
V. CONTENT FOCUS
BASIC SKILLS ASSESSMENT
The components of basic skills assessment reflect an institution's mission and the ability level of its students. In some institutions, the skills learned in college algebra would be basic skills, and a student who had to review college algebra would be in a remedial/developmental mode for the institution. In other institutions, the skills learned in two or two and half years of high school mathematics would be "basic skills." Students who possessed those skills could enter a mainstream mathematics course to complete a general education requirement. Generally speaking, however, the basic skills are mathematics, reading, and writing. Some institutions either expect computer literacy as a basic skills area or require a computer literacy course of all students. Basic skills areas are oftentimes reflected in the general education requirements and are sometimes supplemented by remedial/developmental courses that would provide additional time on task for under-prepared students.
Several different instruments are widely used to assess the basic skill of reading. The Enhanced ACT battery does provide a reading score, which has been equated to an older test of reading skills-- Nelson-Denney-- and newer tests such as the Degrees of Reading Power. At least one Missouri institution has studied the correlations between low reading scores on the ACT or the DRP and GPA/attrition data. In general, older reading tests tend to be timed tests with built in biases for "fast processors." Furthermore, since much of the tests are based on a large number "literary" vocabulary items offered in isolation not in context, certain types of readers with certain reading backgrounds tend to score a good bit higher than other readers with different backgrounds and interests. Newer tests would tend to place vocabulary in context, to be generously timed if timed at all, and to involve reading passages of differing levels of difficulty. Useful for indicating the broad abilities of a whole group and for planning lessons, orientations to the discipline, ordering test books, and the like, basic skills reading tests do not generally diagnose reading problems in individuals nor do particular scores suggest strategies or difficulties individual students may have with reading.
Writing Assessment--if well done--is time consuming and costly, since it will involve students writing samples in a fairly natural situation. Unfortunately, few institutions can manage basic writing assessment humanely and practically, so students generally are asked to sit for an hour to an hour and a half, responding to a "prompt" written and pilot-tested by a faculty team months before. The essays are then read by at least two trained readers who each independently give the essay a score based on an agreed up rubric, describing key attributes of an essay that would receive a particular score. If the two readers substantially agree, the essay receives the score or average of two scores. If they do not agree, the essay is read by a third reader, who adjudicates the final score.
If the students may be scheduled into two or more courses in an academic area such as science or social science, it is advisable to assess their backgrounds in these areas, using broad, screening instruments such as the TASK Social Science and TASK Science tests. These 25-minute tests are not useful for students with high abilities in a particular area. But it is very helpful for advisors to know which students should not be scheduled into two courses in social science or two classes in the sciences in the first semester. By carefully sequencing the work of the general education component, the advisor can help under-prepared students develop their skills and their backgrounds systematically. In addition, basic skills scores used in combination (a low science score with a low mathematics score) help the advisor decide on the sequencing of specific courses. If the student, for example, indicated she was quite likely to become an agriculture major, the advisor would know that the mathematics skill should be addressed before the student took the first class in chemistry.
Basic skills assessment information studied systematically with other data will provide advisors and faculty with a clear framework with which to make advising and instructional decisions about both individuals and groups.
- Contributed by Pat Van Dyke
UTILIZATION OF MAJOR FIELDS
Major field assessment using multiple measures of assessment can inform a department about the effectiveness and efficiency of their program and can lead to continuous quality improvement within the curriculum. Clearly, departments need a variety of assessment information:
- employers' and alumni surveys and interviews
- senior surveys and focus interview
- performance measures indicating the student's ability to handle the writing, speaking quantitative, research, and analytical tasks of the profession; and
- comparative measures using appropriate criteria or normed performances on standard instruments.
Performance measures, although labor intensive, provide extremely valuable information to the faculty. Some academic units use senior seminars to require a single, capstone project, which demonstrates the student's ability to conduct independent research, to write in the field, and to present the work to peers. Over time, performance standards for these senior projects become part of the departmental culture. It is important, then, to establish clear rubrics and high standards from the outset. Other academic units require each senior to submit a portfolio of his or her work for serious review by a team of faculty. Some departments establish clear standards to guide the student's selection of material; they reinforce their program goals by requiring papers/activities which demonstrate the qualities they sought to address in the program. The portfolio, then, becomes a tool of program evaluation as well as a learning/culminating activity for the senior. In a structure interview or in a cover letter to the portfolio itself, the student can demonstrate reflective and meta-analytic skills.
In the case of assessment of the major field, faculties have recourse to great number of measures in some fields--nationally normed and widely used instruments such as Graduate Record Examination, the National Teachers' Examination, or the Major Field Examination. In other fields, no nationally normed measure has been developed at the current time; furthermore, since the numbers of students who graduate in that field is relatively small, there is little likelihood that such a test would be developed in the future. (Notable areas without nationally normed tests currently include agriculture, art, speech, and theater.) In other cases, the nationally normed test may not be appropriate for the program on a particular campus, especially if the campus has sought to distinguish its programs by developing a unique mission and emphasis. In such cases, the department may utilize the national test but should request item analysis whenever available and discount items that are not of the program mission. Departments may also supplement such tests by devising additional items or developing local tests, working collaboratively with an external consultant or a cooperating department in another institution for purposes of external review and as a step toward establishing validity.
- Contributed by Delores Honey
VI. ASSESSMENT TECHNIQUES
The term "bench marking" has both casual and exact meanings. In the casual usage, bench marking means comparing a performance or an end result to a better performance or better end result. At worst, then, the casual usage encourages facile and superficial "snapshot" comparisons such as "We bench marked our performance against that of Mega University." An unsophisticated use of comparative norms within a state, for example, might seek to identify a university whose students earn the top scores as the "benchmark institution." Since this practice would discount any differences in student background, costs per unit of instruction, or institutional processes, the bench marking project would not be helpful.
A more exact use of the term would be found in projects in which there had been:
- a systematic self-study of the key processes and outcomes of an academic or organizational unit
- the identification of a comparable institution or unit that performed that key function or achieved the outcomes to a significantly higher degree; and
- an open, cooperative undertaking of partnership between the group seeking specific improvements and the "best in class" group, whose outcomes/processes are exemplary.
Bench marking, in this sense, is a focused process, undertaken by relatively sophisticated organizations which have first identified key processes in need of improvement and second know their potential partners, who are accessible and willing leaders in the field in performing that process.
It should be noted that bench marking is one of the labor intensive assessment practices; it has been said on more than one occasion that bench marking is also a practice of particularly secure and mature organizations with a good deal of experience in assessment of other kinds. In practice, these organizations support trained teams with significant amounts of assigned time for bench marking activities.
Bench marking activities may focus on results only, as noted above, without self-study and without identifying the key processes or outcomes that should be addressed. In that case, little improvement should be expected in the institutions/units with ostensibly lower performances. Indeed, since the comparisons are being used to rank and sort rather than to improve through deliberate study, the project may prove counterproductive. Each institution may know its place in the comparative ordering, but none may set out to identify and change the processes the led to the results. Instead, institutions and units will focus on" improving the scores." In all likelihood they will continue to do the same things harder and faster than they did them before. When those redoubled efforts fail to produce the results desired, the institution/unit will fell that improvement (as defined) is impossible for them. They will say, "We did everything we could to raise the scores, and we just didn't make any improvement at all."
Bench marking projects may be undertaken based on the result of either nationally normed or criterion-based tests. Beginning with a perceived difference in outcomes and desire to improve the practices that led to those outcomes, an institution would begin analyzing its own operation in-depth. When the analysis was well along, the institution would narrow its attention to the key processes that it wanted to improve. At that point, the institution would move to identify likely bench marking partners that would be willing to undertake a cooperative activity aimed at improvement. This systematic approach should lead to improvements as well as heighten awareness of internal processes and the linkages between those processes and the desired outcomes.
- Contributed by Pat Van Dyke
PEER REVIEW/EXTERNAL EXAMINERS
An extremely valuable professional association can develop between a department and consulting "partner." Departments which have articulated their program goals arrange with a respected member or members of their field to assess the products of their program. Since the students themselves are products, the department will have to provide for sufficient time on site for the reviewer/examiner to interview students and to observe their thinking and professional behaviors. The work the students have one can also be called products; these products can be mailed to the reviewers/examiners with all pertinent materials such as program mission, goals, and objectives, curriculum, and assessment information. Working with the department, the reviewer helps clarify strengths and areas of concern for departmental action.
The role of the reviewer and the external examiner may differ in that the examiner generally spends the majority of the time evaluating student performance and knowledge and submits her review of the strengths and weaknesses of the program based on what students can demonstrate. A reviewer--on the other hand-- may split the time between processes and products and focus on the dynamics of learning and achievement the department can realistically address. But, this distinction is blurred when a review's strengths and interests lie in students and their demonstrations of knowledge of skills or when an examiner works in a institution which is also addressing the relationships between curriculum, teaching and student learning outcomes.
Many departments feel that the preparations they make to inform an external examiners/reviewer are very beneficial, regardless of the outcome of the review/examination. Some departments routinely arrange for colleagues from other institutions to be invited to recitals and productions and solicit feedback; these departments have established a connection to the outside, then, and are ready for systematic study in a collaborative setting with the purpose of sharpening the program and improving the learning outcomes.
- Contributed by Pat Van Dyke
Performance Assessment. Although any assessment of a student's knowledge or skill, e.g., a multiple choice test or essay exam, could legitimately be considered a performance assessment, the term performance assessment is usually reserved for those assessments that require students to provide tangible evidence (a product of some type) of their learning. Typical products would include oral presentations both individual and group, videos, an original piece of artwork or music, papers, group projects or any combination thereof. Performance assessments also tend to measure more than just knowledge and often are designed to assess multiple skills. For example, in a particular type of performance assessment called an 'in-basket' exercise, a student might be presented with a set of tasks or problems which require she/he to demonstrate content knowledge, problem solving ability, organizational skills, analysis, oral and written communication abilities, and facility with certain presentation hardware and software (e.g., scanners, projectors, Power Point). Ideally, the performance assessment should be as authentic, i.e., as close to real life, as possible. In fact, a commonly used name for performance assessment in the education field is "authentic assessment."
Another key attribute of performance assessment is the use of student learning outcomes that clearly define the learning goals for a given performance exercise. These learning outcomes in turn are usually accompanied by a set of component abilities which make up or define the more general learning outcome. For instance, if one of the learning outcomes to be assessed in a performance exercise is social interaction, a set of component abilities might include such things as listening skills, tolerance of difference of opinion, respect for others, reinforcement of opinions of others, inclusion of all members of group in discussions, etc. For each of these component abilities there would be a set of criteria to be used by the evaluator to determine if the student had shown competency or mastery of the given outcome 'social interaction.' These criteria are a major part of what is typically called a scoring rubric.
The scoring rubric can be thought of as the 'set of rules' used to judge or evaluate the student's performance. Although scoring rubrics come in many forms, they frequently involve the use of categories or levels. For instance, faculty might decide to use the categories Highly Proficient, Proficient, Adequate, Not Adequate-Some Remediation Needed, Seriously Inadequate to describe the competency level of the student. Each of these categories would in turn have a set of criteria or performance standards against which the evaluator(s) would judge the student's performance to determine category assignment. Expert judgment therefore is a key component of most performance-based assessments. The use of individuals from outside the institution as part of an evaluation team is quite common and even preferred.
The scoring rubric is commonly made public so that the teacher, student and the individual assessor (or evaluation team) are all aware of the standards and criteria by which the student's performance is to be judged. There are many benefits of making the criteria public. The students know how they will be assessed. The students can internalize the criteria to help them evaluate present and future performances for themselves and their peers. There is no ambiguity in the determination of the students' "grades." The feedback provided to the students is explicit and more meaningful allowing for greater learning. The criteria also serve to focus the teaching of faculty who hopefully have discussed and come to an agreement as to the component abilities of each outcome to be included in the performance assessment (or class, or program). The teacher can be more confident the student "knows the material" because she/he has demonstrated the ability to use their knowledge and skills.
The use of experts from outside the academy in the development of the assessment and the criteria is quite common as well. The involvement of faculty from other institutions, alumni, employers, etc. in the development and evaluation of student performance adds validity and credibility to the assessment for the student and the general public. The involvement of "outsiders" typically improves student motivation as well. Their participation adds a certain element of real world relevance that is lacking in the normal classroom.
In summary, performance assessment is a general term to describe a variety of assessment processes, which focus on requiring students to provide tangible evidence of their knowledge and skills. Performance assessments typically require extended periods of time, are often collaborative in nature, are guided by explicit learning outcomes and criteria for excellence, make use of expert judgment in the evaluation of student performance, and provide meaningful feedback to the student and teacher regarding student competency and quality of instruction.
VII. ASSESSMENT ISSUES
ACCOUNTABILITY AND IMPROVEMENT
Assessment as a process is used to promote both accountability and improvement. The relationship between these two purposes is not always clear; some believe the purposes are quite distinct or contradictory while others suggest they should be seen as interdependent.
Assessment that leads to accountability usually takes the form of reporting the results of student and institutional performance on a set of agreed-upon measures, e.g., standardized test scores, freshmen completion rates, or graduation rates to boards and state legislatures. These reports provide a basis for evaluating the adequacy of knowledge and skills possessed by students, as well as the efficiency and effectiveness of institutions.
In contrast assessment for improvement involves using the results of assessment to make changes in programs, e.g., curricular development, course requirements, counseling strategies, or co-curricular activities. Feedback from assessment results are used by faculty, staff and administrators to identify areas in need of improvement, develop new initiatives and monitor the impact of proposed reforms.
It is often challenging to combine the goals of both accountability and improvement into an institution's assessment program. A major challenge for assessment programs is the creation of meaningful measures, appropriate benchmarks utilizing comparison groups, and target goals that can be monitored in a regular and systemic way.
Assessment for accountability is deficient if campuses choose measurements haphazardly simply to please policymakers and if the information gathered is not used for self-evaluation. Accountability in a vacuum may create the image of "looking good" to external constituencies, but has limited utility. Self-evaluation, grounded in evidence, should be a minimum expectation of students, parents, employers, and state taxpayers.
How accountable are institutions, which are not committed to continuous quality improvement? On the other hand, if improvements result from assessment practices, but cannot be used to demonstrate accountability, the significance of improvements will be questioned.
Historical Development of MAC
In August 1991, a small group of assessment coordinators at public four-year institutions responded to an invitation initiated by one colleague to attend an informal meeting with representatives from the CBHE. The intent of the meeting was to share perspectives about assessment. Although less than half of the institutions were represented, the experience was very positive and the group agreed to meet again and encourage participation by their colleagues.
As a result of the commitment of assessment coordinators from public four-year institutions to meet on a regular basis, MAC was formed. From the beginning, MAC was structured to be an informal statewide network of persons responsible for and interested in assessment practices. While several meetings and events have involved larger groups, the core members in MAC have been assessment coordinators from public four-year institutions.
Since its inception, MAC has remained an informal ad hoc entity. All meetings are open to anyone interested in assessment issues. MAC has no officers. Responsibility for meeting arrangements and agendas are delegated to volunteers and the host for a particular meeting often shifts from one institution to another. Meetings are usually held in Columbia, Missouri because of its central location, though MAC also has chosen to meet during some statewide conferences.
The primary goal of MAC is the establishment of an effective statewide network surrounding issues associated with assessment. An update from the academic affairs staff of the DHE is included on the agenda of each MAC meeting. Regular meetings of MAC have helped to share perspectives and good practices across campus boundaries as well as inform state-level discussion on policies involving assessment programs at Missouri's institutions of higher education.
Use of National Standardized Tests
National standardized tests are often part of an institution's assessment program. These externally developed assessment instruments provide summary statistics that are quick to analyze. They also are less labor intensive than developing and evaluating local assessments of student performance. The use of standardized national tests encourage faculty to look beyond the local campus to assessments often designed by professional colleagues from across the country. National tests put the performance of an institution's students within a national context and may provide feedback to an institution about aspects of programs that are in need of attention and improvement. External constituencies are often interested in performance based on national test data.
Utilization of national standardized assessments also has limitations. The test may not cover all areas deemed relevant by a local faculty and may include some areas seen as not important. Data from national standardized tests may have limited utility in providing diagnostic information to individual students about their deficiencies. The use of a single test score for high stakes decisions has been criticized and may involve an inappropriate use of data. Developers of national assessments advise institutions to utilize multiple measures to get a fuller picture of both student and institutional performance.
- Contributed by Robert Stein
ACCOUNTABILITY AND MAC
At the initiation of an assessment coordinator on one campus, an informal meeting of assessment coordinators from public four-year institutions and staff from the CBHE occurred in August 1991. The intent of the meeting was to share perspectives about assessment from the campus perspective as well as the state perspective. Although this first meeting consisted of only a small group (less than half of the institutions were represented), it was a positive experience and the group made a commitment to continue meeting with the intent of forming an informal statewide network.
The commitment of assessment coordinators to meet on a regular basis has helped to launch MAC. Although the group remains very informal and ad hoc, it has succeeded in achieving its primary goal of establishing a statewide network around assessment issues.
MDHE staff have been invited to attend each MAC meeting. In order to preserve continuity, liaison with MAC has been assigned to two DHE staff members, one from planning and one from policy analysis. State level participation at MAC meeting is intentionally for only part of the meeting. This reinforces the independence of MAC from state level control. Regular meetings between DHE staff and MAC members on an informal basis have helped to inform discussions about using assessment for accountability and improvement.
- Contributed by Robert Stein & Candy Young
USE OF NATIONAL TESTS
Standardized and nationally normed tests have been an important part of assessment since the assessment movement began in the 70s. These externally-written exams encourage the campus community to look beyond its own institutions in assessing student performance. Conceptually it is important to remember that universities must respond to constituencies external to the university. If preparing someone as an accountant, the local institution in assessing student performance and the quality of the curriculum must focus on the needs of the accounting profession and society not just on the local campus faculty interests. Externally written exams may do the best job of jolting faculty and staff out of their complacency and presumed quality. They may come to recognize program aspects in need of attention and improvement.
There are several additional advantages of including standardized tests in an assessment system. First they require less faculty time to administer. Secondly, they provide summary statistics that are quick to analyze. The risk of reliance on the standardized tests is that campuses won't add other instruments to the assessment system, thus totally relying upon national test data. There is also the significant challenge to use the data appropriately.
- Contributed by Elissa Lewis
ASSESSMENT POLICY AND GUIDELINES
- POLICY. Guidelines and requirements for assessment of learning received by means of a telecommunications-based delivery system are not unique. While recognizing the complexity of the issue, the affinity group nonetheless maintains that principles of assessment of "telelearning" should in general be the same as those of assessment of learning "on-campus". The assumption is made that assessment should be guided by well articulated, externally validated outcomes.
- PURPOSES OF ASSESSMENT. Assessment should be viewed as a key improvement process that can be used in the academic, administrative, and student support services areas. However, since the core process of institutions of higher education is student learning, assessment in Missouri primarily serves three major purposes:
- improvement of student learning and instruction,
- accomplishment of institutional mission, and
- accountability for achievement of educational goals.
Improvement of Student Learning and Instruction. Assessment cannot be effective without first clearly defining what is to be assessed. It is the responsibility of faculty to determine the student learning outcomes that drive the curriculum. Outcomes can be thought of as student "needs" as defined by faculty, employers, alumni, etc., and consists of a specific set of knowledge, skills, and attitudes. It is likely that an institution has a set of outcomes common to all graduates (e.g. general education) and other outcomes unique to the various degree programs. In order to change and improve student learning and instruction, student data derived from assessment of institutional/program/course outcomes must be used as feedback to students and faculty. Assessment data can also be used to document student and faculty performance for accountability and evaluative purposes.
Achievement of Institutional Mission. The outcomes defined by each program operationally define the institutional mission. The outcomes must flow from and support the institutional goals. Every department/unit mission should be derived from the overarching mission of the institution and assessment should be used as a means of attaining and documenting achievement of that mission.
Accountability for Achievement of Educational Goals. Institutions of higher education in Missouri recognize a variety of state-wide constituencies to which they are appropriately accountable for the effectiveness of their educational programs including but not limited to students, parents, employers, taxpayers, the respective governing boards, the CBHE, and the state legislature. To this end, institutions should collect evidence which includes data verifying that the institution is making progress in meeting institutional goals based on state, national, and/or international academic standards.
- LEVELS/CONTEXTS OF ANALYSIS. It should be recognized that assessment, viewed as a means for general process improvement and accountability, occurs at different levels and contexts of analysis. As such, assessment should be fully integrated into the institution's entire operational system, including academic services, administration, student affairs, institutional planning, and resource allocation. The focus of institutional assessment, however, must be on student learning.
- MODES OF ASSESSMENT. Assessment should be based on multiple measures (i.e., more than one type of assessment of a given outcome in a variety of contexts through a variety of means) appropriate to the program and institution using them. The data collected should be longitudinal and should include both quantitative and qualitative elements. Assessment programs should be based on reliable research and proven practices. In addition, assessment instruments and methods should be continually evaluated to determine their utility in the assessment process.
An institution should have the flexibility in selecting the assessment procedures that, in the judgment of the faculty, administration, and staff, are consistent with the institution's specific mission, the students being assessed, and the purposes for the assessment. It is recommended that each institution establish an external evaluation mechanism whereby both the institution's program outcomes and assessment instruments are regularly reviewed and evaluated by a unit or group external to the institution.
- INSTITUTIONAL ASSESSMENT RESPONSIBILITY. Each institution shall assume responsibility for conducting assessment, analyzing and interpreting the data collected, using the information to improve the assessed processes, and disseminating the results in a professional and ethical manner. It is particularly important that institutions involved in joint ventures cooperate in the sharing and development of assessment instruments and processes. However, it is ultimately the responsibility of the degree granting institution to ensure that appropriate standards of performance are met by its graduates.
- DISSEMINATION OF RESULTS. Given the refinement of missions and the recognition of the need for multiple means of assessment, comparisons between institutions, when appropriate, should be based on multiple measures and include contextual information. The institutions collectively with the support of the CBHE, will work to help the public understand the complexities involved in assessing students given the mix of traditional, non-traditional, and transfer students of varying interests, backgrounds, and abilities served by higher education in Missouri.
VIII. ANATOMY OF A VISITATION BY THE HLC OF THE NCA
A. RECOMMENDED READINGS ON ASSESSMENT
Banta, T, W., J. P. Lund, K. E. Black, & F. W. Oblander. Assessment in Practice: Putting Principles to Work on College CaMpuses. San Francisco: Jossey-Bass, 1996.
Banta, T. W. "Implementing Outcomes Assessment: Promise and Perils." New Directions for Institutional Research #59, Fall 1988.
Ewell, P. T. "Assessing Education Outcomes." New Directions for Institutional Research #47, September 1985.
Fendley, W. R., Jr. and L. T. Seeloff. "Reference Sources: An Annotated Bibliography for Institutional Research." Association for Institutional Research Resources for Institutional Research, Number Eight, 1993.
Jacobi, M., A. Astin, and F. Ayala. College Student Outcomes Assessment: A Talent Development Perspective. ASHE-ERIC Report #7, 1987.
Pascarella, E. T. and P. T. Terenzini. How College Affects Students. San Francisco, Jossey-Bass, 1991.
Ratcliff, J. L. "Assessment and Curriculum Reform." New Directions for Higher Education, 980, Winter 1992.
Stark, J. S. and A. Thomas. "Assessment and Program Evaluation." ASHE Reader Series, Needham Heights, MA.: Ginn Press, 1994.
Suskie, L. A. "Questionnaire Survey Research: What Works." Association for Institutional Research, Number Six, 1992.
Terenzini, P. T. "Assessment with Open Eyes: Pitfalls in Studying Student Outcomes." Journal of Higher Education, (November/December 1989) 60: 644-664.
Whiteley, M. A., J. D. Porter, R. H. Fenske. "The Primer for Institutional Research." Association for Institutional Research Resources
- Contributed by Delores Honey - Spring 2000
B. INSTRUCTIONAL & ASSESSMENT RESOURCES
Angelo, T. A. & Cross, K. P. (1993). Classroom assessment techniques - a handbook for college teachers (2nd ed.). San Francisco: Jossey-Bass Publishers.
Banta, T. W., Lund, J. P., Black, K. E., & Oblander, F. W. (1996). Assessment in practice - putting principles to work on college campuses (1st ed.). San Francisco: Jossey-Bass Publishers.
Brookfield, S. D. (1995). Becoming a Critically reflective teacher (1st ed.). San Francisco: Jossey-Bass Publishers.
Fink, A. (Ed.). (1995). The survey kit - the survey handbook. (Vols 1-9). Thousand Oaks: Sage Publications, Inc.
Fink, A. (1995). The survey kit - how to ask survey questions. Thousand Oaks: Sage Publications, Inc.
Bourque, L. B., & Fielder, E. P. (1995). The survey kit - how to conduct self-administered and mail surveys. Thousand Oaks: Sage Publications, Inc.
Fink, A. (1995). The survey kit - how to design surveys. Thousand Oaks: Sage Publications, Inc.
Fink, A. (1995). The survey kit - how to sample in surveys. Thousand Oaks: Sage Publications, Inc.
Fink, A. (1995). The survey kit - how to analyze survey data. Thousand Oaks: Sage Publications, Inc.
Fink, A. (1995). The survey kit 9 - how to report on surveys. Thousand Oaks: Sage Publications, Inc.
Fowler, F. J. (1995). Improving survey questions - design and evaluation (applied social research methods series Volume 38). Thousand Oaks: Sage Publications.
Frank, H. (1999, April). The University of Michigan-Flint Assessment Handbook. Paper presented at the meeting of the American Association of Higher Education Assessment Forum, Denver, CO. June 1999.
Frey, J. H., & Oishi, S. M. (1995). The survey kit - how to conduct interviews by telephone and in person. Thousand Oaks: Sage Publications, Inc.
Glassick, C. E., Huber, M. T. & Maeroff, G. I. (1997). Scholarship Assessed - evaluation of the professoriate (1st ed.). San Francisco: Jossey-Bass Publishers.
Litwin, M. S. (1995). The survey kit - how to measure survey reliability & validity. Thousand Oaks: Sage Publications, Inc.
López, C. L. (1996, March). Opportunities for Improvement: Advice from Consultant-Evaluators on Programs to Assess Student Learning. Paper presented at the North Central Accreditation Commission on Institutions of Higher Education, Chicago, IL. March, 1996.
Seldin, P. (1980). Successful faculty evaluation programs - a practical guide to improve faculty performance and promotion/tenure decisions. Bolton: Anker Publishing Company, Inc.
Seldin, P. (1997). The teaching portfolio - a practical guide to improved performance and promotion/tenure decisions (2nd ed.). Bolton: Anker Publishing Company, Inc.
Strickland, K., & Strickland, J. (1998). Reflections on assessment - its purposes, methods & effects on learning. Portsmouth: Boynton/Cook Publishers, Inc.
Upcraft, M. L. & Schuh, J. H. (1966). Assessment in student affairs - a guide for practitioners. San Francisco: Jossey-Bass Publishers.
- Contributed by Bertha Doar Spring 2000
C. ACCOUNTABILITY versus IMPROVEMENT
Accountability and improvement are often presented as distinct and even contradictory purposes of assessment, instead of as complementary elements of assessment programs. Assessment for accountability is perceived as the collection of standardized test scores for the purpose of reporting to boards and state legislatures. These reports are intended to provide the basis for evaluation of the adequacy of knowledge and skills possessed by students. Assessment for improvement is typically viewed as providing feedback to faculty, staff, and administration that will assist 1) in identifying areas in need of improvement and 2) in monitoring the impact of proposed reforms.
In the final analysis, it is hard to imagine why these two purposes should not each be seen as essential components of a quality assessment system. Assessment for accountability is deficient if campuses are not using the information to self-evaluate programs for areas in need of improvement. This self-evaluation, grounded in evidence, should be a minimum expectation of students, parents, employers, and state taxpayers. How accountable are we if we are not attempting continuous improvement in the quality of student learning on our campuses? Similarly, how serious about improvement are we if we are unwilling to examine the performance of our students relative to standards set by external examiners from the national discipline associations and relative to the performance of students who graduate from programs throughout the country. In fact, both of these goals ought to be seen as essential for quality assessment systems.
- Contributed by Bob Hess