MAC Meeting Minutes November 2007
Missouri Assessment Consortium
Meeting Notes - Nov. 16, 2007
The meeting was held at the DHE offices. Martha Kirker served as moderator. Janet Juhlin hosted and served as historian. Attendees/participants: Beth Nolte, Delores Honey, Martha Kirker, Chris Frazier, Pat Gillman, Jeremy Kintzel, Hillary Fuhrman, Cindy Heider, John Scheulew, Nancy Winterburg, Cindy Bertalott, Mike Easter, John Clayton, Michael Strait, Kumar Thulasi, Dan Schierenbect, Doug Dunham, Robin Adkinson.
Guest: Tom Reeves, National Director, Assessment Solutions - ETS
10:05 Welcome, Introductions
10:07 Greeting from Tom Reeves/Questions and Discussion
Tom began by explaining that he does not have a background in assessment. Rather, he coordinates meetings like this one and determines who needs to get involved based on the issues presented.
He explained that the Voluntary System of Accountability is in response to the Spellings report. Many provosts and assistant provosts are working on this project, and commonalities of expectation are emerging. On the VSA website there will be a template for transparency, a page to identify a student engagement instrument that will probably be the NSSE, and an accountability part that relates to general education. Assessment instruments published by ETS, ACT, and CLA are being considered as preliminary learning outcomes tests. FIPSE awarded a $2.4 million grant to see how these three interface. The accountability part will not be required for four years.
Question: The MAPP does not seem to be getting as much publicity as the CLA. Is the MAPP appropriate to use?
Tom - CLA focuses on analytical reasoning, problem solving, critical thinking, and written communication. It has been implemented by state departments of higher education who tell institutions in their states to buy in. ETS doesn’t do this. Rather, ETS encourages the development of a culture of assessment on campuses, including the development of good objectives and the use of multiple instruments.
Jeremy noted that the CLA is popular with policymakers, because the institution is the unit of analysis. Additional instruments are needed to assess individual students or programs.
Tom - Fifteen months ago, ETS began to coordinate and disseminate information at high levels. ETS for 60 years has been a research-based, nonprofit organization. It has now acknowledged the need to reach out to policymakers. Within the next three months, ETS will hire people to lobby support at conferences and from campus leaders.
Pat Gillman shared that her institution is looking for program-specific assessment for two-year institutions. Tom explained that ETS soon will have an MFT for an associate’s degree in business.
Question: What are the differences between CLA and MAPP?
Tom - ETS advises that you determine your purposes before you buy an assessment instrument. The MAPP is the only integrated general education assessment that can be completed in 40 minutes and covers all content areas. It is the only general education assessment that allows both norm-referenced and criterion-referenced scores. The MAPP has two forms of delivery: paper and pencil or online. They now also have an online unproctored version. MAPP also allows for custom comparisons with other institutions.
The CAAP consists of modules, and you can choose to use some of the modules or all of them. The CLA is a scenario-based test and is meant to be institution oriented; therefore, it is not diagnostic at the student level.
MAPP informs curriculum and can be mapped to your outcomes. It costs much less than the other two instruments. From a diagnostic point of view, MAPP clearly differentiates itself at the student level when the long form or two-hour version is used. If you are looking for a snapshot of an institution, the others work. Tom encouraged us to download the Culture of Evidence document from their web site. He also noted that 75-80% of institutions who use the MAPP use the short form.
Question: Do others have problems with students completing the online version?
Mike Easter and Delores Honey said that their students use the online version and are doing well. Mike indicated that the students liked getting their scores immediately and that their scores have improved since using the online version. A request was made that students be allowed to see their percentile rank at the same time as their raw score on the online test.
Tom talked about some new products. The newest addition is the unproctored online MAPP. This version was developed in response to the distance learning community or the blended community; however, it is not just for distance learning. This version is available 24 hours a day. Comparative data for it, though, is not yet available. Proctored and unproctored forms can be used together or interchangeably, and score reports can be combined across versions. There is no difference in the cost of proctored or unproctored online tests.
The iSkills™ Assessment helps to ensure students’ readiness to succeed in school and the workforce by measuring their ability to use critical thinking skills in a digital environment.
iSkills is a scenario-based interaction. It uses real-life scenarios and is not a test of the ability to use databases or search engines. It is designed to align with the ACRL library standards. There are two versions: Advanced level for students who have completed at least two years of higher education and Core level for high school juniors, seniors, college freshmen, and adult learners. This instrument contains 14 three-to-five minute scenarios. It provides an individual score report and an institutional skill area report. It is the only performance based assessment that measures students’ abilities to use critical thinking and to manage, evaluate, and convey information within an information technology environment. There is a demo on their webpage. The cost depends on volume, but is between $22 and $25. There is a minimum of 50 tests to be ordered.
At this time, MSU has a required, one credit course to address information literacy. Missouri Southern embeds these competencies into their classes.
The ETS website has some case studies of best practices in this area. On November 27, ETS is offering a webinar from Georgia State University related to best practices. ETS now has webinars on their website that can be downloaded at any time.
Questions: What support does ETS provide faculty in different disciplines in developing locally authored tests? Faculty would like to be able to customize the MFT and add their own questions.
Tom - The MFT is a national exam, but ETS might take a look at developing a bank of questions that users could draw from to match their goals more closely. Tom also mentioned the Student Instructional Report (SIR II). It contains 45 questions in eight areas and can be delivered through Blackboard. The SIR II assesses course and instruction outcomes and measures students’ in-course experiences. State Fair has used this instrument for years.
11:55 Next Meeting
The next MAC meeting will be held in conjunction with the articulation meeting which is in Columbia on January 30 at the Holiday Inn Select. The MAC meeting will be on January 29 at 1:00 p.m. Cindy Heider will host. Details and location will be sent later.
12:00 Information Related to the Entry Level Competencies
Hillary reported that the initial drafts of the work groups for the entering level competencies are on the DHE website at www.dhe.mo.gov/casinitiative.shtml. They have begun work on the exit level competencies. The initial drafts are due January 4. Fifty of the 350 volunteers are K-12 personnel. Communication has been held with MASA and MSTA, among other groups. DHE will have entry level competencies and exit level competencies for 13 courses by 08-09. However, this doesn’t mean that they will be operationalized by then. They are not looking at state-wide assessment at this time. They will seek input on assessment practices.
Question: There is concern on some campuses related to the general education matrix and its relationship to the competencies being developed. Is there an attempt to cross reference or align the matrix competencies and the new exit competencies?
Hillary - The legislative intent has been to try to give some understanding of the students who transfer courses without completing the 42-hour general education block. Part of the intent of this work is to support students who transfer individual courses. The understanding is that if we define competencies and best practice in operationalizing the competencies, students who complete these courses should have reached the competencies. The 42-hour block will still be used for those students who complete it. DHE was asked by the group to provide a short statement related to the current use of the general education matrix and its relationship to the competencies being developed.
Hillary noted that they were not going to use common course numbering or common syllabi, so there wouldn’t be a common assessment.
A question was raised about performance measures. Jeremy shared that he was told that the Higher Education Funding and Finance group is exploring potential measures. A new strategic plan is also being drafted. The purpose of this group is to formalize strategies and goals for the system of higher education in the state, but also to provide a foundation for work on performance reporting and measures mandated by SB389.
12:31 Additional Note
Jeremy shared that one school is looking for a speaker for a faculty professional development workshop on January 14. If anyone is interested, they are to let him know.
Pat Gillman is trying to put together a Missouri Consortium of AQUIP schools. They are working on some topics that they think will be of interest to other schools. Anyone interested in being a part of this should contact Pat.
Delores Honey shared that they are having a reception for Charles Van Middlesworth at Penn Valley Community College on Monday, November 19. He was also honored with the Julia Duckwall Scholarship.
12:35 Adjournment
