Office of Institutional Research and Effectiveness

Institutional Effectiveness Report Examples

The following examples are excerpts from various IE reports from across campus that scored well in IE Committee feedback reports for outcomes, measures, evidence, and use of results.

Expected Outcomes (Column 1)

Each year, units identify their desired outcomes of what they want students to think, know, or do.

Industrial Engineering (BS), College of Engineering

Industrial Engineering uses standards provided by its specialized accrediting body to assess its students.

Expected Outcome Assessment Procedure Criteria for Success
(Column 1) (Column 2)

Design with Constraints -Students will have the ability to design a system, component, or process to meet desired needs within realistic constraints such as economic, environmental, social, political, ethical, health and safety, manufacturability, and sustainability

Students in IE 3323 Manufacturing Processes will complete a Design Project Report.  Evaluate how well the reported system design takes into account the following aspects: (1) economic issues – this involves assessing the validity of their costs analysis of alternatives: materials, processes and equipment (2) safety of workers – this involves assessing how well safety issues are addressed in the selection of materials, and the specification of processes and labor requirements (quantity and skills) for machines and (3) manufacturability of the product – this involves assessing how well they evaluated possible design changes to recommend for simplifying manufacturability. The average score will be 75% or higher on the report.

Business Administration (MBA), College of Business

This example from Business incorporates its specialized accrediting standard into their assessment of a graduate level accounting class.

Expected Outcome Assessment Procedure Criteria for Success
(Column 1) (Column 2)

MBA students will demonstrate analytical skills and reflective thinking when applying managerial principles, analytical methods, and problem solving techniques to achieve business goals. (AACSB Goal 1 a/b/c)

Students will demonstrate analytical skills through an exercise in which they evaluate a firm’s financial condition based on appropriate analysis through an exercise in ACC 8112 Financial & Accounting Report Analysis. (AACSB Goal 1a).

Given a set of financial statements, students will:

  1. compute the return on assets and the components of the return on asset
  2. compute the return on equity and the components of the return on equity
  3. identify the appropriate measures used for liquidity analysis and evaluate them
  4. identify, measure and interpret the operating cycle
  5. identify the effect of transactions on the current ratio.

The exercise is measured with a mix of multiple choice questions and problems on exams.

The average score will be 75% or higher on the report.

Health Promotion & Wellness, Student Affairs

After meeting as a department and brainstorming on its strengths and weaknesses, this unit identified its most important outcomes. The following outcome not only supports this office's own goals, but it also supports the Division of Student Affairs' goals and objectives.

Expected Outcome Assessment Procedure Criteria for Success
(Column 1) (Column 2)

Students will realize the relationship between well-being and success.

HPW will implement programming, social media messages, social norming, motivational interviewing in drug, tobacco, alcohol and other specific health/wellness segments. The office will then conduct student focus groups every semester. The purpose of the focus group is to gauge whether the majority of students recognize the relationship between well-being and success.

Assessment Measures & Criteria for Success (Column 2)

Units must identify how they will evaluate to what extent they have fulfilled their outcomes. The units in this section have set assessment procedures and criteria for success that challenge them to make improvements.

Anthropology (BA), College of Arts & Sciences

Anthropology (BA) used a research paper as demonstration for its students’ research skills. Previously, the unit had used small group projects and decided to move to individual assignments instead; therefore, the assessment procedure was modified to enhance student learning.

Expected Outcome Assessment Procedure Criteria for Success
(Column 1) (Column 2)

Students will demonstrate research and application skills.

Students will design and conduct an original research project (20-25 page paper) which involves collection and analysis of ethnographic data. 80% of the students will score 80% or higher on the research paper.

Landscape Architecture (BLA), College of Ag & Life Sciences

In this example, Landscape Architecture uses a nation exam for its assessment of its students’ disciple-specific content knowledge. The external benchmark encourages the program to continue improving student learning with a comparison to peer institutions.

Expected Outcome Assessment Procedure Criteria for Success
(Column 1) (Column 2)

Students will demonstrate knowledge in the landscape architecture discipline

BLA students will take the national Landscape Architectural Examination (L.A.R.E.). 75% will pass the exam at or above the rate of the National Passage rates.

Building Construction Science (BS), College of Architecture, Art, & Design

This example uses an external review panel to score its students’ work on construction-related problems based on a faculty-developed rubric. Having a panel review student learning decreases subjectivity of the grading process and provides triangulation for other assessment data.

Expected Outcome Assessment Procedure Criteria for Success
(Column 1) (Column 2)

Students will be prepared for careers in construction or construction-related fields.

Students will complete a series of construction design problems iteratively in BCS 4126 Building Construction Studio 6. The evaluation will be scored by an external review panel. 80% of the students will score at least 70 out of 100 on a faculty-developed rubric.

Results & Use of Results (Columns 3 & 4)

The use of results column poses the most confusion for the assessment process. Rather than determine future changes or activities, the units are encouraged to reflect on previous assessment cycles and note what improvements were made as a result of the assessment process. Units often use both columns 3 and 4 to describe improvements in the data, which is why they are discussed together in this section.

Communication, College of Arts & Sciences

Communication has broken out its assessment by concentration to get a holistic view of the program. In this excerpt, the instructor realized that external factors were decreasing student scores on an assignment and is working to correct those issues.

Expected Outcome Assessment Procedure Criteria for Success Results Use of Results
(Column 1) (Column 2) (Column 3) (Column 4)
Students will understand the theoretical, ethical and practical processes of producing print newspapers and similar communication pieces. (Journalism Concentration) Students enrolled in CO 3443 (Advanced News Writing) will be required to research, write and edit original news stories with informative, accurate, and interesting content that adheres to professional standards. Students will gather information for and write six to eight original stories using proper grammar and accepted structures specific to newspapers and magazines and related websites. Students will be evaluated using the following criteria: 10% idea development, 30% completeness of information gathering, 30% use of effective story structures, 30% use of grammar and Associated Press style. 85% of students will show competency in gathering information and writing stories worthy of newspaper or magazine publication.

79% (23/29) of students showed competency in gathering information and writing stories worthy of newspaper or magazine publication.

Results do not reflect fully reflect competency as a result of some students not turning in assignments on deadline.
Instructor is placing emphasis on students turning all assignments in on time.

Agriculture (MS) – Poultry Science, College of Ag & Life Sciences

Poultry Science modified a course to better prepare students for their theses.

Expected Outcome Assessment Procedure Criteria for Success Results Use of Results
(Column 1) (Column 2) (Column 3) (Column 4)
Students will demonstrate mastery level of poultry science fundamentals. Students will complete an oral component of their thesis defense on their first attempt by demonstrating effective oral exam arguments, knowledge of the subject area, and organizational skills as evaluated with a rubric. The rubric scale goes from 1-3 (does not meet expectations, meets expectations, exceeds expectations) in the four categories: overall quality of presentation, overall breadth of knowledge, quality of response to questions, and overall assessment. At least 80% of the students will score 8 points or higher out of a 12 points rubric.  100% (n=3) scored 8 points or higher out of a 12 point scale rubric. The average number of total points received was 9.6.  The averages for each category were as follows: overall quality of science – 2.3, contribution to discipline – 2.4, quality of writing – 2.4, overall assessment – 2.3. To better prepare students for writing the literature review section of their thesis, the Graduate Seminar in Poultry Science course (PO 8011) was modified to included development of an annotated bibliography.  Students were required to provide 5 annotated citations every week during the semester that were then peer-reviewed by other students in the course.  Students were then required to accept or rebut the suggested corrections from the peer review. The final revised annotated bibliography was due at the end of the semester to the professor for evaluation.

Finance (BBA), College of Business

In Column 4, a professor recommended a curricular standard that would challenge students earlier in their academic program so that they were better prepared for their senior year.  

Expected Outcome Assessment Procedure Criteria for Success Results Use of Results
(Column 1) (Column 2) (Column 3) (Column 4)
Students will demonstrate competency in the use of financial concepts. Students will be assessed through four pre-identified, embedded test question groups in Senior Seminar 75% of all students will demonstrate proficiency on all four questions where proficiency is defined by the attached rubric.  A total of 57 students were enrolled in FIN 4243.  The percentage of students demonstrating proficiency for individual concepts in Analysis, Capital Structure, Cost of Capital, and time value of money were 14%, 44%, 47%, and 63% respectively. Students do not appear to be retaining information from previous courses, which is putting a strain on FIN 4243.  The current instructor recommended that the department require all students to have earned a grade of B or higher in both FIN 3123 and FIN 4223 as prerequisite for FIN 4243.  The department is also discussing a comprehensive exit exam for the degree, which may motivate students to retain the important concepts.

Forestry (BS), College of Forest Resources

In this excerpt, a faculty committee met to address strengths and weaknesses of student learning based on the state exam. Although the students' scores are increasing every year, the faculty committee continues to analyze what areas could be strengthened.

Expected Outcome Assessment Procedure Criteria for Success Results Use of Results
(Column 1) (Column 2) (Column 3) (Column 4)
Graduates from the Forestry major will be able to apply their knowledge of forestry as a professional forester. Students will take the State of Mississippi BORF exam. A mastery level of 70% or higher will be expected from our students on this exam. In our capstone course FO 4423 Professional Practice, 28 of 33 students took the exam and a total of 60.7% (17 of 28 students) scored 70% or higher. This compared to 2014-2015 where 30 of 33 students took the exam and 19 of 30 had passing scores of 70% or greater. The overall average for students taking the exam was 71.8%, an increase over the previous year (71.8% versus 70.7%). Students passing the exam had an average score of 75.2%. This was an increase in success over the previous year (75.2% versus 74.2%). Despite some improvement, students are falling below the Criterion.

The root cause of failing to meet the Criterion related to certain subject areas. The Department analyzed test scores by nine different categories (which is the exam structure). In 2015-2016, 3 categories were above 70% (Ethics, Forest Management, Pest Management), and 6 were below (Silvicuture, Pest Management, Spatial Technology, Wildlife Management. Economics, Harvesting, Measurements).

When the BORF examination was redeveloped in 2014-2015, the Department and BORF recalibrated the process to ensure all questions were reflective of what the students should be learning as well as being appropriate for honing their job skills. When instructors made course changes to improve performance on those subjects, students who received the improved instruction have now taken the BORF exam with the exception of Measurements. The Department is now going to revisit those instructors with the bank of questions related to their discipline to see what changes in instruction were made and to ask them to make sure each question has relevance for our students in regard to this exam.

Veterinary Medical Technology (BS), College of Veterinary Medicine

In Column 4, the instructor observed that a change in assignment deadlines improved student performance on a quiz.

Expected Outcome Assessment Procedure Criteria for Success Results Use of Results
(Column 1) (Column 2) (Column 3) (Column 4)
Students will understand the basic disease processes in domestic animal species. Students in CVM 3013 Small Animal Diseases and Management will be assigned 5 learning objectives for various disease processes. The students will be given a quiz over this information. The learning objectives pertained to basic information that a veterinary technologists would need to know about basic disease processes of domestic small animals. 90% of the students will score an avg of 7.5 pts or higher out of the possible 10 pts possible for each of the 5 learning issues.

Quiz #1: 84.2% (16 out of 19 students) scored 7.5 (75%) or higher.

Quiz #2: 63.2% (12 out of 19 students) scored 7.5 (75%) or higher.

Quiz #3: 89.5% (17 out of 19 students) scored 7.5 (75%) or higher.

Quiz #4: 89.5% (17 out of 19 students) scored 7.5 (75%) or higher.

Based on the grades that were being made on quiz #2, the instructor decided to allow the students to submit the objective sheets after the lecture instead of before. This led to an increase in score on the quizzes due to the students being able to ask questions during class time. It is believed that this may have led to better understanding of the material. Since the drastic improvement was seen after the submission guidelines were changed, next year the objective sheets will not be due until after the lecture on the specific objectives is complete. This allows students to have more time to understand and ask questions about the topics covered, which facilitates retention and performance.

The objective sheets are also returned to the students with explanations of why an answer was wrong, and what the correct answer is. Also, if the whole class misses a concept, the instructor sends an email to the class and follows it up with an in-class discussion of the concept to clear the misconceptions.

International Institute, Academic Affairs

This example shows a collaboration of units across campus to enhance students’ experiences with different cultures.

Expected Outcome Assessment Procedure Criteria for Success Results Use of Results
(Column 1) (Column 2) (Column 3) (Column 4)
The International Institute will assist students in developing a knowledge and awareness of different cultures and global perspectives across MSU campus. Cultural events will be hosted to promote knowledge and global perspectives across MSU campus as well as provide awareness of cultural differences. At least 90 student cultural events will be held annually providing awareness of cultural differences and global opportunities. In 2015-2016, a total of 131 student cultural events were held. This included weekly Conversation Connections meetings, International Education Week events (multiple), Intro to Global Studies (ISE 1103) partnerships, Study Abroad Fairs, and a number of outreach functions. Coordinators collaborated with other units on campus to provide new cultural experiences, including pairing with Holmes Cultural Diversity Center's peer mentors. Efforts were made to market programming by using social media, the calendar of events, and the International Institute's website, which opened events to a wider audience of domestic and international students.

Institutional Research & Effectiveness

The office has met its criteria for success for numerous years. The Institutional Effectiveness staff met to discuss ways that the office could improve in the future. Based on data from previous IE Committee cycles, the office determined that columns 3 and 4 needed the most work across all IE reports. This example highlights a change in the unit’s outcome and assessment procedures in response to previous years’ use of results.

Expected Outcome Assessment Procedure Criteria for Success
(Column 1) (Column 2)

Assessment units will engage in meaningful institutional effectiveness initiatives.

The Institutional Effectiveness staff have created a rubric, which determines whether assessment unit measures could be classified as excellence or quality assurance. The staff will review all assessment plans using this rubric to determine individual scores and then will compare individual scores to arrive at an overall validated score for each unit. Using this metric, staff can then better assist units during help sessions. During the 2015-16 year, 328 units completed assessment plans. For the 2016-17 year, Institutional Effectiveness hopes to increase the number of measures in the excellence category by 10%.

 

Contact Information

269-A Allen Hall
Mailstop: 9712

Phone: 662.325.3920
Fax: 662.325.3514
oir@ir.msstate.edu