Description
Within this study, an examination of 2- and 4-year institutions that administer the Educational Testing Service (ETS) Proficiency Profile and how they incorporate evidence from direct assessment measures of student learning into their institutional decision-making, effectiveness, and planning practices was conducted. Additionally, this study explored how assessment and effectiveness leaders at these institutions, if they had their preferences, would like to integrate evidence and results from direct assessment measures and the ETS Proficiency Profile into those decision-making practices.Data were collected via a self-report, online survey containing questions about barriers, gaps, and tensions between external accountability expectations and an institution’s own improvement efforts, how and how often direct and indirect measures of student learning are used in decision-making, levels of confidence and effectiveness with results of the ETS Proficiency Profile, other general knowledge and skills measures and institution may use, types of direct measures of student learning that are used and how evidence from those measures inform institutional decisions, and institutional use of the ETS Proficiency Profile and its results. The final sample was comprised of 64 respondents, which consisted of assessment and institutional research leaders from postsecondary institutions in the United States that administer the ETS Proficiency Profile. With a sample size of 64 and no assumptions about the population’s distribution posited, a non-parametric test was used to analyze ordinal data collected between 2- and 4-year institutions. Variables that were not significant were presented using descriptive analyses. The descriptive approach facilitated an adequate and accurate interpretation of findings to describe present conditions. Document analyses were also conducted to substantiate reliability and validity of those survey data collected. Using Senge’s (2006) organization learning theory to ground this study, results indicated that 2- and 4-year institutions that administer the ETS Proficiency Profile incorporate direct assessment evidence into their decision-making, effectiveness, and planning practices. Assessment leaders at these institutions engage in organizational learning as they actively utilize data, information, and inquiry methods to create approaches to organizational work; however, those leaders also indicated they would like to more widely integrate evidence from direct assessment measures and the ETS Proficiency Profile in their decision-making and planning practices to better determine the potential for institutions to meet requirements for accreditation and accountability initiatives; evaluate and inform teaching and learning; and highlight strengths and areas of improvement.
