etips logo

ETIPS

Educational Theory into Practice Software

Embody Theory and Research

Home

Case Topics

Research & Evaluation

Try ETIPS

PlanMaps and AutoMagic

Papers

Automated Essay Score Predictions as Formative Assessment Tools
Abstract: Survey data and focus group responses provide insights into students' experiences with and attitudes toward automated essay scoring software. This formative assessment tool was available for their use while completing short answer essays in online cases. Students' reactions to the scorer were both positive and negative, but were strong. They reported that a numeric score, even in conjunction with a rubric, was not sufficient information to know how to revise their work. The accuracy of the scoring software was low, which led some students to lose interest in revising their work after receiving a score prediction. The implication of a reliable automated essay scoring tool in use within network-based online learning environments is that it makes possible formative feedback, and presumably improved performances, on complex sets of skills.
( Dowload this paper as a PDF.)

Experimental Evidence on the Effectiveness of Automated Essay Scoring in Teacher Education Cases
Abstract: This study tests the impact of an automated essay scorer (AES) which provides formative feedback on essay drafts written as part of a series of online teacher education case studies. A total of 70 preservice teachers in four teacher education courses were assigned to complete two cases. Each student was randomly assigned to either a condition where the scorer was available (experimental condition) or a condition where the scorer was unavailable (control condition). Those students in the experimental condition submitted higher-quality final essays and conducted more efficient searches of the case than students in the control condition. Essay scores were positively associated with the number of drafts submitted to the scorer for formative feedback.
( Download this paper as a PDF.)

Expert Teachers' Technology Integration Knowledge and Skills
Abstract: A cognitive task analysis of twenty experienced in-service teachers to report the key knowledge, concepts and skills they draw upon to plan technology-integrated instruction. Data, collected through focus groups, show expert technology-integrating teachers' focus on curriculum goals, consider their students' technology skills and interests, and find models of the technology product expected of students important for guiding their pupils' understanding of their assignment. How much teachers plan for logistics and trouble-shooting depends on the level of access to technology in their school.
Download this paper as a PDF.)

Formative Feedback Via An Automated Essay Scorer: Its Impact On Learners
Abstract: The purpose of this research is to analyze preservice teachers’ use of and reactions to an automated essay scorer as a means of formative feedback in an online, case-based learning environment. Data analyzed include pre- and post-semester surveys, a user log of students’ actions within the cases, instructor-assigned scores on final essays, and in-depth interviews with four selected students. Analysis illustrates that extensive use of the scorer was associated with improved essays as measured by their final human-assigned scores.  Study findings also suggest factors that must be considered in using technology to provide formative feedback including the design, nature, accuracy, and specificity of feedback, all of which were found to impact student performances and experiences. These outcomes boost confidence in the potential utility of instructional technology to aid in formative assessment within the ETIPS cases learning environment.
( Download this paper as a PDF.)

Online Simulations as a Strategy for Instruction on Technology Integration
Abstract: A mix of faculty interviews, site observations, and student pre and post-surveys were used to examine the implementation of eTIP cases during the 2002-2003 academic year in teacher education courses. Case implementation varied in the extent to which the cases were discussed in class, instructors integrated the cases into other course topics, and instructors offered formative feedback on student work with cases. Hierarchical linear modeling was used with data from 243 students nested in 18 different course sections. Individual student ratings of case usefulness in learning about technology integration were predictive of gains in self-assessed technology integration skill. The impact of case ratings varied by class although the models employed did not correctly specify class effects.
( Download this paper as a PDF.)

Technical Report 1: Characteristics of Relevancy Index and Number of Steps
Abstract: The work of 449 preservice teachers using the ETIPS cases in 2002-2003 is examined with a focus on how their information searches within the cases resemble those of experts (relevancy index scores) and how this is related to the extensiveness of their search (number of steps). In general, the relevancy index scores increased over multiple cases and were negatively related to number of steps taken in a case. Users made an average of 31 separate steps to access information in a case, an average which decreased over multiple cases. Individual search strategies for achieving high relevancy index scores included broad, sweeping searches of available information and shorter, focused searches of relevant information only.
( Download this paper as a PDF.)

Technical Report 2: Analysis of ETIPS Case Essay Scores
Abstract: Characteristics of scores assigned to ETIPS case essays were explored. Two essay scoring rubrics were tested in 2002-2003 with 133 preservice teachers in 12 courses using the cases. Instructors made greater use of the full range of values (1-3) for each criterion with a seven-score rubric employed in fall 2002 than a three-score rubric employed in spring 2003. Overall, individual scores were strongly associated with one another for a student on a given case. Scores were only moderately correlated across cases for the same individual. There was no evidence of systematic growth in essay scores over time with either rubric.
( Download this paper as a PDF.)

Technical Report 3: The Relationship between ETIPS Case Essay Scores and Relevancy Scores
Abstract: The statistical relationship between scores assigned to ETIPS case essays and measures of the relevancy of the user’s search to the case question are explored. The sample includes data from 117 student users in ten course sections over two semesters. There was no significant relationship between essay scores and the relevancy index (sum of relevancy weights divided by the number of steps taken in the case). There was a weak relationship between the number of steps taken in a case and essay scores such that highly rated essays reflected at least a minimal search of the case. An alternative measure of relevancy, the number of unique items relevant to the case question accessed, did have a statistically significant relationship to essay scores. This relationship was strongest in later rather earlier cases completed by the user.
( Download this paper as a PDF.)

Technical Report 4: ETIPS Essay Scores, Relevancy, and Case Search
Abstract: The appropriateness of a 2 x 2 typology of case users was explored using data from the first year of field testing. The typology was based on the interactions between two measures: quality of case essays and degree of expertise in case search (relevancy). The typology includes four types of users: (1) those having a high quality essay and relevant search; (2) those having a low quality essay but relevant search; (3) those having a high quality essay and irrelevant search; and (4) those having a low quality essay and irrelevant search. The validity of the typology was confirmed through an examination of the case search characteristics of each of the four types of users. Those having an irrelevant search typically accessed much less information than those having a relevant search although they tended to focus heavily on information about available technology (without hitting relevant items). There were users, however, who could offer a thoughtful essay response without relying on relevant case information. Nevertheless, access to relevant case information does not guarantee that users will translate such information into a high quality essay response about the case.
( Download this paper as a PDF.)

Technical Report 5: User Characteristics and Case Outcomes
Abstract: A series of statistical tests were carried out to examine the relationship between individual user characteristics (age, gender, student teaching experience, technology skill) and student performance in the ETIP cases. Measures of case outcomes include instructor-assigned essay scores, relevancy of case search, extent of case search, and proportion of search devoted to different categories of case information. Overall, individual student characteristics have a weak and inconsistent relationship to how students work with the cases. By comparison, between class variance accounted from 9 to 39 percent of the variance in case outcomes.
(Download this paper as a PDF.)