Automated Essay Scoring: A Cross-disciplinary Perspective

Download Citation on ResearchGate | On Jan 1, , M. D. Shermis and others published Automated Essay Scoring: A Cross-Disciplinary Perspective }.
Table of contents

Page 1 of 1 Start over Page 1 of 1. Do you want to improve your thinking, decision making and solve problems faster?


  • Automated Essay Scoring: A Cross-disciplinary Perspective - Google Books?
  • JSTOR: Access Check!
  • No Results Found.
  • Great Managers Are From Earth: Six Down-to-Earth Rules for New Managers!

The Magic of Accelerated Learning: Discover Strategies for Effective Learning, Imp Discover advanced methods to learn better, memorize more and master any skill faster. Learn the important content and secret test taking strategies to pass the Praxis Biology Test. Master Your Time in 10 Minutes a Day: When you spend a day you have one less day to spend.

So make sure you spend each one wisely. Learn to Read with Great Speed: Save your time by reading more in less time. Exercise in minute chunks. Your employees are key to your company's success.

Learn how to engage them via relevant, productive and memorable brainstorming workshops. Do you find other people are making progress much faster than you? Are you getting left behind? Revolutionize your learning capabilities today! Routledge; 1 edition December 1, Language: Related Video Shorts 0 Upload your video. Try the Kindle edition and experience these great reading features: Share your thoughts with other customers. Write a customer review. There was a problem filtering reviews right now.

Please try again later. This book is a welcome addition to the growing body of literature on automated essay scoring, also known as automated essay evaluation technology AEET. I am writing my dissertation on this subject from a composition instructor's perspective , and the book has helped me tremendously. Automated Essay Scoring is organized into five major sections: These students were required to take THEA as part of their developmental writing class to exit the program. The current study utilized a quantitative correlational study design. Human scoring and automated essay scoring were selected as variables for computing correlation coefficients.

Writing responses gathered from WritePlacer Plus test were graded by an automated essay scoring tool—IntelliMetric—as well as by trained human raters. For WritePlacer Plus , both the automated essay scoring tool and human raters assigned a holistic score for the overall quality of a writing response and analytic scores on five dimensions of each writing response—Focus, Development, Organization, Sentence Structure, and Mechanics.

Altogether, three sets of variables were examined in the correlational study, as indicated in Figure 1.

Automated essay scoring a cross-disciplinary perspective, Custom paper Academic Writing Service

Correlational study design model. Participants took the WritePlacer Plus test first, and their writing samples were graded by the IntelliMetric instantly. After the same group of students had taken THEA organized and proctored by the Testing Office at the college and after the THEA scores became available, the researcher obtained the score report. At this point, the database was screened, and students who had a WritePlacer score only or THEA scores only were deleted. After the screening, cases had both sets of scores and were kept in the SPSS database.

Automated Essay Scoring Versus Human Scoring: A Correlational Study

Then, the chosen papers were assigned a number ranging from 1 to The retrieved writing samples were each graded by two trained human raters, who were volunteers from the Developmental English Department of the college where the research was conducted. In addition, both of them received two more recent trainings. After the grading was finished, the results were entered into the SPSS database with the other sets of results. Table 1 displays the means, medians, and standard deviations for each scoring method.

The Spearman rank correlation coefficient tests were run separately for analyzing the overall holistic scores and for each set of dimensional scores SPSS The significance level was set at. The detailed results are presented in Table 2. The detailed results are displayed in Table 3. Results based on the correlational data analyses showed no statistically significant correlation between IntelliMetric scoring and human scoring in terms of overall holistic scores. This finding does not corroborate previous studies conducted by Vantage Learning, which reported strong correlations between IntelliMetric scoring and human scoring for overall ratings Elliot, The different results produced by the current study seem to indicate that the IntelliMetric scoring model built by a pool of essays written by a different student population may not be generalizable to the student population in South Texas.

The lack of significant correlation also raises the question whether IntelliMetric scoring can be consistent with human scoring at all times and in all situations. In terms of correlations between IntelliMetric scoring and human scoring in different dimensions of the essays, data analyses showed no statistically significant correlations in Dimension 1, 2, 3, and 5. However, a statistically significant correlation was found in Dimension 4 — Sentence Structure.

Secondary menu

These findings suggest that IntelliMetric seems to be more consistent with human scoring in assessing sentence structures, but not in assessing other dimensions. In general, findings from the study challenged the research results published by Vantage Learning, which demonstrated strong correlations between AES and human scoring. These findings also raised the question whether AES models built by writing samples from one student population were generalizable to writing samples from other student populations. If AES models are not generalizable, pending the confirmation of future studies, then it may be necessary for a specific AES model to be built for a specific student population.

In that case, the cost of using AES tools may become a question of concern. Furthermore, the results of the current study pointed out the possibility of AES being significantly correlated to human raters in assessing Sentence Structure rather than in content-related features. If this finding is true, pending the confirmation of further studies, then it may mean that AES tools can be utilized more specifically in assisting student writers with feedback on improving their sentence skills.

Finally, the serendipitous finding from the current study indicating significant correlation between the two teams of human raters may mean human raters are more consistent with each other in assigning essay scores than with AES tools.

Покупки по категориям

A finding of this nature, if confirmed by future studies, may also call into question the validity of AES tools. The correlational analyses, using the nonparametric test Spearman Rank Correlation Coefficient, showed that the overall holistic scores assigned by IntelliMetric had no significant correlation with the overall holistic scores assigned by faculty human raters, nor did it bear a significant correlation with the overall scores assigned by NES human raters.

On the other hand, there was a statistically significant correlation, with an effect size of medium coefficient, between the two sets of overall holistic scores assigned by the two teams of human raters. Spearman Rank Correlation analyses of dimensional scores showed a significant correlation between IntelliMetric scoring and faculty human scoring in Dimension 4 — Sentence Structure but no significant correlations in other dimensions.

On the whole, the results from the current study support the conclusion that IntelliMetric did not seem to correlate well with human raters in scoring essays and that findings published by Vantage Learning did not appear to be generalizable to the student population in South Texas. The discrepancies between the findings of the current study and those published by Vantage Learning may be attributed to the following factors:.

As the interest in adopting AES tools increases, and as the development of AES technologies undergoes rapid changes, they still hold a promising future for writing assessment programs; therefore, continuous research and investigation in the validity and generalizability of the AES tools are inevitable. Based on the findings of the current study, further studies should be conducted to determine the validity and generalizability of the AES tools.

In the interim, school administrators who make decisions about what assessment tools to use need to take the validity of AES tools into consideration.

Therefore, it is a matter of choosing between efficiency and quality of assessment methods. However, when the validity of the AES tools is still in question, the use of machine grading should be restricted to spelling checks and sentence skills feedback. The results here have important implications for English teacher education as well.

While English educators may want to expose pre- and in-service teachers to AES tools, their utility is limited at this point, and well-documented assessment strategies, like writing portfolios and writing conferences, and a keen awareness of the process writing approach should still be included in English education methods courses and in the methods repertoire of practicing English language arts teachers. Although an overwhelming grading load is often a reality for writing instructors, scholars such as Zinn have explored ways to ease the load.

Zinn suggested using student-generated grading criteria and focusing on a couple of special grading problems. Instructors should also make writing assignment topics clear, so that the end product will be easy to grade. Sample papers and specific grading criteria will also assist with the grading process.

Group responses and feedback to early drafts can also be used to help lighten the load. Too much commentary should be avoided. The means to achieve this end lies in the hands of human raters, rather than machines. Responding to and assessing student writing: The uses and limits of technology. Impact on higher education institutions. Handbook of test development, Second Edition.

A Growing Body of Knowledge. In the Handbook of Writing Research Eds. From Teacher Professional Development to the Classroom: Journal of Educational Computing Research, 51 1: In Ruslan Mitkov Ed. The Case of Noun-Noun Compounds. Using pivot-based paraphrasing and sentiment profiles to improve a subjectivity lexicon for essay data. Transactions of the Association for Computational Linguistics.

Sentiment Analysis Detection for Essay Evaluation.