Chapter 1 Introduction
1.1 Rationale for the study
1.2 Objectives of the study
1.3 Organization of the thesis
1.4 Definition of terms
1.4.1 Online
1.4.2 Marking
1.4.3 Online marking
1.4.4 Online Marking System (OMS)
1.4.5 Local Area Network (LAN)
Chapter 2 Research Questions and Methodology of the
Study
Chapter 3 Issues in the Direct Testing of EFL/ESL
Writing Ability
3.1 Introduction
3.2 What is a direct writing test?.
3.3 EFL/ESL writing ability: What shall we test?
3.4 Issues in validity
3.4.1 What is validity?
3.4.2 Types of validity
3.5 Issues in reliability
3.5.1 What is reliability?
3.5.2 Methods of judging reliability of writing assessments
3.6 The relationship between validity and reliability
3.7 Four components of a direct writing test
3.7.1 The task
3.7.2 The writer
3.7.3 The scoring procedure
3.7.4 The rater
3.8 Washback
3.8.1 Washback in general
3.8.2 Washback of direct tests of writing
3.9 Practicality
3.10 Summary
Chapter 4 The CET Writing Test
4.1 Introduction
4.2 The writing test required by the CET
4.2.1 A direct test
4.2.2 Positive washback
4.3 The scoring of CET compositions
4.3.1 The scoring approach currently adopted
4.3.2 Procedures involved in scoring CET essays
4.3.2.1 Scoring Principles and Marking Scheme
4.3.2.2 Range-finders and sample essays
4.3.2.3 Rater training
4.3.2.4 Rating process
4.3.2.5 Monitoring raters' scoring during the scoring
sessions
4.3.2.6 Recording essay scores
4.4 Computer-aided adjustment of writing scores
4.5 Discussion
Chapter 5 The First Experimental Study
5.1 Introduction
5,2 Compositions
5.3 Participants
5.4 Data collection procedure
5.5 The introspection and retrospection studies
5.5.1 Introduction
5.5.2 Data elicitation
5.5.3 Tape transcription
5.5.4 Data analysis
5.6 The questionnaire studies
5.6.1 Design of the questionnaires
5.6.2 Analysis of questionnaire responses
5.7 Findings from the introspection, retrospection and
questionnaire studies
5.7.1 Issues and problems in rating CET essays online
5.7.2 Decision-making behaviors while rating CET-4 essays.
5.7.3 Summary of comments made by the raters on essays
5.7.3.1 Overall summary
5.7.3.2 Variations in raters' comments
5.7.4 Essay elements' influences on raters' decision-making.
5.7.5 Elements of good CET essays in the raters' eyes
5.8 Analysis of writing scores
5.9 Summary and discussion
5.9.1 About the issues and problems involved
5.9.2 About the raters' scoring decisions
5.9.3 About the writing scores
Chapter 6 The Second Experimental Study
6.1 Introduction
6.2 Compositions
6.3 Participants
6.4 Data collection procedure
6.5 Problems encountered
6.6 Data analysis
6.7 Results
6.8 Summary
Chapter 7 Design of the OMS
7.1 Introduction
7.2 Literature review on online marking of compositions
7.2.1 Automated scoring of essays
7.2.1.l Overview of four major automated scoring methods
7.2.1.2 Analysis of the four major automated scoring methods
7.2.1.3 Summary
7.2.2 Online scoring of essays by human raters
7.2.2.1 Overview of online scoring of essays by human raters
7.2.2.2 Empirical research on online scoring of essays by human raters
7.2.2.3 Summary
7.3 A preliminary model of marking essays online
7.4 Overview of the CET Online Marking System (OMS)
7.4.1 The data management module
7.4.l.1 Basic information management
7.4.1.2 Essay management
7.4.1.3 Search and report
7.4.2 The training module
7.4.3 The rating module
7.4.4 The monitoring module
7.5 Operation of the OMS and the rater interface
7.5.1 Overview of the operation of the OMS
7.5.2 The OMS rater interface
7.6 Main features of the CEI- OMS
7.6.1 Random distribution of scripts
7.6.2 Efficient score recording
7.6.3 Online real-time monitoring of scoring
7.6.4 Quality control of raters
7.6.4.1 Adherence to the CET Scoring Principles and Marking Scheme
7.6.4.2 Rater training
7.6.4.2.1 Compulsory training
7.6.4.2.2 Individual rater's self training
7.6.4.2.3 Forced training
7.6.4.3 Online discussion
7.6.4.4 Back-reading and score revising
7.6.4.5 Time control
7.7 Advantages of the CE-I OMS
7.7.1 Real and efficient random distribution of scripts at the
national level
7.7.2 Real-time online monitoring of raters
7.7.3 Assured quality control of scoring
7.7.4 Overall efficiency
7.7.5 Efficient and economical storage of scripts
7.7.6 Express retrieval of scripts and scores
7.7.7 Efficient management and potential utilization of test data for research
7.8 Limitations of online scoring and solutions
7.9 Summary
Chapter 8 The Third Experimental Study
8.1 Context of the experiment
8.2 Participants
8.3 Compositions
8.4 Data collection
8.4.1 Step1: Online marking
8.4.1.l The first round ordme marking
8.4.1.2 The second round onlLne markLng
8.4.2 Step 2: Conference marking
8.5 Data analysis
8.6 Results
8.7 Summary and discussion
Chapter 9 Data Analysis Using FACETS
9.1 FACETS and method
9.2 The first approach: comparison of rater severity and consistency from the online setting and the conference setting
9.2.1 Rater severity and consistency: the online setting
9.2.1.1 Rater severity: the online setting
9.2.1.2 Rater consistency: the online setting
9.2.2 Rater severity and consistency: the conference setting
9.2.2.1 Rater severity: the conference setting
9.2.2.2 Rater consistency: the conference setting
9.2.3 Comparison of rater severity and consistency in two settings
9.2.4 Comparison of rater severity change between two settings
9.3 The second approach: bias analysis
9.3.1 Bias analysis: rater by essay interactions
9.3.2 Bias analysis: rater by setting interactions
9.4 Conclusion
9.5 Discussion
Chapter 10 Summaries, Discussions, Implications and Recommendations
10.1 A refined model of online scoring of CET essays and its main features
10.2 Benefits proceeding from online scoring
10.3 Practicality
10.4 Scoring quality
10.5 Raters' comments
10.6 Suggestions for the improvement of the Online Marking System
10.7 Implications for other writing tests
10.8 Suggestions and recommendations for future research
10.8.1 Suggestions for future research in online marking of compositions
10.8.2 Recommendations for future research in EFL writingassessment
10.9 Theoretical and practical significance of the study
References
Appendices
后记
