CA2491238A1 - Automated essay annotation system and method - Google Patents

Automated essay annotation system and method Download PDF

Info

Publication number
CA2491238A1
CA2491238A1 CA002491238A CA2491238A CA2491238A1 CA 2491238 A1 CA2491238 A1 CA 2491238A1 CA 002491238 A CA002491238 A CA 002491238A CA 2491238 A CA2491238 A CA 2491238A CA 2491238 A1 CA2491238 A1 CA 2491238A1
Authority
CA
Canada
Prior art keywords
essay
sentence
discourse
feature
rhetorical
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CA002491238A
Other languages
French (fr)
Other versions
CA2491238C (en
Inventor
Jill Burstein
Daniel Marcu
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Educational Testing Service
Original Assignee
Individual
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Individual filed Critical Individual
Publication of CA2491238A1 publication Critical patent/CA2491238A1/en
Application granted granted Critical
Publication of CA2491238C publication Critical patent/CA2491238C/en
Anticipated expiration legal-status Critical
Expired - Fee Related legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/10—Text processing
    • G06F40/166—Editing, e.g. inserting or deleting
    • G06F40/169—Annotation, e.g. comment data or footnotes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/279—Recognition of textual entities
    • G06F40/289—Phrasal analysis, e.g. finite state techniques or chunking
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B11/00—Teaching hand-writing, shorthand, drawing, or painting
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B19/00—Teaching not covered by other main groups of this subclass
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B7/00—Electrically-operated teaching apparatus or devices working with questions and answers
    • G09B7/02—Electrically-operated teaching apparatus or devices working with questions and answers of the type wherein the student is expected to construct an answer to the question which is presented or wherein the machine gives an answer to the question presented by a student

Abstract

An automatic discourse analysis application (ADAA) (180) is used to automatically annotate an essay. The ADAA (180) includes a user interface (300), which receives and essays and forwards the essay to feature extractor (302) and receives the annotated essay from a discourse analysis modeler (318). The feature extractor (302) includes a position identifier (304), lexical item identifier (306), punctuation identifier (314), and a rhetorical relation identifier (316).

Claims (63)

1. A method for annotating an essay comprising:
identifying a sentence of an essay;
determining a feature associated with said sentence;
determining a probability of said sentence being a discourse element by mapping said feature to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.
2. The method according to claim 1, wherein said discourse element is at least one of:
title; background; thesis statement; main points; support; and conclusion.
3. The method according to claim 1, further comprising:
receiving said essay.
4. The method according to claim 1, wherein said feature comprises a positional feature.
5. The method according to claim 4, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
and modifying said entry to include data associated with said positional feature.
6. The method according to claim 5, wherein said positional feature comprises at least one of:
a sentence position, said sentence position being associated with a position of said sentence within said essay;
a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay;
a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of a paragraph within said essay.
7. The method according to claim 47, wherein said lexical feature comprises at least one of:
a category-specific cue, said category-specific cue being typically associated with a discourse element;
a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship.
8. The method according to claim 49, further comprising:
generating a rhetorical structure tree based on said flat file; and identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of:
a discourse structure, said discourse structure being typically associated with an elementary discourse unit;
a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising:
a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.
9. The method according to claim 8, wherein said rhetorical structure tree is mapped to a plurality of models and said probability being determined based on a voting algorithm.
10. A method of annotating an essay comprising:
identifying a sentence of an essay;
generating a flat file for said essay, said flat file including an entry for said sentence;
determining a positional feature associated with said sentence;
modifying said entry to include data associated with said positional feature;
identifying a lexical feature associated with said sentence;
modifying said entry to include data associated with said lexical feature;
identifying a rhetorical feature associated with said sentence;
modifying said entry to include data associated with said rhetorical feature;
determining a probability of said sentence being a discourse element by mapping said flat file to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.
11. The method according to claim 10, wherein said discourse element is at least one of:
title; background; thesis statement; main points; support; and conclusion.
12. The method according to claim 10, further comprising:
receiving said essay.
13. The method according to claim 10, wherein said positional feature comprises at least one of:
a sentence position, said sentence position being associated with a position of said sentence within said essay;
a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay;
a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of a paragraph within said essay.
14. The method according to claim 10, wherein said lexical feature comprises at least one of:
a category-specific cue, said category-specific cue being typically associated with a discourse element;
a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship.
15. The method according to claim 10, further comprising:
generating a rhetorical structure tree based on said flat file; and identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of a discourse structure, said discourse structure being typically associated with an elementary discourse unit;
a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising:
a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.
16. The method according to claim 10, further comprising:
identifying a punctuation associated with said sentence; and modifying said entry to include data associated with said punctuation.
17. The method according to claim 10, wherein said flat file is mapped to a plurality of models and said probability being determined based on a voting algorithm.
18. A method of annotating an essay comprising:
receiving an essay;
identifying a sentence of said essay;
generating a flat file for said essay, said flat file including an entry for said sentence;
determining a positional feature associated with said sentence, wherein said positional feature comprises at least one of:
a sentence position, said sentence position being associated with a position of said sentence within said essay;
a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay;
a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of a paragraph within said essay;
modifying said entry to include data associated with said positional feature;
identifying a lexical feature associated with said sentence, wherein said lexical feature comprises at least one of:
a category-specific cue, said category-specific cue being typically associated with a discourse element;
a general vocabulary cue, general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship;
modifying said entry to include data associated with said lexical feature;
identifying a punctuation associated with said sentence;
modifying said entry to include data associated with said punctuation;
generating a rhetorical structure tree based on said flat file;

identifying a rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of:
a discourse structure, said discourse structure being typically associated with an elementary discourse unit;
a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising:
a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures;
modifying said entry to include data associated with said rhetorical feature;
determining a probability of said sentence being a discourse element by mapping said flat file to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.
19. The method according to claim 18, wherein said discourse element is at least one of:
title; background; thesis statement; main points; support; and conclusion.
20. A process comprising:
training a first judge to identify a sentence within an essay as being a discourse element;
accepting a first annotation of said essay from said first judge;
evaluating said first judge based on a comparison of said first annotation to a second annotation of a second judge; and calculating an empirical probability based on said first annotation in response to said evaluation exceeding a predetermined value, said empirical probability further based on at least one of:
a positional feature of said sentence within said essay;
a category-specific feature of said sentence within said essay;
a lexical feature of said sentence within said essay;
a key term of said sentence within said essay; and a punctuation of said sentence within said essay.
21. A computer readable medium on which is embedded computer software, said software comprising executable code for performing a method of annotating an essay comprising:
identifying a sentence of an essay;
determining a feature associated with said sentence;
determining a probability of said sentence being a discourse element by mapping said feature to a model, said model having been generated by a machine learning application based on at least one annotated essay; and annotating said essay based on said probability.
22. The method according to claim 21, wherein said discourse element is at least one of:
title; background; thesis statement; main points; support; and conclusion.
23. The method according to claim 21, further comprising:
receiving said essay.
24. The method according to claim 21, wherein said feature comprises a positional feature.
25. The method according to claim 24, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
and modifying said entry to include data associated with said positional feature.
26. The method according to claim 25, wherein said positional feature comprises at least one of:
a sentence position, said sentence position being associated with a position of said sentence within said essay;
a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay;
a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of a paragraph within said essay.
27. The method according to claim 53, wherein said lexical feature comprises at least one of:
a category-specific cue, said category-specific cue being typically associated with a discourse element;
a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and a key term, said key term being typically associated with a discourse relationship.
28. The method according to claim 55, further comprising:
generating a rhetorical structure tree based on said flat file; and identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of:
a discourse structure, said discourse structure being typically associated with an elementary discourse unit;
a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures;,and a status, said status comprising:
a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures..
29. The method according to claim 28, wherein said rhetorical structure tree is mapped to a plurality of models and said probability being determined based on a voting algorithm.
30. An automatic essay annotator comprising:
means for identifying a sentence of an essay;
means for determining a feature associated with said sentence;
means for determining a probability of said sentence being a discourse element, said means for determining said probability being configured to map said feature to a model, said model having been generated by a machine learning application based on at least one annotated essay and said discourse element being at least one of title;
background; thesis statement; main points; support; and conclusion; and means for annotating said essay based on said probability.
31. The automatic essay annotator according to claim 30, further comprising:
means for receiving said essay.
32. The automatic essay annotator according to claim 30, wherein said means for determining said feature further comprises means for determining a positional feature.
33. The automatic essay annotator according to claim 32, further comprising:
means for generating a flat file for said essay, said flat file including an entry for said sentence; and means for modifying said entry to include data associated with said positional feature.
34. The automatic essay annotator according to claim 33, wherein said means for determining a positional feature comprises at least one of:
means for determining a sentence position, said sentence position being associated with a position of said sentence within said essay;
means for determining a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay;
means for determining a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and means for determining a relative paragraph position, said relative paragraph position being associated with a relative position of a paragraph within said essay.
35. The automatic essay annotator according to claim 59, wherein said means for determining a lexical feature comprises at least one of:
means for identifying a category-specific cue, said category-specific cue being typically associated with a discourse element;
means for identifying a general vocabulary cue, said general vocabulary cue being typically associated with a discourse structure; and means for identifying a key term, said key term being typically associated with a discourse relationship.
36. The automatic essay annotator according to claim 61, further comprising:
means for generating a rhetorical structure tree based on said flat file; and means for identifying said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of:
a discourse structure, said discourse structure being typically associated with an elementary discourse unit;
a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising:
a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.
37. The automatic essay annotator according to claim 36, wherein said means for determining said probability further comprises means for mapping said rhetorical structure tree to a plurality of models and said probability being determined based on a voting algorithm.
38. An automatic essay annotator comprising:
a feature extractor, said feature extractor comprising:
a position identifier configured to determine a position feature associated with a sentence of said essay, said position identifier further configured to generate a flat file, said flat file including an entry for said sentence, said entry including data associated with said positional feature;
a lexical item identifier configured to identify a lexical feature associated with said sentence, said lexical item identifier further configured to modify said entry to include data associated with said lexical feature; and a rhetorical relation identifier configured to identify a rhetorical feature, said rhetorical relation identifier further configured to modify said entry to include data associated with said rhetorical feature; and a discourse analysis modeler configured to determine a probability of said sentence being a discourse element, said discourse analysis modeler being configured to determine said probability by mapping said flat file to a model, said model having been generated by a machine learning application based on at least one annotated essay, said discourse analysis modeler being further configured to annotate said essay based on said probability.
39. The automatic essay annotator according to claim 38, wherein said discourse analysis modeler is further configured to determine said probability of said sentence being at least one of a plurality of discourse elements, said plurality of discourse elements including: title;
background; thesis statement; main points; support; and conclusion.
40. The automatic essay annotator according to claim 38, wherein said feature extractor is configured to receive said essay.
41. The automatic essay annotator according to claim 38, wherein said position identifier is further configured to determine at least one of:
a sentence position, said sentence position being associated with a position of said sentence within said essay;
a relative sentence position, said relative sentence position being associated with a relative position of said sentence within said essay;
a paragraph position, said paragraph position being associated with a position of said sentence within a paragraph of said essay; and a relative paragraph position, said relative paragraph position being associated with a relative position of said sentence within a paragraph of said essay.
42. The automatic essay annotator according to claim 38, wherein said lexical item identifier further comprises:
a category-specific cue identifier configured to identify a cue typically associated with a discourse element;
a general vocabulary cue identifier configured to identify a cue typically associated with a discourse structure; and a key term identifier configured to identify a key term, said key term being typically associated with a discourse relationship.
43. The automatic essay annotator according to claim 38, further comprising:
a punctuation identifier configured to identify a punctuation associated with said sentence, said punctuation identifier further configured to modify said entry to include data associated with said punctuation.
44. The automatic essay annotator according to claim 38, wherein said rhetorical relation identifier is further configured to generate a rhetorical structure tree based on said flat file and identify said rhetorical feature based on said rhetorical structure tree, wherein said rhetorical feature comprises at least one of:
a discourse structure, said discourse structure being typically associated with an elementary discourse unit;
a rhetorical relation, said rhetorical relation describing a manner of association between a plurality of said discourse structures; and a status, said status comprising:
a nucleus, said nucleus being associated with a relatively more important one of said plurality of discourse structures; and a satellite, said satellite being associated with a relatively less important one of said plurality of discourse structures.
45. The automatic essay annotator according to claim 38, wherein said discourse analysis modeler being further configured to map said rhetorical structure tree to a plurality of models and determine a probability of said sentence being a discourse element based on a voting algorithm.
46. The method according to claim 1, wherein said feature comprises a lexical feature.
47. The method according to claim 46, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
and modifying said entry to include data associated with said lexical feature.
48. The method according to claim 1, wherein said feature comprises a rhetorical feature.
49. The method according to claim 48, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
and modifying said entry to include data associated with said rhetorical feature.
50. The method according to claim 1, wherein said feature comprises a punctuation.
51. The method according to claim 50, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
identifying said punctuation being associated with said sentence; and modifying said entry to include data associated with said punctuation.
52. The method according to claim 21, wherein said feature comprises a lexical feature.
53. The method according to claim 52, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
and modifying said entry to include data associated with said lexical feature.
54. The method according to claim 21, wherein said feature comprises a rhetorical feature.
55. The method according to claim 54, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
and modifying said entry to include data associated with said rhetorical feature.
56. The method according to claim 21, wherein said feature comprises a punctuation.
57. The method according to claim 56, further comprising:
generating a flat file for said essay, said flat file including an entry for said sentence;
identifying said punctuation being associated with said sentence; and modifying said entry to include data associated with said punctuation.
58. The automatic essay annotator according to claim 30, wherein said means for determining said feature comprises means for determining a lexical feature.
59. The automatic essay annotator according to claim 58, further comprising:
means for generating a flat file for said essay, said flat file including an entry for said sentence; and means for modifying said entry to include data associated with said lexical feature.
60. The automatic essay annotator according to claim 30, wherein said means for determining said feature comprises means for determining a rhetorical feature.
61. The automatic essay annotator according to claim 60, further comprising:
means for generating a flat file for said essay, said flat file including an entry for said sentence; and means for modifying said entry to include data associated with said rhetorical feature.
62. The automatic essay annotator according to claim 30, wherein said means for determining said feature comprises means for determining a punctuation.
63. The automatic essay annotator according to claim 62, further comprising:
means for generating a flat file for said essay, said flat file including an entry for said sentence;
means for identifying said punctuation being associated with said sentence;
and means for modifying said entry to include data associated with said punctuation.
CA2491238A 2002-06-24 2003-06-23 Automated essay annotation system and method Expired - Fee Related CA2491238C (en)

Applications Claiming Priority (3)

Application Number Priority Date Filing Date Title
US10/176,534 2002-06-24
US10/176,534 US7127208B2 (en) 2002-01-23 2002-06-24 Automated annotation
PCT/US2003/019868 WO2004001700A1 (en) 2002-06-24 2003-06-23 Automated essay annotation system and method

Publications (2)

Publication Number Publication Date
CA2491238A1 true CA2491238A1 (en) 2003-12-31
CA2491238C CA2491238C (en) 2010-08-03

Family

ID=29999066

Family Applications (1)

Application Number Title Priority Date Filing Date
CA2491238A Expired - Fee Related CA2491238C (en) 2002-06-24 2003-06-23 Automated essay annotation system and method

Country Status (9)

Country Link
US (3) US7127208B2 (en)
EP (1) EP1535261A4 (en)
JP (1) JP4334474B2 (en)
KR (1) KR100931515B1 (en)
AU (1) AU2003278846A1 (en)
CA (1) CA2491238C (en)
GB (1) GB0500438D0 (en)
MX (1) MXPA05000237A (en)
WO (1) WO2004001700A1 (en)

Families Citing this family (56)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
MXPA03006566A (en) * 2001-01-23 2004-10-15 Educational Testing Service Methods for automated essay analysis.
US7127208B2 (en) * 2002-01-23 2006-10-24 Educational Testing Service Automated annotation
US8210850B2 (en) * 2002-03-07 2012-07-03 Blank Marion S Literacy education system for students with autistic spectrum disorders (ASD)
US7088949B2 (en) * 2002-06-24 2006-08-08 Educational Testing Service Automated essay scoring
US7149468B2 (en) * 2002-07-25 2006-12-12 The Mcgraw-Hill Companies, Inc. Methods for improving certainty of test-taker performance determinations for assessments with open-ended items
US8128414B1 (en) 2002-08-20 2012-03-06 Ctb/Mcgraw-Hill System and method for the development of instructional and testing materials
CA2516160A1 (en) * 2003-02-14 2004-09-02 Ctb/Mcgraw-Hill System and method for creating, assessing, modifying, and using a learning map
US7980855B1 (en) 2004-05-21 2011-07-19 Ctb/Mcgraw-Hill Student reporting systems and methods
US20070042335A1 (en) * 2005-05-11 2007-02-22 Ctb Mcgraw-Hill System and method for assessment or survey response collection using a remote, digitally recording user input device
US8170466B2 (en) * 2005-05-27 2012-05-01 Ctb/Mcgraw-Hill System and method for automated assessment of constrained constructed responses
US20070009871A1 (en) * 2005-05-28 2007-01-11 Ctb/Mcgraw-Hill System and method for improved cumulative assessment
US20070031801A1 (en) * 2005-06-16 2007-02-08 Ctb Mcgraw Hill Patterned response system and method
US8688026B2 (en) * 2007-01-19 2014-04-01 Barry Morse Essay writing system
US8463594B2 (en) * 2008-03-21 2013-06-11 Sauriel Llc System and method for analyzing text using emotional intelligence factors
US7937386B2 (en) * 2008-12-30 2011-05-03 Complyon Inc. System, method, and apparatus for information extraction of textual documents
US20100169359A1 (en) * 2008-12-30 2010-07-01 Barrett Leslie A System, Method, and Apparatus for Information Extraction of Textual Documents
US20110301941A1 (en) * 2009-03-20 2011-12-08 Syl Research Limited Natural language processing method and system
US9679256B2 (en) 2010-10-06 2017-06-13 The Chancellor, Masters And Scholars Of The University Of Cambridge Automated assessment of examination scripts
US9208139B2 (en) * 2012-01-05 2015-12-08 Educational Testing Service System and method for identifying organizational elements in argumentative or persuasive discourse
US9355372B2 (en) 2013-07-03 2016-05-31 Thomson Reuters Global Resources Method and system for simplifying implicit rhetorical relation prediction in large scale annotated corpus
AU2014285073B9 (en) * 2013-07-03 2017-04-06 Thomson Reuters Enterprise Centre Gmbh Method and system for simplifying implicit rhetorical relation prediction in large scale annotated corpus
GB2524796A (en) * 2014-04-03 2015-10-07 Finned Oy Electronic arrangement and method for educational purposes
US10198428B2 (en) * 2014-05-06 2019-02-05 Act, Inc. Methods and systems for textual analysis
JP6499537B2 (en) * 2015-07-15 2019-04-10 日本電信電話株式会社 Connection expression structure analysis apparatus, method, and program
US10192456B2 (en) * 2015-12-01 2019-01-29 President And Fellows Of Harvard College Stimulating online discussion in interactive learning environments
US11151894B1 (en) * 2017-02-02 2021-10-19 Educational Testing Service Systems and methods for scoring argument critique written essays
US11615145B2 (en) 2017-05-10 2023-03-28 Oracle International Corporation Converting a document into a chatbot-accessible form via the use of communicative discourse trees
US10796102B2 (en) * 2017-05-10 2020-10-06 Oracle International Corporation Enabling rhetorical analysis via the use of communicative discourse trees
US11373632B2 (en) * 2017-05-10 2022-06-28 Oracle International Corporation Using communicative discourse trees to create a virtual persuasive dialogue
US11586827B2 (en) * 2017-05-10 2023-02-21 Oracle International Corporation Generating desired discourse structure from an arbitrary text
US11960844B2 (en) * 2017-05-10 2024-04-16 Oracle International Corporation Discourse parsing using semantic and syntactic relations
US20220284194A1 (en) * 2017-05-10 2022-09-08 Oracle International Corporation Using communicative discourse trees to detect distributed incompetence
US10679011B2 (en) * 2017-05-10 2020-06-09 Oracle International Corporation Enabling chatbots by detecting and supporting argumentation
US10817670B2 (en) * 2017-05-10 2020-10-27 Oracle International Corporation Enabling chatbots by validating argumentation
US10599885B2 (en) * 2017-05-10 2020-03-24 Oracle International Corporation Utilizing discourse structure of noisy user-generated content for chatbot learning
US11386274B2 (en) * 2017-05-10 2022-07-12 Oracle International Corporation Using communicative discourse trees to detect distributed incompetence
US10839154B2 (en) * 2017-05-10 2020-11-17 Oracle International Corporation Enabling chatbots by detecting and supporting affective argumentation
US10839161B2 (en) 2017-06-15 2020-11-17 Oracle International Corporation Tree kernel learning for text classification into classes of intent
US11100144B2 (en) 2017-06-15 2021-08-24 Oracle International Corporation Data loss prevention system for cloud security based on document discourse analysis
US11182412B2 (en) 2017-09-27 2021-11-23 Oracle International Corporation Search indexing using discourse trees
EP3688626A1 (en) 2017-09-28 2020-08-05 Oracle International Corporation Enabling autonomous agents to discriminate between questions and requests
US11809825B2 (en) * 2017-09-28 2023-11-07 Oracle International Corporation Management of a focused information sharing dialogue based on discourse trees
CN117114001A (en) * 2017-09-28 2023-11-24 甲骨文国际公司 Determining a paraphrasing interrelationship across documents based on resolution and identification of named entities
CN108121702B (en) * 2017-12-26 2020-11-24 浙江讯飞智能科技有限公司 Method and system for evaluating and reading mathematical subjective questions
CN111670435A (en) 2018-01-30 2020-09-15 甲骨文国际公司 Detecting requests for interpretation using a communication utterance tree
US11537645B2 (en) 2018-01-30 2022-12-27 Oracle International Corporation Building dialogue structure by using communicative discourse trees
EP3791292A1 (en) 2018-05-09 2021-03-17 Oracle International Corporation Constructing imaginary discourse trees to improve answering convergent questions
US11455494B2 (en) 2018-05-30 2022-09-27 Oracle International Corporation Automated building of expanded datasets for training of autonomous agents
US11237713B2 (en) * 2019-01-21 2022-02-01 International Business Machines Corporation Graphical user interface based feature extraction application for machine learning and cognitive models
US11449682B2 (en) 2019-08-29 2022-09-20 Oracle International Corporation Adjusting chatbot conversation to user personality and mood
US11556698B2 (en) 2019-10-22 2023-01-17 Oracle International Corporation Augmenting textual explanations with complete discourse trees
US11580298B2 (en) 2019-11-14 2023-02-14 Oracle International Corporation Detecting hypocrisy in text
US11501085B2 (en) 2019-11-20 2022-11-15 Oracle International Corporation Employing abstract meaning representation to lay the last mile towards reading comprehension
US11775772B2 (en) 2019-12-05 2023-10-03 Oracle International Corporation Chatbot providing a defeating reply
US11847420B2 (en) 2020-03-05 2023-12-19 Oracle International Corporation Conversational explainability
WO2024039649A1 (en) * 2022-08-15 2024-02-22 BrainPOP IP LLC Automated evaluation of free-form answers and generation of actionable feedback to multidimensional reasoning questions

Family Cites Families (27)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4958284A (en) 1988-12-06 1990-09-18 Npd Group, Inc. Open ended question analysis system and method
US4978305A (en) 1989-06-06 1990-12-18 Educational Testing Service Free response test grading method
JPH0743728B2 (en) 1990-08-02 1995-05-15 工業技術院長 Summary sentence generation method
US5672060A (en) 1992-07-08 1997-09-30 Meadowbrook Industries, Ltd. Apparatus and method for scoring nonobjective assessment materials through the application and use of captured images
US5565316A (en) 1992-10-09 1996-10-15 Educational Testing Service System and method for computer based testing
JP3202381B2 (en) 1993-01-28 2001-08-27 株式会社東芝 Document search device and document search method
US5437554A (en) 1993-02-05 1995-08-01 National Computer Systems, Inc. System for providing performance feedback to test resolvers
JP2957875B2 (en) 1993-03-17 1999-10-06 株式会社東芝 Document information search device and document search result display method
US5778397A (en) 1995-06-28 1998-07-07 Xerox Corporation Automatic method of generating feature probabilities for automatic extracting summarization
US5918240A (en) 1995-06-28 1999-06-29 Xerox Corporation Automatic method of extracting summarization using feature probabilities
US5878386A (en) 1996-06-28 1999-03-02 Microsoft Corporation Natural language parser with dictionary-based part-of-speech probabilities
GB2337844B (en) * 1997-03-21 2001-07-11 Educational Testing Service System and method for on-line essay evaluation
US6115683A (en) 1997-03-31 2000-09-05 Educational Testing Service Automatic essay scoring system using content-based techniques
US6181909B1 (en) * 1997-07-22 2001-01-30 Educational Testing Service System and method for computer-based automatic essay scoring
US6356864B1 (en) 1997-07-25 2002-03-12 University Technology Corporation Methods for analysis and evaluation of the semantic content of a writing based on vector length
US6267601B1 (en) 1997-12-05 2001-07-31 The Psychological Corporation Computerized system and method for teaching and assessing the holistic scoring of open-ended questions
JP2000029894A (en) 1998-07-13 2000-01-28 Ntt Data Corp Subject sentence extraction system
US6332143B1 (en) * 1999-08-11 2001-12-18 Roedy Black Publishing Inc. System for connotative analysis of discourse
GB0006721D0 (en) 2000-03-20 2000-05-10 Mitchell Thomas A Assessment methods and systems
AU2001261505A1 (en) * 2000-05-11 2001-11-20 University Of Southern California Machine translation techniques
US6461166B1 (en) 2000-10-17 2002-10-08 Dennis Ray Berman Learning system with learner-constructed response based testing methodology
US6866510B2 (en) * 2000-12-22 2005-03-15 Fuji Xerox Co., Ltd. System and method for teaching second language writing skills using the linguistic discourse model
MXPA03006566A (en) 2001-01-23 2004-10-15 Educational Testing Service Methods for automated essay analysis.
US20030031996A1 (en) 2001-08-08 2003-02-13 Adam Robinson Method and system for evaluating documents
US7127208B2 (en) 2002-01-23 2006-10-24 Educational Testing Service Automated annotation
US7088949B2 (en) 2002-06-24 2006-08-08 Educational Testing Service Automated essay scoring
US20040073510A1 (en) 2002-06-27 2004-04-15 Logan Thomas D. Automated method and exchange for facilitating settlement of transactions

Also Published As

Publication number Publication date
WO2004001700A1 (en) 2003-12-31
US7127208B2 (en) 2006-10-24
US20070077542A1 (en) 2007-04-05
MXPA05000237A (en) 2005-08-26
KR20050035523A (en) 2005-04-18
US20030138758A1 (en) 2003-07-24
AU2003278846A1 (en) 2004-01-06
CA2491238C (en) 2010-08-03
JP2005531068A (en) 2005-10-13
US7796937B2 (en) 2010-09-14
US20100285434A1 (en) 2010-11-11
GB0500438D0 (en) 2005-02-16
KR100931515B1 (en) 2009-12-14
US8626054B2 (en) 2014-01-07
JP4334474B2 (en) 2009-09-30
EP1535261A4 (en) 2011-02-09
EP1535261A1 (en) 2005-06-01

Similar Documents

Publication Publication Date Title
CA2491238A1 (en) Automated essay annotation system and method
CN1457041B (en) System for automatically annotating training data for natural language understanding system
Rei et al. Artificial error generation with machine translation and syntactic patterns
US6988063B2 (en) System and method for accurate grammar analysis using a part-of-speech tagged (POST) parser and learners' model
CN1542649B (en) Sentence realization system
US7013262B2 (en) System and method for accurate grammar analysis using a learners' model and part-of-speech tagged (POST) parser
US20130325442A1 (en) Methods and Systems for Automated Text Correction
WO2004061593A2 (en) Automated essay scoring
CN102915321A (en) Method and system for processing data
US20040194036A1 (en) Automated evaluation of overly repetitive word use in an essay
CN101479743A (en) Techniques for filtering handwriting recognition results
CN112749563A (en) Named entity identification data labeling quality evaluation and control method and system
KR20190044359A (en) Self-designing modeling system and method using artificial intelligence
CN111126031A (en) Code text processing method and related product
CN112487334A (en) Method, apparatus, computer device and medium for front end page language translation
CN114357961A (en) Project feasibility research report generation method, device, equipment and storage medium
CN107015955B (en) Document editing method and device
Charitsis et al. Assessing function names and quantifying the relationship between identifiers and their functionality to improve them
CN113408296B (en) Text information extraction method, device and equipment
CN114219438A (en) Document file distribution method, device, equipment and medium based on RPA and AI
CN111767222A (en) Data model verification method and device, electronic equipment and storage medium
JP6827610B1 (en) Development support equipment, programs and development support methods
CN117786054A (en) Method and system for generating text visual question and answer based on prompt learning
CN117112271A (en) Random forest model-based method for predicting StarkOverflow question-answer update condition
CN116166533A (en) Interface testing method, device, terminal equipment and storage medium

Legal Events

Date Code Title Description
EEER Examination request
MKLA Lapsed

Effective date: 20210623