WO2004075528A2

WO2004075528A2 - Method of providing evaluation or similar service

Info

Publication number: WO2004075528A2
Application number: PCT/US2004/004059
Authority: WO
Inventors: Lawrence W. Heiser
Original assignee: Heiser Lawrence W
Priority date: 2003-02-13
Filing date: 2004-02-11
Publication date: 2004-09-02
Also published as: WO2004075528A3

Abstract

A method of managing an on-line evaluation includes accessing a network site that displays multiple items each including a statement. The network site also displays a selection control for selecting among the items and a termination control for indicating that selection is complete. The method further includes operating the selection control to select a set of items and operating the termination control to indicate that selection is complete. The network site responds to the termination control being operated by displaying the selected set of items.

Description

METHOD OF PROVIDING EVALUATION OR SIMILAR SERVICE

BACKGROUND OF THE INVENTION [0001] This invention relates to a method of providing evaluation or a similar service.

[0002] In some fields, a great deal of effort has been expended in developing successful strategies for performing a particular task. For example, in the field of new car sales, strategies have been developed to maximize the likelihood that any given sales person will be able to close a sale with any given customer and that the customer will be satisfied with the outcome of the transaction. Such a strategy may involve several components, and one component of a strategy for new car sales might be, for example, that the sales person should be knowledgeable about the particular model of car that the customer is interested in. This component of the sales strategy necessitates that the sales person should receive training regarding the main features of several models of cars. The training may be conducted using various methods that focus on the different models of new cars. Training techniques that are currently used include video presentations and live presentations conducted by in-house presenters, contractor-provided presenters and manufacturer- provided presenters .

[0003] Generally, a new car sales manager wishes to be certain that the sales people learn from the training they receive. In many circumstances, however, it may be difficult for the sales manager to determine whether the sales person has grasped the points made by the trainer during the training session and accordingly the sales manager might not know whether it would be advantageous for the sales person to repeat the training session. This disadvantage of the training program can be eliminated or reduced by testing the sales person after the training session. Further, periodically testing the sales person may allow the sales manager to determine whether the sales person has maintained an acceptable level of knowledge over time. Creating and administering a test are, however, time consuming activities. For example, it is difficult and time consuming to tailor a test to each different video presentation that a sales person might be required to review for the different models in a car manufacturer's lineup of vehicles. An employer might reduce the time commitment for testing by purchasing a ready-made or standard test for administering to the sales people, but such tests are typically composed of a fixed set of multiple choice or true-false questions and tests of this nature are notoriously susceptible to cheating on the part of the test takers. Also, in some fields, an employer who purchases a ready-made test might find that the prescribed answer to a question does not match the employer's practices and therefore it is possible that a given answer will be correct for one employer and incorrect for another employer. Further, conventional tests that are commercially available are difficult or inconvenient to supplement with additional questions and do not provide an opportunity to make modifications to the existing questions. For the foregoing reasons and others, many businesses that would benefit from testing forgo testing.

[0004] A great deal of effort is also expended in conducting surveys and polls in order to gauge opinion on a range of issues. In a typical survey or poll, the respondent is presented with a statement and a number of potential responses. The potential responses may be in the form of a true/false choice, fill-in-the-blank, matching, multiple choices, etc. that express the strength of the respondent's opinion regarding the statement .

[0005] Testing and conducting surveys and polls involve evaluating the respondent's reaction to a statement. The form of the potential responses are typically different depending on whether the statements are presented as a survey or poll or as a test, but nevertheless the goal is to obtain and analyze the responses to the statements. Accordingly, the term "evaluation" is used herein to cover testing and conducting surveys and polls. The person who responds to the statements, whether in the form of a test or a survey or poll, may be referred to as a respondent, although in the case of a test the respondent may alternatively be referred to as a test taker. The term "evaluation" also covers other activities that might not be generally considered to be testing or conducting surveys or polls but where the goal is to obtain responses to statements for subsequent analysis.

SUMMARY OF THE INVENTION

[0006] In accordance with a first aspect of the present invention there is provided a method of managing an on-line evaluation, comprising (a) accessing a network site that displays multiple items each including a statement, the network site also displaying a selection control for selecting among the items and a termination control for indicating that selection is complete, (b) operating the selection control to select a set of items, and (c) operating the termination control to indicate that selection is complete, and wherein the network site responds to the termination control being operated by displaying the set of items selected in step (b) .

[0007] In accordance with a second aspect of the present invention there is provided a method of managing an on-line evaluation, comprising accessing a network site that displays an editable form that allows composition of an item that includes a statement, using the editable form to compose a statement, and operating a termination control to indicate that composition is complete.

[0008] In accordance with a third aspect of the present invention there is provided a method of operating an on-line evaluation site, comprising providing an administrator with access to a network site that displays multiple items, the network site also displaying a selection control for selecting among the items and a termination control for indicating that selection is complete, permitting the administrator to operate the selection control to select a set of items and to operate the termination control to indicate that selection is complete, and responding to operation of the termination control by displaying the set of items selected by the administrator.

[0009] In accordance with a fourth aspect of the present invention there is provided a cooperative method of conducting an on-line evaluation, wherein an administrator accesses a network site that displays multiple items each comprising a statement, the network site also displaying an administrator selection control for selecting among the items and a termination control for indicating that selection is complete, the administrator operates the administration selection control to select a set of items, the administrator operates the termination control to indicate that selection is complete, a respondent accesses the network site, which displays the items selected by the administrator and an respondent selection control for selecting one of at least two possible responses for each item selected by the administrator, and the respondent operates the respondent selection control to select from the possible responses for each item.

[0010] In accordance with a fifth aspect of the present invention there is provided a method of creating an on-line test, comprising (a) accessing a network site that displays an editable form for recording a statement and a selection control for indicating a correct response to the test statement, (b) using the editable form to record a test statement, (c) using the selection control to indicate the correct response to the test statement, (d) repeating steps

(b) and (c) .

BRIEF DESCRIPTION OF THE DRAWINGS [0011] For a better understanding of the invention, and to show how the same may be carried into effect, reference will now be made, by way of example, to the accompanying drawings, in which

FIG. 1 is a simplified schematic block diagram illustrating a system for carrying out a method embodying the invention, FIG. 2 is a schematic illustration of a database that is used in carrying out the method described with reference to FIG. 1,

FIGS. 3-8 show web pages that are used in carrying out the method described with reference to FIG. 1,

DETAILED DESCRIPTION

[0012] For the purpose of this invention, an evaluation, such as a test, is composed of one or more statements (which may be declarative or interrogative) , two or more potential responses to each statement and an indication of the response that is considered correct. A statement, even if declarative, or a combination of a statement and the potential responses, may be referred to herein as a question; a potential response may be referred to as an answer; and the response that is indicated as being considered correct may be referred to as the correct answer.

[0013] There are references herein to a web server displaying information on a client computer. It will be appreciated that the web server does not in fact display anything on the client computer; rather, information accessed by the client computer from or through the web server controls the nature of the display provided by the client computer .

[0014] FIG. 1 illustrates a web server 2 and multiple client computers 4A, 4B,...4N all connected to a computer network 8, which may be the Internet. The different client computers 4A, 4B etc. may be located at the offices of different respective employers, referred to for the sake of convenience as Employer A, Employer B etc.. Generally, there will be several client computers 4A_X, 4A₂, 4A₃, etc. located at the office of Employer A, and similarly for the other employers, but for the sake of simplicity only one client computer for each office is illustrated. Although the following description refers to the client computers as being located at the offices of the different employers, any computer with Internet access and the appropriate password can access the site as a client computer. [0015] For the purposes of the system that is described with reference to the drawings, a person who uses a client computer is a site administrator, a manager or a respondent. The site administrator has the highest privilege level, the manager has an intermediate privilege level and the respondent has the lowest privilege level .

[0016] The web server 2 runs a database application that implements a testing service and hosts, or has network access to, a database 10 (FIG. 2) and interacts with the database for reading data from the database and writing data to the database. The database is composed of several tables. In the implementation described with reference to the drawings, the tables include a client table tClients, in which each record contains a unique client ID, a client name, and particulars of that client's site administrator, managers and respondents .

[0017] The database also includes a test table tTests having a one-to-many relationship with the client table and in which each record contains a client ID. Let us assume that one of the clients is Employer A, and that Employer A' s client ID is IDA. There will generally be several records in the test table for Employer A, and each record for Employer A is identified by the client ID IDA. Each record of the test table also contains a test ID (such as Test 3) and a test name (such as 2003 Climate Control) .

[0018] The database includes a question table tQuestions having a one-to-many relationship with the test table and in which each record contains a unique question ID, a test ID and a question or statement text. A given test, such as Test 3, may be composed of 50 questions and thus for Test 3 there are 50 records in the question table, each having a unique question ID and each containing Test 3 as the test ID and the text of the statement part of the question. Correspondingly, the database includes an answer table tAnswers having a one- to-many relationship with the question table and in which each record contains a question ID and an answer ID that represents the correct answer to the question that is identified by the specific question ID. It will be appreciated that in the case of an evaluation that is a survey or poll as opposed to a test, there will not generally be a correct response to a statement and so for some question IDs there might not be corresponding entries in the answers table .

[0019] The database includes a question library tQuestionLibrary in which each record contains a unique question library ID and the text of a question. Correspondingly, the database includes an answer library tAnswerLibrary in which each record contains a question library ID and an answer library ID.

[0020] Each record in a test library tTestLibrary contains a unique test library ID and several question library IDs. Finally, the database shown in FIG. 2 includes a test history table in which each record contains an employee ID, a test ID and a score value along with the date and time of each test taken.

[0021] In the event that an employer wishes to use the evaluation service for testing employees, the employer designates a site administrator. Different functions associated with the site administrator may be performed by different individuals and accordingly it should be understood that the site administrator might be more than one person. In the case of an employer who uses the system for testing the effectiveness of training of sales people engaged in new car sales, the site administrator may be the general manager. Generally, each sales person responsible for interacting with potential customers will report to a sales manager. A sales manager may also be responsible for interacting with potential customers and will report to his own manager, who may be the site administrator.

[0022] Referring to FIG. 3, the website is composed of a home page with links to several secondary pages (each of which may have links to lower level pages) . In order to enroll with the testing service, the site administrator of Employer A, for example, opens a browser application on the computer 4A, initiates a network protocol session, and navigates to the testing service's home page (which may be, for example, testmenow.com) and selects the ENROLL link 20. The ENROLL link leads to an enroll page (not shown) at which the site administrator provides particulars (e.g. name, address, phone number) regarding the business of the employer, payment information and the administrator's email address and selected login name. The administrator can then log out . The web server 2 generates a random password for the administrator and notifies the administrator of the password by email. During the enrollment session, a persistent cookie is set in the client computer 4A so that when a user of that computer subsequently navigates to the home page, the client computer 4A will display an employer page (FIG. 4) that is partially customized by reference to the employer particulars, e.g. employer name 24. The site administrator has additional customization options such as the addition of a logo and modification of site colors. Further, if the administrator navigates to the testing service's home page from another computer, such as a home computer, and enters his name and the employer password, the home computer will then display the partially customized home page and a corresponding persistent cookie will be set in the home computer.

[0023] During the enrollment session, the administrator makes a choice regarding the scope of the license under which Employer A will use the system. The scope of the license determines the number of users (site administrators, managers and test takers) who may use the system simultaneously on behalf of Employer A. By way of example, the default license may allow three simultaneous users, but an employer who needs to accommodate more than three simultaneous users might purchase more licenses. Each additional license might allow, for example, one additional simultaneous user.

[0024] The employer page includes buttons 26 for giving a test taker, a manager or a site administrator access to the system at their respective privilege levels. When one of the buttons is clicked, the client computer displays one of three login pages (test taker, manager or site administrator) . The site administrator login page is shown in FIG. 5 by way of example. Each login page includes a login control 28 for entering a name and a password or PIN. [0025] Assuming that the person attempting to log in is a site administrator and that the site administrator successfully logs in, the client computer displays a site administrator page (FIG. 6) . The site administrator page includes links allowing the site administrator to add an employee or site administrator, modify the details of an existing employee or site administrator, edit employer information, create a new test, modify an existing test, view reports and add a license.

[0026] The manager login page includes links allowing the manager to create a new test, edit an existing test and view reports relating to employees who report to that manager. [0027] When the site administrator selects the ADD EMPLOYEE link 34, the client computer displays a form including data aware controls allowing the site administrator to add the particulars of an employee, including an indication of whether the employee is a manager. The site administrator may then enroll a potential test taker and assign the test taker a login name and personal identification number (PIN) . The web server records these employee particulars in the employer table. The administrator informs the employee of his login name and PIN. [0028] A site administrator who wishes to modify employee or administrator information selects the individual's name in the appropriate drop-down box 30 and presses the GO button. The client computer then displays a form that allows the site administrator to edit the particulars of the selected individual .

[0029] If the user is a site administrator or a manager and selects the CREATE A TEST link 38, the client computer displays a form that allows the user to assign a name and other particulars to a new test. The client computer then displays an editable control that allows the user to compose a question, designate the type of question (multiple choice or true/false, for example) and provide an indication of the correct answer to each question. The server saves the test name (indexed by client ID) in the test table and saves the question (indexed by test ID) in the questions table and the correct answer (indexed by question ID) in the answers table. The client computer gives the user the option of composing another question or indicating that the test is complete.

[0030] When creating the new test, the user may also select the number of questions to be given when the test is taken. For example, the user might have composed 50 questions but determine that only 30 of these questions should be selected at random and presented to the employee . In the event the user selects that the test should be composed of fewer than all the available questions, the user may also select whether the limited number of questions should be presented at random or in a predetermined order. If the user selects that the test should be composed of all the composed questions, the user may, again, select whether the questions should be presented at random or in a predetermined order. The manager may further determine that the desired set of, e.g., 30 questions should be composed of a first subset of questions, e.g. three questions, that should be included in all administrations of the test and a second subset of 27 questions (in a 30 question test) selected at random from the remaining 47 questions that are available from the 50 -question standard test, and whether the questions of the first subset should be interspersed among the questions of the second subset or placed in a separate group from the questions of the second subset.

[0031] When the user has made all the necessary selections regarding the mode of presentation of the test, he will press a NEXT button and the server will save the information representing the selections made by the user (e.g. regarding the sequence of questions) .

[0032] The administrator page may also include a SELECT READY-MADE TEST link that provides access to one or more ready-made tests that are concerned with the employer's business needs, e.g. new Ford sales in the case of this example, allowing the administrator to select a standard test instead of composing a test from scratch. For example, there may be ready-made tests for each model of Ford vehicle and the site administrator may initially select the tests for Ford passenger cars only, thereby excluding trucks. As noted previously, the test library table does not itself contain questions but contains the names of the ready-made tests and pointers to the questions in the questions library. In response to the site administrator's selection, the server places copies of the questions and answers that form this test in the questions and answers tables respectively, indexed by a test ID that is stored in the tests table and is in turn indexed by the client ID. The test is then immediately made available for a test taker associated with the client ID and can be modified at any time by the administrator or a manager associated with the client ID.

[0033] Instead of selecting a ready-made test for presenting to the test taker with the questions in the form provided by the testing service, the administrator or manager may compile a new test by selecting questions from the question library. As in the case of a test created from scratch, the administrator assigns a name to the test and the server saves the test in the tTests table and labels it with the respective client ID. As in the case of the ready-made test, by saving the test in the employer table the possibility is provided of later modifying the test.

[0034] In the event that a manager logs in, the client computer displays the manager's personal page (FIG. 8), which includes a data aware control with links to test reports, quality assurance reports and employee reports. The test report displays statistics relating to tests (including the number of times taken and the passing rate) . From the test report, a quality assurance report for that test can be accessed, which displays each question contained in the test, the number of times the question has been asked and a percentage reflecting the number of times the question has been answered correctly. A manager who wishes to view the specific test question and answer set can double-click on the question and view or modify the question, add or delete a diagram, and modify the answer set . The employee report shows the performance of the employees who report to that manager (the tests that each employee has taken, the score achieved on the test, and whether the employee passed or failed the test) . The site administrator also may view employee reports by selecting the appropriate link from the administrator page (FIG. 6) . In this case the administrator does not select by employee but selects one or more managers and views the performance of the employees who report to the selected manager (s) .

[0035] When the site administrator or a manager decides that an employee should take a particular test, he notifies the test taker of the name of the test. For example, the administrator may instruct the test taker to take the test on 2003 Climate Control systems. The test taker navigates to the test taker login page and logs in by supplying his login name and PIN. The client computer 4A then displays a personal test taker page (FIG. 7) . The test taker page includes a data aware label 40 that identifies the test taker's manager and a control 42 that allows the test taker to update his manager information. If the test taker's manager has not previously been associated with this test taker, the data aware label 40 instructs the test taker to identify his manager. The personal test taker page also includes a control 44 listing the tests that are available and allowing the test taker to select one of the listed tests, as well as a control 46 that identifies the tests that have previously been taken by this employee and the score achieved on each test .

[0036] When the test taker has selected a test, the client computer 4A displays a page that contains an explanation of how the test will be presented and how the selected answers should be recorded. The personal test taker page also displays controls allowing the test taker either to start the test or cancel . Assuming that the test taker chooses to take the test, the client computer displays the statement and potential responses of the first question and a control that allows the test taker to select one of the potential responses. When the test taker has completed that question, the test taker proceeds by clicking NEXT. The client computer then displays the second question and the potential responses and a control that allows the test taker to select one of the responses to the second question. This sequence continues until the test taker has selected answers to all questions and has thereby signified completion of the test. The client computer then displays a summary page that shows the test taker all the questions and the selected answers and gives the test taker an opportunity to modify the answers that were previously selected. Once the test taker is satisfied with the answers, he presses a button SUBMIT TEST and no further changes are possible. The web server saves the answers, indexed by the test taker's ID, compares the answers selected by the test taker with the correct answers and generates a score, expressed as a percentage. The web server presents the results on the screen and displays the name and the particulars of the test and the score on the test taker's personal page. Scores and other test particulars are emailed to the manager and stored for reporting purposes. By viewing the results of the tests, the site administrator and the test taker's manager are able to evaluate rapidly and consistently whether the training given to the test taker has been effective. This information can be valuable both with respect to whether a particular employee is benefitting from training and with respect to whether a particular trainer is effective.

[0037] By providing the sales manager at the Ford dealership with access to tests that are designed to determine whether a sales person has grasped the points made in a specific training presentation, with mechanisms that allow the sales manager to customize the questions to take account of circumstances at the particular dealership, when necessary, and guard against cheating, the burden of testing sales people to determine the efficacy of the training is greatly reduced.

[0038] Although the testing method described above has been described in detail with reference to testing sales people involved in new car sales, it will be appreciated that the principles are also applicable to other fields. For example, the testing technique may be used to evaluate training of managers with respect to responsibilities under current labor law. In this application, the site administrator might be a law specialist who logs onto the server 2 from the client computer 4B in Employer B's office to create a labor law test. In another application, the system may be used for administering a professional examination, such as a bar examination or an accountancy examination. In this case, the test takers and administrator generally are not employed by the same employer. [0039] It will be appreciated that by interacting with a web server that has access to the database, the testing function can be managed and tests can be given without its being necessary to download software to the client computer. [0040] It will be appreciated that the invention is not restricted to the particular embodiment that has been described, and that variations may be made therein without departing from the scope of the invention as defined in the appended claims and equivalents thereof. For example, although the invention has been described in the context of a test to determine adequacy of training, where there are questions with wrong and right answers, in another implementation the invention may be applied to a survey composed of statements and responses that express strength of feeling regarding the statement. In the latter case, the respondent may be the recipient of an email that includes a hypertext link to the testing site and an invitation to click on the link in order to participate in the survey. Unless the context indicates otherwise, a reference in a claim to the number of instances of an element, be it a reference to one instance or more than one instance, requires at least the stated number of instances of the element but is not intended to exclude from the scope of the claim a structure or method having more instances of that element than stated.

Claims

1. A method of managing an on-line evaluation, comprising:

(a) accessing a network site that displays multiple items each including a statement, the network site also displaying a selection control for selecting among the items and a termination control for indicating that selection is complete,

(b) operating the selection control to select a set of items, and

(c) operating the termination control to indicate that selection is complete, and wherein the network site responds to the termination control being operated by displaying the set of items selected in step (b) .

2. A method according to claim 1, comprising, prior to step (a) , initiating a network protocol session, carrying out steps (a) , (b). and (c) during the network protocol session.

3. A method according to claim 1, wherein each item displayed in step (a) further includes at least two potential responses to the statement and an indication that one of the responses is correct.

4. A method according to claim 1, wherein the network site displays at least two of the items simultaneously and the selection control allows each item to be selected independently of the other item.

5. A method according to claim 1, wherein the network site displays the multiple items consecutively and the selection control allows selection of the displayed item.

6. A method according to claim 1, wherein the network site also displays a sequence control for selecting one of at least two display sequences for display of the items selected in step (a) and the method further comprises operating the sequence control to select one of said display sequences.

7. A method according to claim 1, comprising, prior to step (a) , initiating a first network protocol session, carrying out steps (a) , (b) and (c) during the first network protocol session, and (this is the activity of the test taker taking the test) subsequently initiating a second network protocol session, and wherein the network site responds to the termination control being operated by displaying the set of items during the second network protocol session.

8. A method according to claim 7, wherein the network site displays the statement of a selected item during the second network protocol session and also displays a response control for selecting a response to the statement, and the method further comprises operating the response control to select a response to the statement.

9. A method according to claim 8, wherein each item displayed in step (a) further includes at least two potential responses to the statement and an indication that one of the responses is correct, and the method further comprises displaying an indication of whether the response selected by operation of the response control is the correct response.

10. A method of managing an on-line evaluation, comprising :

(a) accessing a network site that displays an editable form that allows composition of an item that includes a statement,

(b) using the editable form to compose a statement, and

(c) operating a termination control to indicate that composition is complete.

11. A method according to claim 10, for managing an online test, wherein the editable form allows composition of at least two potential responses to the statement and includes a control for selecting whether the item is a multiple choice question or a true/false question and a control for indicating which response is correct.

12. A method according to claim 11, wherein the form includes a control for selecting a sequence in which the statements will be presented to a respondent.

13. A method according to claim 10, for composing an online test, wherein the network site responds to a test control being operated by displaying a test including the item composed in step (b) .

14. A method according to claim 13, further comprising, between steps (b) and (c) , clearing the editable form and using the editable form to compose an additional test item, and wherein the network site responds to the test control being operated by displaying a test including the item composed in step (b) and the additional test item.

15. A method according to claim 10, further comprising, between steps (b) and (c) , clearing the editable form and using the editable form to compose an additional item.

16. A method according to claim 10, for composing an online test, comprising, prior to step (a), initiating a first network protocol session, carrying out steps (a) , (b) and (c) during the first network protocol session, and subsequently initiating a second network protocol session, and wherein the network site responds to a test control being operated during the second network protocol session by displaying the test item composed in step (a) .

17. A method of operating an on-line evaluation site, comprising : providing an administrator with access to a network site that displays multiple items, the network site also displaying a selection control for selecting among the items and a termination control for indicating that selection is complete, permitting the administrator to operate the selection control to select a set of items and to operate the termination control to indicate that selection is complete, and responding to operation of the termination control by displaying the set of items selected by the administrator.

18. A method according to claim 17, comprising allowing the administrator to access the network site during a first network protocol session and wherein the method further comprises displaying said set of items during a second network protocol session.

19. A method according to claim 17, of operating an online testing service, wherein the multiple items displayed by the network site constitute a standard test composed of multiple statements and at least two potential responses to each statement, and the method comprises responding to operation of the termination control by displaying the multiple statements and their potential responses.

20. A cooperative method of conducting an on-line evaluation, wherein: an administrator accesses a network site that displays multiple items each comprising a statement, the network site also displaying an administrator selection control for selecting among the items and a termination control for indicating that selection is complete, the administrator operates the administration selection control to select a set of items, the administrator operates the termination control to indicate that selection is complete, a respondent accesses the network site, which displays the items selected by the administrator and an respondent selection control for selecting one of at least two possible responses for each item selected by the administrator, and the respondent operates the respondent selection control to select from the possible responses for each item.

21. A method according to claim 20, wherein the administrator accesses the network site, operates the administration selection control and operates the termination control during a first network protocol session, and the respondent accesses the network site and operates the respondent selection control during a second network protocol session.

22. A method of creating an on-line test, comprising:

(a) accessing a network site that displays an editable form for recording a statement and a selection control for indicating a correct response to the test statement,

(b) using the editable form to record a test statement,

(c) using the selection control to indicate the correct response to the test statement,

(d) repeating steps (b) and (c) .