CA2235364C - Automated meaningful phrase clustering - Google Patents
Automated meaningful phrase clustering Download PDFInfo
- Publication number
- CA2235364C CA2235364C CA002235364A CA2235364A CA2235364C CA 2235364 C CA2235364 C CA 2235364C CA 002235364 A CA002235364 A CA 002235364A CA 2235364 A CA2235364 A CA 2235364A CA 2235364 C CA2235364 C CA 2235364C
- Authority
- CA
- Canada
- Prior art keywords
- meaningful
- automated
- task
- phrases
- clusters
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
- 238000000034 method Methods 0.000 claims abstract description 49
- 238000012360 testing method Methods 0.000 claims abstract description 10
- 238000013528 artificial neural network Methods 0.000 claims 3
- 238000013179 statistical model Methods 0.000 claims 3
- 238000001514 detection method Methods 0.000 description 20
- 238000009826 distribution Methods 0.000 description 10
- 239000012634 fragment Substances 0.000 description 6
- 238000013459 approach Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000012545 processing Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000002474 experimental method Methods 0.000 description 3
- 230000006978 adaptation Effects 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000004044 response Effects 0.000 description 2
- 238000010187 selection method Methods 0.000 description 2
- 239000000126 substance Substances 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000012549 training Methods 0.000 description 2
- 238000013519 translation Methods 0.000 description 2
- 230000014616 translation Effects 0.000 description 2
- 241000282412 Homo Species 0.000 description 1
- 208000013016 Hypoglycemia Diseases 0.000 description 1
- 101150020162 ICS1 gene Proteins 0.000 description 1
- 101100116913 Saccharomyces cerevisiae (strain ATCC 204508 / S288c) DJP1 gene Proteins 0.000 description 1
- 241000030538 Thecla Species 0.000 description 1
- 230000009471 action Effects 0.000 description 1
- 230000004075 alteration Effects 0.000 description 1
- 230000000052 comparative effect Effects 0.000 description 1
- 230000002596 correlated effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 238000003780 insertion Methods 0.000 description 1
- 230000037431 insertion Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000005192 partition Methods 0.000 description 1
- 238000001558 permutation test Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012552 review Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G10—MUSICAL INSTRUMENTS; ACOUSTICS
- G10L—SPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
- G10L15/00—Speech recognition
- G10L15/08—Speech classification or search
- G10L15/18—Speech classification or search using natural language modelling
- G10L15/1815—Semantic context, e.g. disambiguation of the recognition hypotheses based on word meaning
Abstract
A system and method for automated task selection is provided where a selected task is identified from the natural speech of the user making the selection. The system and method incorporate the selection of meaningful phrases through the use of a test for significance. The selected meaningful phrases are then clustered. The meaningful phrase clusters are input to a speech recognizer that determines whether any meaningful phrase clusters are present in the input speech. Task-type decisions are then made on the basis of the recognized meaningful phrase clusters.
Description
AUTOMATED MEANINGFUL PHRASE CLUSTERING
BACKGROUND OF THE INVENTION
1. Field of Invention This invention relates to speech processing, and more particularly to a system and method for automated clustering of meaningful phrases in relation to the performance of one or more desired tasks.
BACKGROUND OF THE INVENTION
1. Field of Invention This invention relates to speech processing, and more particularly to a system and method for automated clustering of meaningful phrases in relation to the performance of one or more desired tasks.
2. Description of Related Art In communications networks there are many instances where it is desirable to provide for automated implementation of particular tasks desired by a user of such a network -- i.e., implementation of such a task without human intervention. In the prior art, such automated task implementation is generally carried out via a plurality of menu choices which must be selected by designated signals from a user, general numeric signals generated by a keypad associated with a user's telephone set, and in some cases by the user pronouncing such numerals as key words. In many cases such menu-based automated task implementation arrangements involve mufti-tiered menus. Such mufti-tiered menu structures are generally unpopular with users and remarkably inefficient at achieving the desired objective.
The percentage of successful routings through such a multi-tiered menu structure can be quite low. Stated differently, in such circumstances, many of the calls accessing such a mufti-tiered menu structure might be either terminated without the caller having reached the desired objective or else defaulted to an operator (or other manned default station).
The limitations in the prior art were addressed in Canadian Application No. 2,180,687, "Automated Phrase Generation", and Canadian Application No. 2,204,868, "Automated Call Router System" both filed September 15, 1999. These applications provide a methodology for automated task selection where the selected task is identified in the natural speech of a user making such a selection. A fundamental aspect of this method is a determination of a set of meaningful phrases. Such meaningful phrases are determined by a grammatical inference algorithm which operates on a predetermined corpus of speech utterances, each such utterance being associated with a specific task objective, and wherein each utterance is marked with its associated task objective.
The determination of the meaningful phrases used in the above noted application is founded in the concept of combining a measure of commonality of words and/or structure within the language -- i.e., how often groupings of things co-occur -- with a measure of significance to a defined task for such a grouping. That commonality measure within the language can be manifested as the mutual information in n-grams derived from a database of training speech utterances and the measure of usefulness to a task is manifested as a salience measure.
Mutual information ("MI"), which measures the likelihood of co-occurrence for two or more words, involves only the language itself. For example, given War and Peace in the original Russian, one could compute the mutual information for all the possible pairings of words in that text without ever understanding a word of the language in which it is written. In contrast, computing salience involves both the language and its extra-linguistic associations to a device's environment. Through the use of such a combination of MI and a salience factor, meaningful phrases are selected which have both a positive MI
(indicating relative strong association among the words comprising the phrase) and a high salience value.
However, such methods are based upon the probability that separate sets of salient words occur in the particular input utterance. For example, the salient phrases "made a long distance", "a long distance call", and "long distance call", while being spoken by the users to achieve the same objective, would be determined as separate meaningful phrases by that grammatical inference algorithm based on their individual mutual information and salience values.
Thus, many individual phrases which are virtually identical and have the same meaning, are generated, remain separate, and represent independent probabilities of occurrence in the grammatical inference algorithm. By not grouping these "alike" salient phrases, the above methods could provide inferior estimates of probability and thus ultimately provide improper routing of requests from users.
SUMMARY OF THE INVENTION
A method and system for automated task selection is provided where a selected task is identified from the natural speech of the user making the selection. The method and system incorporate the selection of meaningful phrases through the use of a test for significance. The selected meaningful phrases are then clustered using a combination of string and semantic distortions. The meaningful phrase clusters are input to a speech recognizer that determines whether any meaningful phrase clusters are present in the input speech. Task-type decisions are then made on the basis of the recognized meaningful phrase clusters.
3a In accordance with one aspect of the present invention, there is provided a method for automated task classification which operates on a task objective of a user, the task objective being expressed in natural speech of the user, comprising: selecting meaningful phrases from a plurality of transcribed speech utterances; clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters; detecting meaningful phrase clusters present in input speech utterances; and making task-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
In accordance with another aspect of the present invention, there is provided a method for automated call type classification which operates on the call routing objective of a user, the call-routing objective being expressed in natural speech of the user, comprising:
selecting meaningful phrases from a plurality of transcribed speech utterances, clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters; detecting meaningful phrase clusters present in input speech utterances; and making call-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
In accordance with yet another aspect of the present invention, there is provided an automated task classification system which operates on task objectives of a user, the task objectives being expressed in natural speech of the user, comprising: a meaningful phrase selector that selects a plurality of meaningful phrases from a set of speech utterances, each of the meaningful phrases being 3b selected based on one of a predetermined set of the task objectives; a meaningful phrase clustering device for clustering selected meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
an input recognizing device for detecting any of the clustered meaningful phrases in input speech of the user, the input recognizing device having as an input the meaningful phrases clustered by the meaningful phrase clustering device; and a classification device responsive to an input of the detected meaningful phrase clusters for making a classification decision based on the detected meaningful phrase clusters as to one of the set of predetermined task objectives.
The percentage of successful routings through such a multi-tiered menu structure can be quite low. Stated differently, in such circumstances, many of the calls accessing such a mufti-tiered menu structure might be either terminated without the caller having reached the desired objective or else defaulted to an operator (or other manned default station).
The limitations in the prior art were addressed in Canadian Application No. 2,180,687, "Automated Phrase Generation", and Canadian Application No. 2,204,868, "Automated Call Router System" both filed September 15, 1999. These applications provide a methodology for automated task selection where the selected task is identified in the natural speech of a user making such a selection. A fundamental aspect of this method is a determination of a set of meaningful phrases. Such meaningful phrases are determined by a grammatical inference algorithm which operates on a predetermined corpus of speech utterances, each such utterance being associated with a specific task objective, and wherein each utterance is marked with its associated task objective.
The determination of the meaningful phrases used in the above noted application is founded in the concept of combining a measure of commonality of words and/or structure within the language -- i.e., how often groupings of things co-occur -- with a measure of significance to a defined task for such a grouping. That commonality measure within the language can be manifested as the mutual information in n-grams derived from a database of training speech utterances and the measure of usefulness to a task is manifested as a salience measure.
Mutual information ("MI"), which measures the likelihood of co-occurrence for two or more words, involves only the language itself. For example, given War and Peace in the original Russian, one could compute the mutual information for all the possible pairings of words in that text without ever understanding a word of the language in which it is written. In contrast, computing salience involves both the language and its extra-linguistic associations to a device's environment. Through the use of such a combination of MI and a salience factor, meaningful phrases are selected which have both a positive MI
(indicating relative strong association among the words comprising the phrase) and a high salience value.
However, such methods are based upon the probability that separate sets of salient words occur in the particular input utterance. For example, the salient phrases "made a long distance", "a long distance call", and "long distance call", while being spoken by the users to achieve the same objective, would be determined as separate meaningful phrases by that grammatical inference algorithm based on their individual mutual information and salience values.
Thus, many individual phrases which are virtually identical and have the same meaning, are generated, remain separate, and represent independent probabilities of occurrence in the grammatical inference algorithm. By not grouping these "alike" salient phrases, the above methods could provide inferior estimates of probability and thus ultimately provide improper routing of requests from users.
SUMMARY OF THE INVENTION
A method and system for automated task selection is provided where a selected task is identified from the natural speech of the user making the selection. The method and system incorporate the selection of meaningful phrases through the use of a test for significance. The selected meaningful phrases are then clustered using a combination of string and semantic distortions. The meaningful phrase clusters are input to a speech recognizer that determines whether any meaningful phrase clusters are present in the input speech. Task-type decisions are then made on the basis of the recognized meaningful phrase clusters.
3a In accordance with one aspect of the present invention, there is provided a method for automated task classification which operates on a task objective of a user, the task objective being expressed in natural speech of the user, comprising: selecting meaningful phrases from a plurality of transcribed speech utterances; clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters; detecting meaningful phrase clusters present in input speech utterances; and making task-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
In accordance with another aspect of the present invention, there is provided a method for automated call type classification which operates on the call routing objective of a user, the call-routing objective being expressed in natural speech of the user, comprising:
selecting meaningful phrases from a plurality of transcribed speech utterances, clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters; detecting meaningful phrase clusters present in input speech utterances; and making call-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
In accordance with yet another aspect of the present invention, there is provided an automated task classification system which operates on task objectives of a user, the task objectives being expressed in natural speech of the user, comprising: a meaningful phrase selector that selects a plurality of meaningful phrases from a set of speech utterances, each of the meaningful phrases being 3b selected based on one of a predetermined set of the task objectives; a meaningful phrase clustering device for clustering selected meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
an input recognizing device for detecting any of the clustered meaningful phrases in input speech of the user, the input recognizing device having as an input the meaningful phrases clustered by the meaningful phrase clustering device; and a classification device responsive to an input of the detected meaningful phrase clusters for making a classification decision based on the detected meaningful phrase clusters as to one of the set of predetermined task objectives.
BRIEF DESCRIPTION OF THE DRAWINGS
Fig. 1 illustrat~as examples of false and missed detection by a classifier for an automated call routing system based on use of "meaningful phrases";
Fig. 2 illustrates examples of correct detection by a classifier for an automated call routing system based on use of "meaningful phrases";
Fig. 3 illustrates an example of the advantage provided by the "meaningful phrase" classification parameter of 'the system of the invention;
Fig. 4 presents in block diagram form an exemplary structure of the system of the invention;
Fig. 5 depicts vhe method of the invention in flowchart form;
Fig. 6 illustrates a Meaningful Phrase Cluster;
Fig. 7 illustrates a Meaningful Phrase Cluster using approximate matching; and Fig. 8 is a graph of ROC curves illustrating the performance of the invention.
DETAILED DESCRIPTION OF INVENTION
The discussion following will be presented partly in terms of algorithms and symbolic representations of operations on data bits within a computer system. As will be understood, these algorithmic descriptions and representations are a means ordinarily used by those skilled in the computer processing arts to convey the substance of their work to others skilled in the art.
As used herein (arid generally) an algorithm may be seen as a self-contained sequence of steps leading to a desired result. Thf~se steps generally involve manipulations of physical quantities. Usually, though not necessarily, these quantivies take the form of electrical or magnetic signals capable of being stored, transferred, ___ combined, compared and otherwise manipulated. For convenience of reference, as well as to comport with common usage, these signals will be described from time to time in terms of bits, values, elements, symbols, characters, 5 terms, numbers, or the like. However, it should be emphasized that these and similar terms are to be associated with the appropriate physical quantities -- such terms being merely convf~nient labels applied to those quantities.
It is important as well that the distinction between the method of opei°ations and operating a computer, and the method of computation itself should be kept in mind. The present invention relates to methods for operating a computer in processing electrical or other (e. g., mechanical, chemical) physical signals to generate other desired physical signals.
For clarity of explanation, the illustrative embodiment of the present invention is presented as comprising individual functional blocks (including funcaional blocks labeled as "processors"). The functions they>e blocks represent may be provided through the use of either shared or dedicated hardware, including, but not limited to, hardware capable of executing software. For example the functions of ~~rocessors presented in Figure 4 may be provided by a single shared processor. (Use of the term "processor" should not be construed to refer exclusively to hardware capable of executing software.) Illustrative embodiments may comprise microprocessor and/or digital signal processor (DSP) hara.ware, such as the ABC&T DSPI6 or DSP32C, read-only memcry (ROM) for storing software performing the operations discussed below, and randorn access memory (RAM) for storing results. Very large scale integration (VLSI) hardware __ embodiments, as well as custom VLSI circuitry in combination with a genera:L purpose DSP circuit, may also be provided.
_ A fundamental objective of the invention is a task selection method involving communication between human user and machine, that shift:> the burden of understanding a specialized vocabulary from the user to the machine. Thus, in a generalized embodimE:nt, the invention is represented as a task selection method having the following characteristics:
First, a user accessing s system will be presented with a greeting similar to "How may I help you?"
After the user responds to that greeting with a natural speech statement of the user's objective (such as the implementation of a desired task), the user's request may be classified into one of a number of predefined task objE=_ctives with the task objective then being implemented.
In the preferre<~ embodiment of the invention described hereafter, implementation of the invention's syst=em and method will occasionally be illustrated by exemplary applications in the form user-requested tasks to be carried out by a telephone system or network. In such exemplary applications, task objectives for a user rnay inc7_ude call billing options (e. g. , collect, third-party) , dialing information, billing questions, credit requests (as for a wrong or mis-dialed number), area codes, etc.
I. DETAILED DESCRIPTIc~N OF PREFERRED EMBODIMENTS
In traditional conununications environments, a user is often required to know separate numbers and/or dialing patterns to access different services available at a given communications destination, as well as possibly having to navigate a menu-driven system which then routes the user to the desired objective. With the system and method of the invention, the user is able to access a central number and the user's objective will be implemented by the communications recipient on the basis of its content.
An example of such content-based routing would be where a caller responds to a "how may I help you" prompt with I
want to reverse the charges. The appropriate action is to connect the caller to an automated subsystem which processes collect calls. Another example would be a caller response of I am having a problem understanding my bill, in which case the caller should be connected to the telephone carrier's business office. The system thus needs to understand spoken language to the extent of routing the call appropriately.
A. Baseline Approach The basic construct of such a system has been described by one of the inventors in Gorin, A. "On automated language acquisition", J. Acoust. Soc. Am., 97 3441-3461, (June, 1995) [hereafter referred to as Gorin 95]. A number of considerations from that baseline approach are material to the system and method of the invention. Certain of those considerations will be briefly reviewed hereafter. As a preface to that review, it is to be noted that the approach described in Gorin 95, the classification parameter for determination of an association between input text or speech and one of a set of predefined task objectives is implemented as salient words derived from a corpus of speech utterances having marked associated task objectives. In the adaptation of that method describe herein, the classificati~~n parameter is implemented as meaningful phrase cluster; derived from a corpus of speech utterances having marked associated routing objectives.
Central to the a~>proach here is a database of a large number of utterances, each of which is related to one of a predetermined set of routing objectives. This database forms an input to a classification parameter algorithm. Preferably, such utterances will be extracted from actual user response: to a prompt of "How may I help you:"' (or similar words to the same effect). Each uttE:rance is then transcr=_bed and labeled with one of the predetermined set of routing objectives. Illustrative uttE:rances from the database utilized by the inventors are as follows Yeah, I want to reverse the charges I was just disconnected from this number I was trying to hang-up I am trying reach Mexico Charge this to my home phone In a related article co-authored by one of the inventors [Gorin, A.L., Hanek, H., Rose, R. and Miller, L., "Spoken ,.
Language Acquisition for Automated Call Routing , in Proceedings of the International Conference on Spoken Language Processing (ICS1~P 94) , Yokohama (Sept. 18-22, 1994)] [hereafter Gorin 94AJ, it is noted that the distribution of routing objectives in such a data base may be substantially skewed. The implications of such skewing may well be taken into account in the determination of the particular set of routing objectives to be supported on an automated basis by the system of the invention.
A salience princi~~le as related to the system of the invention has been defined in another article co-__ authored by one of the inventors [Gorin, A.L., Levinson, S.
E. and Sankar, A. "An Experiment in Spoken Language Acquisition," IEEE Trans, on Speech and Audio, Vol. 2, No.
l, Part II, pp. 224-240 (January 1994)] [hereafter Gorin 94]. Specifically, the salience of a word is defined as the information content of that word for the task under consideration. It can be interpreted as a measure of how meaningful that word is for the task. Salience can be distinguished from and compared to the traditional Shannon information content, which measures the uncertainty that a word will occur. As is known, such traditional information content can be estimated from examples of the language, while an estimation of salience requires both language and its extra-linguistic associations.
As previously noted, Gorin 95 uses as a classification parameter, words from test speech utterances which have a salient association with particular objective routings.
Canadian Application Nos. 2,180,687 and 2,204,868 represent a significant point of departure from that methodology through the use of meaningful phrases as the classification parameter. Before describing the method for determining such meaningful phrases, it is useful to define two types of errors experienced in such an automated routing system and a related "success" concept:
False detection of a routing objective can occur when a salient (meaningful) phrase related to one routing objective is detected in a caller's input speech when the caller's actual request was directed to another routing objective. The probability of such a false detection occurring will hereafter be referred to by the designation: PFD
Missed detection of a routing objective occurs when the callers input speech is directed to that routing objective an~~ none of the meaningful phrases which are associated with that routing objective are detected in the input speech. The probability of such a missed detection occurring will hereafter be 5 referred to by the designation. PMD
Coverage for a routing objective refers to the number of successful translations by the system of a request for a routing objective to that routing objective relative to the total number of input 10 requests for that routing objective. As an illustrative example, a routing objective for which 60 successful translations occurred out of 100 input requests for that routing objective would be said to experience 60o coverage. It is noted that Coverage 1-P~
Of the two erro:_ types defined above, one is significantly more "costly" than the other. The consequence of a false detection error is the routing of a use~~. to a different task objective than was requested by the user. Such a result is at a minimum very annoying to the user. The possibility also exists that such an error could result in a direct cost to the system provider -- an annoyed customer or potential customer being classified here as an indirect cost -- through some non-system error resulting from the caller being connected to an incorrect rouging objective. The consequence of a missed detection error, on the other hand, is simply the routing of the user to a default operator po=>ition and the only cost is the lost: opportunity cost of r..ot handling that particular task on an automated basis. Thus, while ideally the probabilities of both missed detection and false detection should be near zero, it is far more important from the user's perspective that this objective be realized for __ false detection errors. As will be seen below, there are circumstances where tradeoffs must be made between minimizing one or another of these error probabilities, and this principle will be applied in such circumstances.
B. Adaptation o:E Baseline Approach Figure 1 provides several illustrative examples of False Detections and Missed Detections from the database of speech utterances used by the inventors. While the basis for error in each of these examples is believed to be largely self-explanatory, the error in the first example in each set will be briefly described. In the first example under False Detection, the meaningful phrase is I NE'ED_CREDIT-EOS (end of sentence), and thus this phrase would have been classified as a request for credit.
~5 However, from reading the entire utterance, it is apparent tha-~ the caller actually wanted to be transferred to another carrier (the carrier receiving this request being AT&'C). In the first exam~~le under Missed Detections, there are no meaningful phrases identified in the utterance (and therefore no basis for classifying the caller's objective), although it is apparent= to humans from reading the utterance that the caller is seeking a billing credit. As a comparative illustration, Figure 2 shows several examples of correct detection of a billing credit objective from meaningful phrases in the input speech.
There are two significant advantages of the methodology of the invention in using meaningful phrases as the classification parameter over the use of salient words in the baseline approach described in Gorin 95. First, with the use of words as vhe classification parameter, the word choices for detecting a given routing objective may be highly limited in order to achieve a minimum probability of fal~;e detection -- i.e., use of only words having a near __ 1000 likelihood of predicting the intended routing objective -- and therefore the coverage for such a routing objective is likely to be very low, leading to a high probability of missed detection errors occurring. With meaningful phrases as a classification parameter, on the other hand, both low probability of false detection and low probability of missed detection are achievable.
Figure 3 provides an illustrative example of this advantage. This figure shows the Classification Rate and the Coverage for an exen~.plary routing objective, Billing Credit, as the phrase used for the classification parameter grows in length and/or complexity. The Classification Rate is defined as the probability of the routing objective (CREDIT) having been requested, given the occurrence of the selected phrase in the input speech (i.e., P(CREDIT
phrase). Similarly, the Coverage term is defined as the pro;oability of the selected phrase appearing in the input speech, given that the designated routing objective (CREDIT
has been requested. In the Phrase column, parenthesis surrounding a series of terms separated by " I " indicate one of those terms appearing in the indicated position with other terms in that row. The nomenclature "F(Wrong)"
indicates a grammar fragment surrounding the word "wrong", the phrase in the fourth row of that column being representative of such a grammar fragment surrounding a salient word. The designation "previous" indicates a carry forward of everything on t:he preceding line. And finally, the abbreviation "eos" indicates "end of sentence."
In speech recognition systems, the larger the fragment of speech presenl~ed to such a speech- recognizes, the higher the probability of a correct recognition of that speech fragment. Thus a speech recognizes programmed to spot one of a set of salient words can be expected to fail in its task significantl:~ more often than such a device programmed to spot meaningful phrases, comprising two or more words.
C. Description of the Methodolo y of Invention The methodology of the invention is graphically illustrated in the flowchart of Figure 5. Following the steps of the flowchart, meaningful phrases are generated at step 110 from transcripticns of recognized utterances using est_Lmated posterior distributions over task types and a test: for significance, as set forth below. The meaningful phrases that have been generated are then clustered at step 120 and formed into meaningful phrase clusters using a distance measure between phrases, based on string and semantic distortion. Meaningful phrase clusters are then searched for and detected in input speech in step 130 using exacts or approximate m~~tching procedures. Task-type decisions are made at step 140 based on the detected meaningful phrase clusters.
D. Selection and Clustering of Meaningful Phrases 1. Selection of meaningful phrases It is desirable in a speech recognition system to select phrases that are me,3ningful for the task. A measure of salience may be used to assess, for a particular phrase, the distortion between the prior and posterior distributions over the call types. However, this does not take into account the frequency with which a fragment occurs. For example, a fortunate conjunction of events can give a low-frequency phrase a high apparent salience purely by chance.
Here, this shortcoming is avoided by testing, for each phrase, the null hypos:hesis that is governed simply by the prior probabilities (and therefore occurs at random).
Fig. 1 illustrat~as examples of false and missed detection by a classifier for an automated call routing system based on use of "meaningful phrases";
Fig. 2 illustrates examples of correct detection by a classifier for an automated call routing system based on use of "meaningful phrases";
Fig. 3 illustrates an example of the advantage provided by the "meaningful phrase" classification parameter of 'the system of the invention;
Fig. 4 presents in block diagram form an exemplary structure of the system of the invention;
Fig. 5 depicts vhe method of the invention in flowchart form;
Fig. 6 illustrates a Meaningful Phrase Cluster;
Fig. 7 illustrates a Meaningful Phrase Cluster using approximate matching; and Fig. 8 is a graph of ROC curves illustrating the performance of the invention.
DETAILED DESCRIPTION OF INVENTION
The discussion following will be presented partly in terms of algorithms and symbolic representations of operations on data bits within a computer system. As will be understood, these algorithmic descriptions and representations are a means ordinarily used by those skilled in the computer processing arts to convey the substance of their work to others skilled in the art.
As used herein (arid generally) an algorithm may be seen as a self-contained sequence of steps leading to a desired result. Thf~se steps generally involve manipulations of physical quantities. Usually, though not necessarily, these quantivies take the form of electrical or magnetic signals capable of being stored, transferred, ___ combined, compared and otherwise manipulated. For convenience of reference, as well as to comport with common usage, these signals will be described from time to time in terms of bits, values, elements, symbols, characters, 5 terms, numbers, or the like. However, it should be emphasized that these and similar terms are to be associated with the appropriate physical quantities -- such terms being merely convf~nient labels applied to those quantities.
It is important as well that the distinction between the method of opei°ations and operating a computer, and the method of computation itself should be kept in mind. The present invention relates to methods for operating a computer in processing electrical or other (e. g., mechanical, chemical) physical signals to generate other desired physical signals.
For clarity of explanation, the illustrative embodiment of the present invention is presented as comprising individual functional blocks (including funcaional blocks labeled as "processors"). The functions they>e blocks represent may be provided through the use of either shared or dedicated hardware, including, but not limited to, hardware capable of executing software. For example the functions of ~~rocessors presented in Figure 4 may be provided by a single shared processor. (Use of the term "processor" should not be construed to refer exclusively to hardware capable of executing software.) Illustrative embodiments may comprise microprocessor and/or digital signal processor (DSP) hara.ware, such as the ABC&T DSPI6 or DSP32C, read-only memcry (ROM) for storing software performing the operations discussed below, and randorn access memory (RAM) for storing results. Very large scale integration (VLSI) hardware __ embodiments, as well as custom VLSI circuitry in combination with a genera:L purpose DSP circuit, may also be provided.
_ A fundamental objective of the invention is a task selection method involving communication between human user and machine, that shift:> the burden of understanding a specialized vocabulary from the user to the machine. Thus, in a generalized embodimE:nt, the invention is represented as a task selection method having the following characteristics:
First, a user accessing s system will be presented with a greeting similar to "How may I help you?"
After the user responds to that greeting with a natural speech statement of the user's objective (such as the implementation of a desired task), the user's request may be classified into one of a number of predefined task objE=_ctives with the task objective then being implemented.
In the preferre<~ embodiment of the invention described hereafter, implementation of the invention's syst=em and method will occasionally be illustrated by exemplary applications in the form user-requested tasks to be carried out by a telephone system or network. In such exemplary applications, task objectives for a user rnay inc7_ude call billing options (e. g. , collect, third-party) , dialing information, billing questions, credit requests (as for a wrong or mis-dialed number), area codes, etc.
I. DETAILED DESCRIPTIc~N OF PREFERRED EMBODIMENTS
In traditional conununications environments, a user is often required to know separate numbers and/or dialing patterns to access different services available at a given communications destination, as well as possibly having to navigate a menu-driven system which then routes the user to the desired objective. With the system and method of the invention, the user is able to access a central number and the user's objective will be implemented by the communications recipient on the basis of its content.
An example of such content-based routing would be where a caller responds to a "how may I help you" prompt with I
want to reverse the charges. The appropriate action is to connect the caller to an automated subsystem which processes collect calls. Another example would be a caller response of I am having a problem understanding my bill, in which case the caller should be connected to the telephone carrier's business office. The system thus needs to understand spoken language to the extent of routing the call appropriately.
A. Baseline Approach The basic construct of such a system has been described by one of the inventors in Gorin, A. "On automated language acquisition", J. Acoust. Soc. Am., 97 3441-3461, (June, 1995) [hereafter referred to as Gorin 95]. A number of considerations from that baseline approach are material to the system and method of the invention. Certain of those considerations will be briefly reviewed hereafter. As a preface to that review, it is to be noted that the approach described in Gorin 95, the classification parameter for determination of an association between input text or speech and one of a set of predefined task objectives is implemented as salient words derived from a corpus of speech utterances having marked associated task objectives. In the adaptation of that method describe herein, the classificati~~n parameter is implemented as meaningful phrase cluster; derived from a corpus of speech utterances having marked associated routing objectives.
Central to the a~>proach here is a database of a large number of utterances, each of which is related to one of a predetermined set of routing objectives. This database forms an input to a classification parameter algorithm. Preferably, such utterances will be extracted from actual user response: to a prompt of "How may I help you:"' (or similar words to the same effect). Each uttE:rance is then transcr=_bed and labeled with one of the predetermined set of routing objectives. Illustrative uttE:rances from the database utilized by the inventors are as follows Yeah, I want to reverse the charges I was just disconnected from this number I was trying to hang-up I am trying reach Mexico Charge this to my home phone In a related article co-authored by one of the inventors [Gorin, A.L., Hanek, H., Rose, R. and Miller, L., "Spoken ,.
Language Acquisition for Automated Call Routing , in Proceedings of the International Conference on Spoken Language Processing (ICS1~P 94) , Yokohama (Sept. 18-22, 1994)] [hereafter Gorin 94AJ, it is noted that the distribution of routing objectives in such a data base may be substantially skewed. The implications of such skewing may well be taken into account in the determination of the particular set of routing objectives to be supported on an automated basis by the system of the invention.
A salience princi~~le as related to the system of the invention has been defined in another article co-__ authored by one of the inventors [Gorin, A.L., Levinson, S.
E. and Sankar, A. "An Experiment in Spoken Language Acquisition," IEEE Trans, on Speech and Audio, Vol. 2, No.
l, Part II, pp. 224-240 (January 1994)] [hereafter Gorin 94]. Specifically, the salience of a word is defined as the information content of that word for the task under consideration. It can be interpreted as a measure of how meaningful that word is for the task. Salience can be distinguished from and compared to the traditional Shannon information content, which measures the uncertainty that a word will occur. As is known, such traditional information content can be estimated from examples of the language, while an estimation of salience requires both language and its extra-linguistic associations.
As previously noted, Gorin 95 uses as a classification parameter, words from test speech utterances which have a salient association with particular objective routings.
Canadian Application Nos. 2,180,687 and 2,204,868 represent a significant point of departure from that methodology through the use of meaningful phrases as the classification parameter. Before describing the method for determining such meaningful phrases, it is useful to define two types of errors experienced in such an automated routing system and a related "success" concept:
False detection of a routing objective can occur when a salient (meaningful) phrase related to one routing objective is detected in a caller's input speech when the caller's actual request was directed to another routing objective. The probability of such a false detection occurring will hereafter be referred to by the designation: PFD
Missed detection of a routing objective occurs when the callers input speech is directed to that routing objective an~~ none of the meaningful phrases which are associated with that routing objective are detected in the input speech. The probability of such a missed detection occurring will hereafter be 5 referred to by the designation. PMD
Coverage for a routing objective refers to the number of successful translations by the system of a request for a routing objective to that routing objective relative to the total number of input 10 requests for that routing objective. As an illustrative example, a routing objective for which 60 successful translations occurred out of 100 input requests for that routing objective would be said to experience 60o coverage. It is noted that Coverage 1-P~
Of the two erro:_ types defined above, one is significantly more "costly" than the other. The consequence of a false detection error is the routing of a use~~. to a different task objective than was requested by the user. Such a result is at a minimum very annoying to the user. The possibility also exists that such an error could result in a direct cost to the system provider -- an annoyed customer or potential customer being classified here as an indirect cost -- through some non-system error resulting from the caller being connected to an incorrect rouging objective. The consequence of a missed detection error, on the other hand, is simply the routing of the user to a default operator po=>ition and the only cost is the lost: opportunity cost of r..ot handling that particular task on an automated basis. Thus, while ideally the probabilities of both missed detection and false detection should be near zero, it is far more important from the user's perspective that this objective be realized for __ false detection errors. As will be seen below, there are circumstances where tradeoffs must be made between minimizing one or another of these error probabilities, and this principle will be applied in such circumstances.
B. Adaptation o:E Baseline Approach Figure 1 provides several illustrative examples of False Detections and Missed Detections from the database of speech utterances used by the inventors. While the basis for error in each of these examples is believed to be largely self-explanatory, the error in the first example in each set will be briefly described. In the first example under False Detection, the meaningful phrase is I NE'ED_CREDIT-EOS (end of sentence), and thus this phrase would have been classified as a request for credit.
~5 However, from reading the entire utterance, it is apparent tha-~ the caller actually wanted to be transferred to another carrier (the carrier receiving this request being AT&'C). In the first exam~~le under Missed Detections, there are no meaningful phrases identified in the utterance (and therefore no basis for classifying the caller's objective), although it is apparent= to humans from reading the utterance that the caller is seeking a billing credit. As a comparative illustration, Figure 2 shows several examples of correct detection of a billing credit objective from meaningful phrases in the input speech.
There are two significant advantages of the methodology of the invention in using meaningful phrases as the classification parameter over the use of salient words in the baseline approach described in Gorin 95. First, with the use of words as vhe classification parameter, the word choices for detecting a given routing objective may be highly limited in order to achieve a minimum probability of fal~;e detection -- i.e., use of only words having a near __ 1000 likelihood of predicting the intended routing objective -- and therefore the coverage for such a routing objective is likely to be very low, leading to a high probability of missed detection errors occurring. With meaningful phrases as a classification parameter, on the other hand, both low probability of false detection and low probability of missed detection are achievable.
Figure 3 provides an illustrative example of this advantage. This figure shows the Classification Rate and the Coverage for an exen~.plary routing objective, Billing Credit, as the phrase used for the classification parameter grows in length and/or complexity. The Classification Rate is defined as the probability of the routing objective (CREDIT) having been requested, given the occurrence of the selected phrase in the input speech (i.e., P(CREDIT
phrase). Similarly, the Coverage term is defined as the pro;oability of the selected phrase appearing in the input speech, given that the designated routing objective (CREDIT
has been requested. In the Phrase column, parenthesis surrounding a series of terms separated by " I " indicate one of those terms appearing in the indicated position with other terms in that row. The nomenclature "F(Wrong)"
indicates a grammar fragment surrounding the word "wrong", the phrase in the fourth row of that column being representative of such a grammar fragment surrounding a salient word. The designation "previous" indicates a carry forward of everything on t:he preceding line. And finally, the abbreviation "eos" indicates "end of sentence."
In speech recognition systems, the larger the fragment of speech presenl~ed to such a speech- recognizes, the higher the probability of a correct recognition of that speech fragment. Thus a speech recognizes programmed to spot one of a set of salient words can be expected to fail in its task significantl:~ more often than such a device programmed to spot meaningful phrases, comprising two or more words.
C. Description of the Methodolo y of Invention The methodology of the invention is graphically illustrated in the flowchart of Figure 5. Following the steps of the flowchart, meaningful phrases are generated at step 110 from transcripticns of recognized utterances using est_Lmated posterior distributions over task types and a test: for significance, as set forth below. The meaningful phrases that have been generated are then clustered at step 120 and formed into meaningful phrase clusters using a distance measure between phrases, based on string and semantic distortion. Meaningful phrase clusters are then searched for and detected in input speech in step 130 using exacts or approximate m~~tching procedures. Task-type decisions are made at step 140 based on the detected meaningful phrase clusters.
D. Selection and Clustering of Meaningful Phrases 1. Selection of meaningful phrases It is desirable in a speech recognition system to select phrases that are me,3ningful for the task. A measure of salience may be used to assess, for a particular phrase, the distortion between the prior and posterior distributions over the call types. However, this does not take into account the frequency with which a fragment occurs. For example, a fortunate conjunction of events can give a low-frequency phrase a high apparent salience purely by chance.
Here, this shortcoming is avoided by testing, for each phrase, the null hypos:hesis that is governed simply by the prior probabilities (and therefore occurs at random).
Suppose a phrase f has a total of n occurrences of call-type labels in training, and let (rl,r2,...) denote the set of all possible partiti~~ns of n occurrences into K--15 classes. Let the actual observed distribution of counts for f be rf, and the prior distribution be denoted {Px~tx=z,...,x- Under the null. hypothesis, the probability of a partition ri=n11, ...,nl;~ .is given by the multinomial distribution:
x pkik P(r; n) = n 1 nik ~
~.=i A phrase f of frequency n is accepted at significance level a if ~P(r;ln)_a where A(f) _ {rr:P(rl rc) <_ P(rflrc)}
rt EA( f ) Any phrase for which the ~~bserved distribution can be seen to be a relatively likely random sample from the prior is therefore rejected. This is an exact test of significance, and is therefore valid even for phrases with very small occurrence counts. Impo:~ing a significance level of 5o reduces the total number of phrases generated in the prior art by about 300.
2. Meaningful phrase clustering The second step i~> to cluster the phrases using an aggl.omerative clustering procedure. For this we use a Levenshtein string distance measure ds (fi, fz) between phrases fl,f2, in which the insertion, deletion and __ substitution penalties are weighted by word salience.
However, phrases that a:re similar as strings can have different semantics, for example the phrases "need a credit" and "a credit card" indicate a billing credit 5 request and credit card payment, respectively. It would be undesirable for these phrases to enter the same cluster.
Again, the variability attributable to small samples must be considered in assessing this problem. Therefore, a measure of semantic distortion is used where 10 .f2 r)~
1 [P(c,E ~r f EF )-P(Cx W EF' 2 dM(.fi'.fi)=K~k yar[P(,ck ECI .fi EF)-P(~x ~r f2 EF)I
P(c~; E Ct) f E Ft) is the estimated posterior distribution over call types ck for phrases f, and Ct, Ft are the sets of labels and observed phrases for an utterance t. The 15 denominator is an estimate of the variance of the difference between the estimated posterior values (for each calf_ type) under hypothesis H. This hypothesis states that the two phrases have the :>ame true (but unknown) posterior distribution. If H is true then the expected value of dM(fi,f2) is equal to :1.O regardless of the phrase occurrence counts, so a .Large value for this measure is evidence for divergence between the posterior distributions . The overall measure used for clustering is a combination of the strings and semantic distortions.
Each of the resulting phrase clusters is converted into a Finite State MachinE= (FSM) representing a meaningful phrase cluster. An example is shown in figure 6. In this example, "make a collect call", is clustered with the other meaningful phrases, "place a collect call", "make collect call" and "make a collect phone call".
__ The posterior di~ctribution over the call types is then obtained for each FSM.. Observations in the form of exact or approximate matches to a path through the FSM are then found for the test utterances. The approximate matches are found using a dynamic programming algorithm in which word salience is used to weight the errors. An example of an approximate match is shown in figure 7, where the word a is substitute's by this, both words having low salience.
3. Exploiting multiple phrases In order to combine the evidence from several observations, there are ~,everal methods well known in the art. For example, the ca__1 types can be ranked by adopting a "bag-of-phrases" model:
m P(ck ECr I f EF ) P~Ck E Ct I fl ~. . .~ fm E F ) ;~ P~Ck E CI) T=1 Meaningful phrase clusters are often found to be correlated, and experiments involving higher-order star=istical models are in progress.
4. Classification Experiments The initial results were obtained using a set of meaningful phrases of length up to four words. A test set of 1000 utterances was processed by a large-vocabulary speech recognizer. Matches of phrases to the output were found, and parsed in a manner which favors longer phrases because these tend to be more salient. The call-type was determined by a peak-of-fragments classifier. One of the cal7_-type labels is other, and the desired goal is for the.>e particular calls t~~ be rejected. By varying the acceptance threshold and plotting correct classification __ rage against false rejection rate we obtain the ROC curves shown by the dotted lines in figure 8.
The solid lines in figure 8 show the results obtained using the procedures described above. A
significant improvement :in performance is seen, for the same output of the spee~~h recognition system. This is especially notable in the area of a useful operating point with 87o correct classification rate at rank 2, for 40%
false rejection rate.
E. Exemplary Structure for the Invention Figure 4 shows in block diagram form the essential structure of the invention. As can be seen from figure 4, that structure comprises two related subsystems:
Meaningful Phrase Generation Subsystem 1 and Input Speech Classification Subsystem 2. As already described, Meaningful Phrase Generav~ion Subsystem 1 operates on a database of a large number of utterances each of which is related to one of a predetermined set of routing objectives, where each such utterance is labeled with its associated routing objective. The operation of this subsystem is essentially carried out by Meaningful Phrase Selector 10 which selects as an output a set of meaningful phrases having a probabilistic relationship with one or more of the set of predetermined routing objectives with whi~~h the input speech utterances are associated. The selected meaningful phrasE:s are then input to a Meaningful Phrase Clustering Device 7.5 that clusters those meaningful phrases which are semanti~:ally related. The operation of Meaningful phrase Selectcr 10 and the Meaningful Phrase Clustering Device 15 are generally determined in accordance with the previously described algorithm for selecting and clustering meaningful phrases.
_.
x pkik P(r; n) = n 1 nik ~
~.=i A phrase f of frequency n is accepted at significance level a if ~P(r;ln)_a where A(f) _ {rr:P(rl rc) <_ P(rflrc)}
rt EA( f ) Any phrase for which the ~~bserved distribution can be seen to be a relatively likely random sample from the prior is therefore rejected. This is an exact test of significance, and is therefore valid even for phrases with very small occurrence counts. Impo:~ing a significance level of 5o reduces the total number of phrases generated in the prior art by about 300.
2. Meaningful phrase clustering The second step i~> to cluster the phrases using an aggl.omerative clustering procedure. For this we use a Levenshtein string distance measure ds (fi, fz) between phrases fl,f2, in which the insertion, deletion and __ substitution penalties are weighted by word salience.
However, phrases that a:re similar as strings can have different semantics, for example the phrases "need a credit" and "a credit card" indicate a billing credit 5 request and credit card payment, respectively. It would be undesirable for these phrases to enter the same cluster.
Again, the variability attributable to small samples must be considered in assessing this problem. Therefore, a measure of semantic distortion is used where 10 .f2 r)~
1 [P(c,E ~r f EF )-P(Cx W EF' 2 dM(.fi'.fi)=K~k yar[P(,ck ECI .fi EF)-P(~x ~r f2 EF)I
P(c~; E Ct) f E Ft) is the estimated posterior distribution over call types ck for phrases f, and Ct, Ft are the sets of labels and observed phrases for an utterance t. The 15 denominator is an estimate of the variance of the difference between the estimated posterior values (for each calf_ type) under hypothesis H. This hypothesis states that the two phrases have the :>ame true (but unknown) posterior distribution. If H is true then the expected value of dM(fi,f2) is equal to :1.O regardless of the phrase occurrence counts, so a .Large value for this measure is evidence for divergence between the posterior distributions . The overall measure used for clustering is a combination of the strings and semantic distortions.
Each of the resulting phrase clusters is converted into a Finite State MachinE= (FSM) representing a meaningful phrase cluster. An example is shown in figure 6. In this example, "make a collect call", is clustered with the other meaningful phrases, "place a collect call", "make collect call" and "make a collect phone call".
__ The posterior di~ctribution over the call types is then obtained for each FSM.. Observations in the form of exact or approximate matches to a path through the FSM are then found for the test utterances. The approximate matches are found using a dynamic programming algorithm in which word salience is used to weight the errors. An example of an approximate match is shown in figure 7, where the word a is substitute's by this, both words having low salience.
3. Exploiting multiple phrases In order to combine the evidence from several observations, there are ~,everal methods well known in the art. For example, the ca__1 types can be ranked by adopting a "bag-of-phrases" model:
m P(ck ECr I f EF ) P~Ck E Ct I fl ~. . .~ fm E F ) ;~ P~Ck E CI) T=1 Meaningful phrase clusters are often found to be correlated, and experiments involving higher-order star=istical models are in progress.
4. Classification Experiments The initial results were obtained using a set of meaningful phrases of length up to four words. A test set of 1000 utterances was processed by a large-vocabulary speech recognizer. Matches of phrases to the output were found, and parsed in a manner which favors longer phrases because these tend to be more salient. The call-type was determined by a peak-of-fragments classifier. One of the cal7_-type labels is other, and the desired goal is for the.>e particular calls t~~ be rejected. By varying the acceptance threshold and plotting correct classification __ rage against false rejection rate we obtain the ROC curves shown by the dotted lines in figure 8.
The solid lines in figure 8 show the results obtained using the procedures described above. A
significant improvement :in performance is seen, for the same output of the spee~~h recognition system. This is especially notable in the area of a useful operating point with 87o correct classification rate at rank 2, for 40%
false rejection rate.
E. Exemplary Structure for the Invention Figure 4 shows in block diagram form the essential structure of the invention. As can be seen from figure 4, that structure comprises two related subsystems:
Meaningful Phrase Generation Subsystem 1 and Input Speech Classification Subsystem 2. As already described, Meaningful Phrase Generav~ion Subsystem 1 operates on a database of a large number of utterances each of which is related to one of a predetermined set of routing objectives, where each such utterance is labeled with its associated routing objective. The operation of this subsystem is essentially carried out by Meaningful Phrase Selector 10 which selects as an output a set of meaningful phrases having a probabilistic relationship with one or more of the set of predetermined routing objectives with whi~~h the input speech utterances are associated. The selected meaningful phrasE:s are then input to a Meaningful Phrase Clustering Device 7.5 that clusters those meaningful phrases which are semanti~:ally related. The operation of Meaningful phrase Selectcr 10 and the Meaningful Phrase Clustering Device 15 are generally determined in accordance with the previously described algorithm for selecting and clustering meaningful phrases.
_.
Operation of Input Speech Classification Subsystem 2 begins with the inputt=ing of a user's task objective request, in the caller's natural speech, to Input Speech Recognizer 20. The Input. Speech Recognizer 20 may be of any known design and perf~~rms the function of recognizing, or .spotting, the existence of one or more meaningful phrase in the input speech. A Meaningful Phrase Cluster Detector 25 then detects the meaningful phrase clusters present among the meaningful phrases recognized. As can be seen in the figure, the meaningful phrases clusters developed by Meaningful Phrase Generation Subsystem 1 are provided as an input to Meaningful Phrase Cluster Detector 25.
The output of Meaningful Phrase Detector 25, which wil:L comprise the detect=ed meaningful phrases clusters appearing in the caller's routing objective request, is provided to Classification Processor 30. The Cla:>sification Processor 30 may apply a confidence function, based on the probabilistic relation between the recognized meaningful phrases clusters and selected task objectives, and makes a decision either to implement a particular task objective or a determination that no decision is likely, in which case the user may be defaulted to an operator position.
As will thus be apparent, the meaningful phrase clu:>ters developed by Meaningful Phrase Generation Sub:~ystem 1 are used by the Meaningful Phrase Cluster Detector 25, to define the meaningful phrase clusters which the Detector is programmed to spot, and to Classification Processor 30, defining t:he task objectives related to meaningful phrases input from the Meaningful Phrase Cluster Detector 25 and, if warranted, for establishing the level of confidence for a relation of such input meaningful phrase clusters to a particular task objective.
_, CONfCLUS ION
A method for automated task selection has been disclosed which carries out the function of searching for a classification parameter in natural speech. These classification parameter:> are presented in the form of compact meaningful phrase clusters which are generated from a :>et of selected meaningful phrases. By automatically selecting and clustering meaningful phrases, the overall accuracy of automated roui.ing systems is increased.
Although the pre:~ent embodiment of the invention has been described in detail, it should be understood that various changes, alterations and substitutions can be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
_.
The output of Meaningful Phrase Detector 25, which wil:L comprise the detect=ed meaningful phrases clusters appearing in the caller's routing objective request, is provided to Classification Processor 30. The Cla:>sification Processor 30 may apply a confidence function, based on the probabilistic relation between the recognized meaningful phrases clusters and selected task objectives, and makes a decision either to implement a particular task objective or a determination that no decision is likely, in which case the user may be defaulted to an operator position.
As will thus be apparent, the meaningful phrase clu:>ters developed by Meaningful Phrase Generation Sub:~ystem 1 are used by the Meaningful Phrase Cluster Detector 25, to define the meaningful phrase clusters which the Detector is programmed to spot, and to Classification Processor 30, defining t:he task objectives related to meaningful phrases input from the Meaningful Phrase Cluster Detector 25 and, if warranted, for establishing the level of confidence for a relation of such input meaningful phrase clusters to a particular task objective.
_, CONfCLUS ION
A method for automated task selection has been disclosed which carries out the function of searching for a classification parameter in natural speech. These classification parameter:> are presented in the form of compact meaningful phrase clusters which are generated from a :>et of selected meaningful phrases. By automatically selecting and clustering meaningful phrases, the overall accuracy of automated roui.ing systems is increased.
Although the pre:~ent embodiment of the invention has been described in detail, it should be understood that various changes, alterations and substitutions can be made therein without departing from the spirit and scope of the invention as defined by the appended claims.
_.
Claims (30)
1. A method for automated ~task classification which operates on a task objective of a user, the task objective being expressed in natural speech of the user, comprising:
selecting meaningful phrases from a plurality of transcribed speech utterances;
clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
detecting meaningful phrase clusters present in input speech utterances; and making task-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
selecting meaningful phrases from a plurality of transcribed speech utterances;
clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
detecting meaningful phrase clusters present in input speech utterances; and making task-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
2. The automated task classification method of Claim 1, wherein the meaningful phrases are selected using a test for significance.
3. The automated task classification method of Claim 1, wherein the meaningful phrases are clustered using a distortion measure between the meaningful phrases.
4. The automated task classification method of Claim 3, wherein the distortion measure is based on string distortion.
5. The automated task classification method of Claim 3, wherein the distortion measure is based on semantic distortion.
6. The automated task classification method of Claim 3, wherein the distortion measure is a combination of string and semantic distortion.
7. The automated task classification method of Claim 1, wherein the meaningful phrase clusters are detected in input speech by exactly matching the input speech to any of the meaningful phrase clusters.
8. The automated task classification method of Claim 1, wherein clusters are detected in input speech by exactly or approximately matching the input speech to any of the meaningful phrase clusters.
9. The automated task classification method of Claim 1, wherein the task-type classification decisions are ranked using a statistical model.
10. The automated task classification method of Claim 1, further comprising a neural network for making the task-type classification decisions.
11. A method for automated call-type classification which operates on the call routing objective of a user, the call-routing objective being expressed in natural speech of the user, comprising:
selecting meaningful phrases from a plurality of transcribed speech utterances, clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
detecting meaningful phrase clusters present in input speech utterances; and making call-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
selecting meaningful phrases from a plurality of transcribed speech utterances, clustering selected ones of the meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
detecting meaningful phrase clusters present in input speech utterances; and making call-type classification decisions based on the detected meaningful phrase clusters in the input speech utterances.
12. The automated call-type classification method of Claim 11, wherein the meaningful phrases are selected using a test for significance.
13. The automated call-type classification method of Claim 11, wherein the meaningful phrases are clustered using a distortion measure between meaningful phrases.
14. The automated call-type classification method of Claim 13, wherein the distortion measure is based on string distortion.
15. The automated call-type classification method of Claim 13, wherein the distortion measure is based on semantic distortion.
16. The automated call-type classification method of Claim 13, wherein the distortion measure is a combination of string and semantic distortion.
17. The automated call-type classification method of Claim 11, wherein the meaningful phrase clusters are detected in input speech by exactly matching the input speech to any of the meaningful phrase clusters.
18. The automated call-type classification method of Claim 11, wherein the meaningful phrase clusters are detected in input speech by exactly or approximately matching the input speech to any of the meaningful phrase clusters.
19. The automated call-type classification method of Claim 11, wherein call-type classification decisions are ranked using a statistical model.
20. The automated call-type classification method of Claim 11, wherein the call-type classification decisions are made using a neural network.
21. An automated task classification system which operates on task objectives of a user, the task objectives being expressed in natural speech of the user, comprising:
a meaningful phrase selector that selects a plurality of meaningful phrases from a set of speech utterances, each of the meaningful phrases being selected based on one of a predetermined set of the task objectives;
a meaningful phrase clustering device for clustering selected meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
an input recognizing device for detecting any of the clustered meaningful phrases in input speech of the user, the input recognizing device having as an input the meaningful phrases clustered by the meaningful phrase clustering device; and a classification device responsive to an input of the detected meaningful phrase clusters for making a classification decision based on the detected meaningful phrase clusters as to one of the set of predetermined task objectives.
a meaningful phrase selector that selects a plurality of meaningful phrases from a set of speech utterances, each of the meaningful phrases being selected based on one of a predetermined set of the task objectives;
a meaningful phrase clustering device for clustering selected meaningful phrases which occur in similar semantic contexts into meaningful phrase clusters;
an input recognizing device for detecting any of the clustered meaningful phrases in input speech of the user, the input recognizing device having as an input the meaningful phrases clustered by the meaningful phrase clustering device; and a classification device responsive to an input of the detected meaningful phrase clusters for making a classification decision based on the detected meaningful phrase clusters as to one of the set of predetermined task objectives.
22. The automated task classification system of Claim 21 wherein the meaningful phrase selector selects meaningful phrases using a test for significance.
23. The automated task classification system of Claim 21 wherein the meaningful phrase clustering device clusters meaningful phrases using a distortion measure between meaningful phrases.
24. The automated task classification system of Claim 23, wherein the distortion measure is based on string distortion.
25. The automated task classification system of Claim 23, wherein the distortion measure is based on semantic distortion.
26. The automated task classification system of Claim 23, wherein the distortion measure is a combination of string and semantic distortion.
27. The automated task classification system of Claim 21, wherein the input recognizing device detects meaningful phrase clusters in input speech by exactly matching the input speech to any of the meaningful phrase clusters.
28. The automated task classification system of Claim 21, wherein the input recognizing device detects meaningful phrase clusters in input speech by exactly or approximately matching the input speech to any of the meaningful phrase clusters.
29. The automated task classification system of Claim 21, wherein the classification device ranks decisions using a statistical model.
30. The automated task classification system of Claim 21, wherein the classification device makes decisions using a neural network.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US08/893,888 US5860063A (en) | 1997-07-11 | 1997-07-11 | Automated meaningful phrase clustering |
US08/893,888 | 1997-07-11 |
Publications (2)
Publication Number | Publication Date |
---|---|
CA2235364A1 CA2235364A1 (en) | 1999-01-11 |
CA2235364C true CA2235364C (en) | 2002-07-16 |
Family
ID=25402301
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CA002235364A Expired - Lifetime CA2235364C (en) | 1997-07-11 | 1998-04-20 | Automated meaningful phrase clustering |
Country Status (6)
Country | Link |
---|---|
US (1) | US5860063A (en) |
EP (1) | EP0890942B1 (en) |
JP (1) | JP4053141B2 (en) |
CA (1) | CA2235364C (en) |
DE (1) | DE69818161T2 (en) |
ES (1) | ES2210623T3 (en) |
Families Citing this family (211)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6173261B1 (en) * | 1998-09-30 | 2001-01-09 | At&T Corp | Grammar fragment acquisition using syntactic and semantic clustering |
US7590224B1 (en) * | 1995-09-15 | 2009-09-15 | At&T Intellectual Property, Ii, L.P. | Automated task classification system |
US6044337A (en) * | 1997-10-29 | 2000-03-28 | At&T Corp | Selection of superwords based on criteria relevant to both speech recognition and understanding |
US6012049A (en) * | 1998-02-04 | 2000-01-04 | Citicorp Development Center, Inc. | System for performing financial transactions using a smartcard |
US6317707B1 (en) * | 1998-12-07 | 2001-11-13 | At&T Corp. | Automatic clustering of tokens from a corpus for grammar acquisition |
US7356462B2 (en) | 2001-07-26 | 2008-04-08 | At&T Corp. | Automatic clustering of tokens from a corpus for grammar acquisition |
US6246986B1 (en) * | 1998-12-31 | 2001-06-12 | At&T Corp. | User barge-in enablement in large vocabulary speech recognition systems |
US6519562B1 (en) * | 1999-02-25 | 2003-02-11 | Speechworks International, Inc. | Dynamic semantic control of a speech recognition system |
US20020032564A1 (en) * | 2000-04-19 | 2002-03-14 | Farzad Ehsani | Phrase-based dialogue modeling with particular application to creating a recognition grammar for a voice-controlled user interface |
DE60040536D1 (en) | 1999-06-11 | 2008-11-27 | Telstra Corp Ltd | PROCESS FOR DEVELOPING AN INTERACTIVE SYSTEM |
US6510411B1 (en) * | 1999-10-29 | 2003-01-21 | Unisys Corporation | Task oriented dialog model and manager |
US8392188B1 (en) | 1999-11-05 | 2013-03-05 | At&T Intellectual Property Ii, L.P. | Method and system for building a phonotactic model for domain independent speech recognition |
US7085720B1 (en) * | 1999-11-05 | 2006-08-01 | At & T Corp. | Method for task classification using morphemes |
US6681206B1 (en) | 1999-11-05 | 2004-01-20 | At&T Corporation | Method for generating morphemes |
US7286984B1 (en) | 1999-11-05 | 2007-10-23 | At&T Corp. | Method and system for automatically detecting morphemes in a task classification system using lattices |
US20030191625A1 (en) * | 1999-11-05 | 2003-10-09 | Gorin Allen Louis | Method and system for creating a named entity language model |
US9076448B2 (en) | 1999-11-12 | 2015-07-07 | Nuance Communications, Inc. | Distributed real time speech recognition system |
US7392185B2 (en) * | 1999-11-12 | 2008-06-24 | Phoenix Solutions, Inc. | Speech based learning/training system using semantic decoding |
US7725307B2 (en) * | 1999-11-12 | 2010-05-25 | Phoenix Solutions, Inc. | Query engine for processing voice based queries including semantic decoding |
US7050977B1 (en) * | 1999-11-12 | 2006-05-23 | Phoenix Solutions, Inc. | Speech-enabled server for internet website and method |
US6526382B1 (en) * | 1999-12-07 | 2003-02-25 | Comverse, Inc. | Language-oriented user interfaces for voice activated services |
DE60040015D1 (en) | 1999-12-20 | 2008-10-02 | British Telecomm | LEARNING DIALOGUE STATE AND LANGUAGE MODELS OF THE DESIRED INFORMATION SYSTEM |
US8645137B2 (en) | 2000-03-16 | 2014-02-04 | Apple Inc. | Fast, language-independent method for user authentication by voice |
US7149695B1 (en) * | 2000-10-13 | 2006-12-12 | Apple Computer, Inc. | Method and apparatus for speech recognition using semantic inference and word agglomeration |
US6941266B1 (en) | 2000-11-15 | 2005-09-06 | At&T Corp. | Method and system for predicting problematic dialog situations in a task classification system |
US7184947B2 (en) * | 2001-01-05 | 2007-02-27 | Fujitsu Limited | Document anonymity setting device, method and computer readable recording medium recording anonymity setting program |
US6751591B1 (en) | 2001-01-22 | 2004-06-15 | At&T Corp. | Method and system for predicting understanding errors in a task classification system |
WO2002073449A1 (en) * | 2001-03-14 | 2002-09-19 | At & T Corp. | Automated sentence planning in a task classification system |
US7729918B2 (en) * | 2001-03-14 | 2010-06-01 | At&T Intellectual Property Ii, Lp | Trainable sentence planning system |
US7574362B2 (en) | 2001-03-14 | 2009-08-11 | At&T Intellectual Property Ii, L.P. | Method for automated sentence planning in a task classification system |
US6848542B2 (en) * | 2001-04-27 | 2005-02-01 | Accenture Llp | Method for passive mining of usage information in a location-based services system |
US7970648B2 (en) * | 2001-04-27 | 2011-06-28 | Accenture Global Services Limited | Advertising campaign and business listing management for a location-based services system |
US7698228B2 (en) | 2001-04-27 | 2010-04-13 | Accenture Llp | Tracking purchases in a location-based services system |
US6944447B2 (en) * | 2001-04-27 | 2005-09-13 | Accenture Llp | Location-based services |
US7437295B2 (en) * | 2001-04-27 | 2008-10-14 | Accenture Llp | Natural language processing for a location-based services system |
ITFI20010199A1 (en) | 2001-10-22 | 2003-04-22 | Riccardo Vieri | SYSTEM AND METHOD TO TRANSFORM TEXTUAL COMMUNICATIONS INTO VOICE AND SEND THEM WITH AN INTERNET CONNECTION TO ANY TELEPHONE SYSTEM |
AU2002950336A0 (en) * | 2002-07-24 | 2002-09-12 | Telstra New Wave Pty Ltd | System and process for developing a voice application |
AU2002951244A0 (en) * | 2002-09-06 | 2002-09-19 | Telstra New Wave Pty Ltd | A development system for a dialog system |
US7263486B1 (en) * | 2002-10-25 | 2007-08-28 | At&T Corp. | Active learning for spoken language understanding |
US20030115062A1 (en) * | 2002-10-29 | 2003-06-19 | Walker Marilyn A. | Method for automated sentence planning |
US8645122B1 (en) * | 2002-12-19 | 2014-02-04 | At&T Intellectual Property Ii, L.P. | Method of handling frequently asked questions in a natural language dialog service |
US20040122661A1 (en) * | 2002-12-23 | 2004-06-24 | Gensym Corporation | Method, system, and computer program product for storing, managing and using knowledge expressible as, and organized in accordance with, a natural language |
US8335683B2 (en) * | 2003-01-23 | 2012-12-18 | Microsoft Corporation | System for using statistical classifiers for spoken language understanding |
US20040148170A1 (en) * | 2003-01-23 | 2004-07-29 | Alejandro Acero | Statistical classifiers for spoken language understanding and command/control scenarios |
AU2003900584A0 (en) * | 2003-02-11 | 2003-02-27 | Telstra New Wave Pty Ltd | System for predicting speech recognition accuracy and development for a dialog system |
AU2003902020A0 (en) * | 2003-04-29 | 2003-05-15 | Telstra New Wave Pty Ltd | A process for grammatical inference |
US20050010416A1 (en) * | 2003-07-09 | 2005-01-13 | Gensym Corporation | System and method for self management of health using natural language interface |
US7933774B1 (en) * | 2004-03-18 | 2011-04-26 | At&T Intellectual Property Ii, L.P. | System and method for automatic generation of a natural language understanding model |
DE102004055230B3 (en) * | 2004-11-16 | 2006-07-20 | Siemens Ag | Method for speech recognition from a predefinable vocabulary |
US7634406B2 (en) | 2004-12-10 | 2009-12-15 | Microsoft Corporation | System and method for identifying semantic intent from acoustic information |
US8677377B2 (en) | 2005-09-08 | 2014-03-18 | Apple Inc. | Method and apparatus for building an intelligent automated assistant |
US7633076B2 (en) | 2005-09-30 | 2009-12-15 | Apple Inc. | Automated response to and sensing of user activity in portable devices |
EP2008268A4 (en) * | 2006-04-17 | 2010-12-22 | Call Genie Inc | Method and apparatus for building grammars with lexical semantic clustering in a speech recognizer |
US9245526B2 (en) * | 2006-04-25 | 2016-01-26 | General Motors Llc | Dynamic clustering of nametags in an automated speech recognition system |
US9318108B2 (en) | 2010-01-18 | 2016-04-19 | Apple Inc. | Intelligent automated assistant |
US20080091423A1 (en) * | 2006-10-13 | 2008-04-17 | Shourya Roy | Generation of domain models from noisy transcriptions |
US20080129520A1 (en) * | 2006-12-01 | 2008-06-05 | Apple Computer, Inc. | Electronic device with enhanced audio feedback |
US8380511B2 (en) * | 2007-02-20 | 2013-02-19 | Intervoice Limited Partnership | System and method for semantic categorization |
US8977255B2 (en) | 2007-04-03 | 2015-03-10 | Apple Inc. | Method and system for operating a multi-function portable electronic device using voice-activation |
US9053089B2 (en) * | 2007-10-02 | 2015-06-09 | Apple Inc. | Part-of-speech tagging using latent analogy |
US8620662B2 (en) | 2007-11-20 | 2013-12-31 | Apple Inc. | Context-aware unit selection |
US10002189B2 (en) * | 2007-12-20 | 2018-06-19 | Apple Inc. | Method and apparatus for searching using an active ontology |
US9330720B2 (en) | 2008-01-03 | 2016-05-03 | Apple Inc. | Methods and apparatus for altering audio output signals |
US8065143B2 (en) | 2008-02-22 | 2011-11-22 | Apple Inc. | Providing text input using speech data and non-speech data |
US8996376B2 (en) | 2008-04-05 | 2015-03-31 | Apple Inc. | Intelligent text-to-speech conversion |
US10496753B2 (en) | 2010-01-18 | 2019-12-03 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US8464150B2 (en) | 2008-06-07 | 2013-06-11 | Apple Inc. | Automatic language identification for dynamic text processing |
US20100030549A1 (en) | 2008-07-31 | 2010-02-04 | Lee Michael M | Mobile device having human language translation capability with positional feedback |
US8768702B2 (en) * | 2008-09-05 | 2014-07-01 | Apple Inc. | Multi-tiered voice feedback in an electronic device |
US8898568B2 (en) * | 2008-09-09 | 2014-11-25 | Apple Inc. | Audio user interface |
US8712776B2 (en) * | 2008-09-29 | 2014-04-29 | Apple Inc. | Systems and methods for selective text to speech synthesis |
US8583418B2 (en) | 2008-09-29 | 2013-11-12 | Apple Inc. | Systems and methods of detecting language and natural language strings for text to speech synthesis |
US8676904B2 (en) | 2008-10-02 | 2014-03-18 | Apple Inc. | Electronic devices with voice command and contextual data processing capabilities |
US9959870B2 (en) | 2008-12-11 | 2018-05-01 | Apple Inc. | Speech recognition involving a mobile device |
US8862252B2 (en) | 2009-01-30 | 2014-10-14 | Apple Inc. | Audio user interface for displayless electronic device |
US8380507B2 (en) | 2009-03-09 | 2013-02-19 | Apple Inc. | Systems and methods for determining the language to use for speech generated by a text to speech engine |
US10241644B2 (en) | 2011-06-03 | 2019-03-26 | Apple Inc. | Actionable reminder entries |
US10241752B2 (en) | 2011-09-30 | 2019-03-26 | Apple Inc. | Interface for a virtual digital assistant |
US20120311585A1 (en) | 2011-06-03 | 2012-12-06 | Apple Inc. | Organizing task items that represent tasks to perform |
US9858925B2 (en) | 2009-06-05 | 2018-01-02 | Apple Inc. | Using context information to facilitate processing of commands in a virtual assistant |
US10540976B2 (en) * | 2009-06-05 | 2020-01-21 | Apple Inc. | Contextual voice commands |
US9431006B2 (en) * | 2009-07-02 | 2016-08-30 | Apple Inc. | Methods and apparatuses for automatic speech recognition |
US8682649B2 (en) * | 2009-11-12 | 2014-03-25 | Apple Inc. | Sentiment prediction from textual data |
US8600743B2 (en) * | 2010-01-06 | 2013-12-03 | Apple Inc. | Noise profile determination for voice-related feature |
US8311838B2 (en) | 2010-01-13 | 2012-11-13 | Apple Inc. | Devices and methods for identifying a prompt corresponding to a voice input in a sequence of prompts |
US8381107B2 (en) | 2010-01-13 | 2013-02-19 | Apple Inc. | Adaptive audio feedback system and method |
US10553209B2 (en) | 2010-01-18 | 2020-02-04 | Apple Inc. | Systems and methods for hands-free notification summaries |
US10276170B2 (en) | 2010-01-18 | 2019-04-30 | Apple Inc. | Intelligent automated assistant |
US10705794B2 (en) | 2010-01-18 | 2020-07-07 | Apple Inc. | Automatically adapting user interfaces for hands-free interaction |
US10679605B2 (en) | 2010-01-18 | 2020-06-09 | Apple Inc. | Hands-free list-reading by intelligent automated assistant |
DE202011111062U1 (en) | 2010-01-25 | 2019-02-19 | Newvaluexchange Ltd. | Device and system for a digital conversation management platform |
US8682667B2 (en) | 2010-02-25 | 2014-03-25 | Apple Inc. | User profiling for selecting user specific voice input processing information |
US8713021B2 (en) | 2010-07-07 | 2014-04-29 | Apple Inc. | Unsupervised document clustering using latent semantic density analysis |
US8661018B2 (en) | 2010-08-10 | 2014-02-25 | Lockheed Martin Corporation | Data service response plan generator |
US8719006B2 (en) | 2010-08-27 | 2014-05-06 | Apple Inc. | Combined statistical and rule-based part-of-speech tagging for text-to-speech synthesis |
US8719014B2 (en) | 2010-09-27 | 2014-05-06 | Apple Inc. | Electronic device with text error correction based on voice recognition data |
US10762293B2 (en) | 2010-12-22 | 2020-09-01 | Apple Inc. | Using parts-of-speech tagging and named entity recognition for spelling correction |
US10515147B2 (en) | 2010-12-22 | 2019-12-24 | Apple Inc. | Using statistical language models for contextual lookup |
US9330665B2 (en) * | 2011-01-07 | 2016-05-03 | Nuance Communications, Inc. | Automatic updating of confidence scoring functionality for speech recognition systems with respect to a receiver operating characteristic curve |
US8781836B2 (en) | 2011-02-22 | 2014-07-15 | Apple Inc. | Hearing assistance system for providing consistent human speech |
US9262612B2 (en) | 2011-03-21 | 2016-02-16 | Apple Inc. | Device access using voice authentication |
US10672399B2 (en) | 2011-06-03 | 2020-06-02 | Apple Inc. | Switching between text data and audio data based on a mapping |
US10057736B2 (en) | 2011-06-03 | 2018-08-21 | Apple Inc. | Active transport based notifications |
US8812294B2 (en) | 2011-06-21 | 2014-08-19 | Apple Inc. | Translating phrases from one language into another using an order-based set of declarative rules |
US8706472B2 (en) | 2011-08-11 | 2014-04-22 | Apple Inc. | Method for disambiguating multiple readings in language conversion |
US8994660B2 (en) | 2011-08-29 | 2015-03-31 | Apple Inc. | Text correction processing |
US8762156B2 (en) | 2011-09-28 | 2014-06-24 | Apple Inc. | Speech recognition repair using contextual information |
US10134385B2 (en) | 2012-03-02 | 2018-11-20 | Apple Inc. | Systems and methods for name pronunciation |
US9483461B2 (en) | 2012-03-06 | 2016-11-01 | Apple Inc. | Handling speech synthesis of content for multiple languages |
US9280610B2 (en) | 2012-05-14 | 2016-03-08 | Apple Inc. | Crowd sourcing information to fulfill user requests |
US8775442B2 (en) | 2012-05-15 | 2014-07-08 | Apple Inc. | Semantic search using a single-source semantic model |
US10417037B2 (en) | 2012-05-15 | 2019-09-17 | Apple Inc. | Systems and methods for integrating third party services with a digital assistant |
WO2013185109A2 (en) | 2012-06-08 | 2013-12-12 | Apple Inc. | Systems and methods for recognizing textual identifiers within a plurality of words |
US9721563B2 (en) | 2012-06-08 | 2017-08-01 | Apple Inc. | Name recognition system |
US9495129B2 (en) | 2012-06-29 | 2016-11-15 | Apple Inc. | Device, method, and user interface for voice-activated navigation and browsing of a document |
US9576574B2 (en) | 2012-09-10 | 2017-02-21 | Apple Inc. | Context-sensitive handling of interruptions by intelligent digital assistant |
US9547647B2 (en) | 2012-09-19 | 2017-01-17 | Apple Inc. | Voice-based media searching |
US8935167B2 (en) | 2012-09-25 | 2015-01-13 | Apple Inc. | Exemplar-based latent perceptual modeling for automatic speech recognition |
BR112015018905B1 (en) | 2013-02-07 | 2022-02-22 | Apple Inc | Voice activation feature operation method, computer readable storage media and electronic device |
US9733821B2 (en) | 2013-03-14 | 2017-08-15 | Apple Inc. | Voice control to diagnose inadvertent activation of accessibility features |
US9977779B2 (en) | 2013-03-14 | 2018-05-22 | Apple Inc. | Automatic supplementation of word correction dictionaries |
US9368114B2 (en) | 2013-03-14 | 2016-06-14 | Apple Inc. | Context-sensitive handling of interruptions |
US10572476B2 (en) | 2013-03-14 | 2020-02-25 | Apple Inc. | Refining a search based on schedule items |
US10642574B2 (en) | 2013-03-14 | 2020-05-05 | Apple Inc. | Device, method, and graphical user interface for outputting captions |
US10652394B2 (en) | 2013-03-14 | 2020-05-12 | Apple Inc. | System and method for processing voicemail |
WO2014144579A1 (en) | 2013-03-15 | 2014-09-18 | Apple Inc. | System and method for updating an adaptive speech recognition model |
US11151899B2 (en) | 2013-03-15 | 2021-10-19 | Apple Inc. | User training by intelligent digital assistant |
AU2014251347B2 (en) | 2013-03-15 | 2017-05-18 | Apple Inc. | Context-sensitive handling of interruptions |
KR101759009B1 (en) | 2013-03-15 | 2017-07-17 | 애플 인크. | Training an at least partial voice command system |
US10748529B1 (en) | 2013-03-15 | 2020-08-18 | Apple Inc. | Voice activated device for use with a voice-based digital assistant |
WO2014197336A1 (en) | 2013-06-07 | 2014-12-11 | Apple Inc. | System and method for detecting errors in interactions with a voice-based digital assistant |
US9582608B2 (en) | 2013-06-07 | 2017-02-28 | Apple Inc. | Unified ranking with entropy-weighted information for phrase-based semantic auto-completion |
WO2014197334A2 (en) | 2013-06-07 | 2014-12-11 | Apple Inc. | System and method for user-specified pronunciation of words for speech synthesis and recognition |
WO2014197335A1 (en) | 2013-06-08 | 2014-12-11 | Apple Inc. | Interpreting and acting upon commands that involve sharing information with remote devices |
US10176167B2 (en) | 2013-06-09 | 2019-01-08 | Apple Inc. | System and method for inferring user intent from speech inputs |
CN105264524B (en) | 2013-06-09 | 2019-08-02 | 苹果公司 | For realizing the equipment, method and graphic user interface of the session continuity of two or more examples across digital assistants |
CN105265005B (en) | 2013-06-13 | 2019-09-17 | 苹果公司 | System and method for the urgent call initiated by voice command |
CN105745679B (en) * | 2013-07-26 | 2020-01-14 | 格林伊登美国控股有限责任公司 | System and method for discovering and exploring concepts |
US10061822B2 (en) | 2013-07-26 | 2018-08-28 | Genesys Telecommunications Laboratories, Inc. | System and method for discovering and exploring concepts and root causes of events |
JP6163266B2 (en) | 2013-08-06 | 2017-07-12 | アップル インコーポレイテッド | Automatic activation of smart responses based on activation from remote devices |
US10296160B2 (en) | 2013-12-06 | 2019-05-21 | Apple Inc. | Method for extracting salient dialog usage from live data |
KR20150081981A (en) * | 2014-01-07 | 2015-07-15 | 삼성전자주식회사 | Apparatus and Method for structuring contents of meeting |
US9620105B2 (en) | 2014-05-15 | 2017-04-11 | Apple Inc. | Analyzing audio input for efficient speech and music recognition |
US10592095B2 (en) | 2014-05-23 | 2020-03-17 | Apple Inc. | Instantaneous speaking of content on touch devices |
US9502031B2 (en) | 2014-05-27 | 2016-11-22 | Apple Inc. | Method for supporting dynamic grammars in WFST-based ASR |
US10078631B2 (en) | 2014-05-30 | 2018-09-18 | Apple Inc. | Entropy-guided text prediction using combined word and character n-gram language models |
EP3149728B1 (en) | 2014-05-30 | 2019-01-16 | Apple Inc. | Multi-command single utterance input method |
US10289433B2 (en) | 2014-05-30 | 2019-05-14 | Apple Inc. | Domain specific language for encoding assistant dialog |
US10170123B2 (en) | 2014-05-30 | 2019-01-01 | Apple Inc. | Intelligent assistant for home automation |
US9715875B2 (en) | 2014-05-30 | 2017-07-25 | Apple Inc. | Reducing the need for manual start/end-pointing and trigger phrases |
US9842101B2 (en) | 2014-05-30 | 2017-12-12 | Apple Inc. | Predictive conversion of language input |
US9734193B2 (en) | 2014-05-30 | 2017-08-15 | Apple Inc. | Determining domain salience ranking from ambiguous words in natural speech |
US9430463B2 (en) | 2014-05-30 | 2016-08-30 | Apple Inc. | Exemplar-based natural language processing |
US9760559B2 (en) | 2014-05-30 | 2017-09-12 | Apple Inc. | Predictive text input |
US9785630B2 (en) | 2014-05-30 | 2017-10-10 | Apple Inc. | Text prediction using combined word N-gram and unigram language models |
US9633004B2 (en) | 2014-05-30 | 2017-04-25 | Apple Inc. | Better resolution when referencing to concepts |
US9338493B2 (en) | 2014-06-30 | 2016-05-10 | Apple Inc. | Intelligent automated assistant for TV user interactions |
US10659851B2 (en) | 2014-06-30 | 2020-05-19 | Apple Inc. | Real-time digital assistant knowledge updates |
US10446141B2 (en) | 2014-08-28 | 2019-10-15 | Apple Inc. | Automatic speech recognition based on user feedback |
US9818400B2 (en) | 2014-09-11 | 2017-11-14 | Apple Inc. | Method and apparatus for discovering trending terms in speech requests |
US10789041B2 (en) | 2014-09-12 | 2020-09-29 | Apple Inc. | Dynamic thresholds for always listening speech trigger |
US9668121B2 (en) | 2014-09-30 | 2017-05-30 | Apple Inc. | Social reminders |
US10127911B2 (en) | 2014-09-30 | 2018-11-13 | Apple Inc. | Speaker identification and unsupervised speaker adaptation techniques |
US9646609B2 (en) | 2014-09-30 | 2017-05-09 | Apple Inc. | Caching apparatus for serving phonetic pronunciations |
US10074360B2 (en) | 2014-09-30 | 2018-09-11 | Apple Inc. | Providing an indication of the suitability of speech recognition |
US9886432B2 (en) | 2014-09-30 | 2018-02-06 | Apple Inc. | Parsimonious handling of word inflection via categorical stem + suffix N-gram language models |
US10552013B2 (en) | 2014-12-02 | 2020-02-04 | Apple Inc. | Data detection |
US9711141B2 (en) | 2014-12-09 | 2017-07-18 | Apple Inc. | Disambiguating heteronyms in speech synthesis |
US9865280B2 (en) | 2015-03-06 | 2018-01-09 | Apple Inc. | Structured dictation using intelligent automated assistants |
US9886953B2 (en) | 2015-03-08 | 2018-02-06 | Apple Inc. | Virtual assistant activation |
US10567477B2 (en) | 2015-03-08 | 2020-02-18 | Apple Inc. | Virtual assistant continuity |
US9721566B2 (en) | 2015-03-08 | 2017-08-01 | Apple Inc. | Competing devices responding to voice triggers |
US9899019B2 (en) | 2015-03-18 | 2018-02-20 | Apple Inc. | Systems and methods for structured stem and suffix language models |
US9842105B2 (en) | 2015-04-16 | 2017-12-12 | Apple Inc. | Parsimonious continuous-space phrase representations for natural language processing |
US10083688B2 (en) | 2015-05-27 | 2018-09-25 | Apple Inc. | Device voice control for selecting a displayed affordance |
US10127220B2 (en) | 2015-06-04 | 2018-11-13 | Apple Inc. | Language identification from short strings |
US10101822B2 (en) | 2015-06-05 | 2018-10-16 | Apple Inc. | Language input correction |
US10255907B2 (en) | 2015-06-07 | 2019-04-09 | Apple Inc. | Automatic accent detection using acoustic models |
US10186254B2 (en) | 2015-06-07 | 2019-01-22 | Apple Inc. | Context-based endpoint detection |
US11025565B2 (en) | 2015-06-07 | 2021-06-01 | Apple Inc. | Personalized prediction of responses for instant messaging |
US10304440B1 (en) * | 2015-07-10 | 2019-05-28 | Amazon Technologies, Inc. | Keyword spotting using multi-task configuration |
US10671428B2 (en) | 2015-09-08 | 2020-06-02 | Apple Inc. | Distributed personal assistant |
US10747498B2 (en) | 2015-09-08 | 2020-08-18 | Apple Inc. | Zero latency digital assistant |
US9787819B2 (en) | 2015-09-18 | 2017-10-10 | Microsoft Technology Licensing, Llc | Transcription of spoken communications |
US9697820B2 (en) | 2015-09-24 | 2017-07-04 | Apple Inc. | Unit-selection text-to-speech synthesis using concatenation-sensitive neural networks |
US11010550B2 (en) | 2015-09-29 | 2021-05-18 | Apple Inc. | Unified language modeling framework for word prediction, auto-completion and auto-correction |
US10366158B2 (en) | 2015-09-29 | 2019-07-30 | Apple Inc. | Efficient word encoding for recurrent neural network language models |
US11587559B2 (en) | 2015-09-30 | 2023-02-21 | Apple Inc. | Intelligent device identification |
US10691473B2 (en) | 2015-11-06 | 2020-06-23 | Apple Inc. | Intelligent automated assistant in a messaging environment |
US10049668B2 (en) | 2015-12-02 | 2018-08-14 | Apple Inc. | Applying neural network language models to weighted finite state transducers for automatic speech recognition |
US10223066B2 (en) | 2015-12-23 | 2019-03-05 | Apple Inc. | Proactive assistance based on dialog communication between devices |
US10446143B2 (en) | 2016-03-14 | 2019-10-15 | Apple Inc. | Identification of voice inputs providing credentials |
US9934775B2 (en) | 2016-05-26 | 2018-04-03 | Apple Inc. | Unit-selection text-to-speech synthesis based on predicted concatenation parameters |
US9972304B2 (en) | 2016-06-03 | 2018-05-15 | Apple Inc. | Privacy preserving distributed evaluation framework for embedded personalized systems |
US10249300B2 (en) | 2016-06-06 | 2019-04-02 | Apple Inc. | Intelligent list reading |
US10049663B2 (en) | 2016-06-08 | 2018-08-14 | Apple, Inc. | Intelligent automated assistant for media exploration |
DK179588B1 (en) | 2016-06-09 | 2019-02-22 | Apple Inc. | Intelligent automated assistant in a home environment |
US10192552B2 (en) | 2016-06-10 | 2019-01-29 | Apple Inc. | Digital assistant providing whispered speech |
US10586535B2 (en) | 2016-06-10 | 2020-03-10 | Apple Inc. | Intelligent digital assistant in a multi-tasking environment |
US10490187B2 (en) | 2016-06-10 | 2019-11-26 | Apple Inc. | Digital assistant providing automated status report |
US10509862B2 (en) | 2016-06-10 | 2019-12-17 | Apple Inc. | Dynamic phrase expansion of language input |
US10067938B2 (en) | 2016-06-10 | 2018-09-04 | Apple Inc. | Multilingual word prediction |
DK179049B1 (en) | 2016-06-11 | 2017-09-18 | Apple Inc | Data driven natural language event detection and classification |
DK179343B1 (en) | 2016-06-11 | 2018-05-14 | Apple Inc | Intelligent task discovery |
DK179415B1 (en) | 2016-06-11 | 2018-06-14 | Apple Inc | Intelligent device arbitration and control |
DK201670540A1 (en) | 2016-06-11 | 2018-01-08 | Apple Inc | Application integration with a digital assistant |
KR101868404B1 (en) * | 2016-06-14 | 2018-07-19 | 가천대학교 산학협력단 | Emergency Dispatch System and Method based on Korean Speech Recognition Technology |
JP6729232B2 (en) * | 2016-09-20 | 2020-07-22 | 富士通株式会社 | Message distribution program, message distribution device, and message distribution method |
US10593346B2 (en) | 2016-12-22 | 2020-03-17 | Apple Inc. | Rank-reduced token representation for automatic speech recognition |
DK179745B1 (en) | 2017-05-12 | 2019-05-01 | Apple Inc. | SYNCHRONIZATION AND TASK DELEGATION OF A DIGITAL ASSISTANT |
DK201770431A1 (en) | 2017-05-15 | 2018-12-20 | Apple Inc. | Optimizing dialogue policy decisions for digital assistants using implicit feedback |
Family Cites Families (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5434777A (en) * | 1992-05-27 | 1995-07-18 | Apple Computer, Inc. | Method and apparatus for processing natural language |
US5384892A (en) * | 1992-12-31 | 1995-01-24 | Apple Computer, Inc. | Dynamic language model for speech recognition |
JP3634863B2 (en) * | 1992-12-31 | 2005-03-30 | アプル・コンピュータ・インコーポレーテッド | Speech recognition system |
JPH06318223A (en) * | 1993-05-07 | 1994-11-15 | Nippon Telegr & Teleph Corp <Ntt> | Clustering device |
JPH0934863A (en) * | 1995-07-14 | 1997-02-07 | Hitachi Ltd | Information integral processing method by neural network |
US5675707A (en) * | 1995-09-15 | 1997-10-07 | At&T | Automated call router system and method |
US5794193A (en) * | 1995-09-15 | 1998-08-11 | Lucent Technologies Inc. | Automated phrase generation |
-
1997
- 1997-07-11 US US08/893,888 patent/US5860063A/en not_active Expired - Lifetime
-
1998
- 1998-04-20 CA CA002235364A patent/CA2235364C/en not_active Expired - Lifetime
- 1998-07-03 ES ES98112418T patent/ES2210623T3/en not_active Expired - Lifetime
- 1998-07-03 EP EP98112418A patent/EP0890942B1/en not_active Expired - Lifetime
- 1998-07-03 DE DE69818161T patent/DE69818161T2/en not_active Expired - Fee Related
- 1998-07-03 JP JP18940398A patent/JP4053141B2/en not_active Expired - Lifetime
Also Published As
Publication number | Publication date |
---|---|
JPH11143491A (en) | 1999-05-28 |
DE69818161T2 (en) | 2004-04-08 |
US5860063A (en) | 1999-01-12 |
EP0890942A2 (en) | 1999-01-13 |
EP0890942A3 (en) | 1999-04-21 |
DE69818161D1 (en) | 2003-10-23 |
CA2235364A1 (en) | 1999-01-11 |
ES2210623T3 (en) | 2004-07-01 |
EP0890942B1 (en) | 2003-09-17 |
JP4053141B2 (en) | 2008-02-27 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CA2235364C (en) | Automated meaningful phrase clustering | |
US5794193A (en) | Automated phrase generation | |
EP0797823B1 (en) | Automated call router system and method | |
US8144838B2 (en) | Automated task classification system | |
US7720203B2 (en) | System and method for processing speech | |
US6839671B2 (en) | Learning of dialogue states and language model of spoken information system | |
US5625748A (en) | Topic discriminator using posterior probability or confidence scores | |
US8024188B2 (en) | Method and system of optimal selection strategy for statistical classifications | |
US6223155B1 (en) | Method of independently creating and using a garbage model for improved rejection in a limited-training speaker-dependent speech recognition system | |
US6438520B1 (en) | Apparatus, method and system for cross-speaker speech recognition for telecommunication applications | |
US20030195739A1 (en) | Grammar update system and method | |
US20090055164A1 (en) | Method and System of Optimal Selection Strategy for Statistical Classifications in Dialog Systems | |
Siu et al. | Evaluation of word confidence for speech recognition systems | |
US7346507B1 (en) | Method and apparatus for training an automated speech recognition-based system | |
Wright et al. | Automatic acquisition of salient grammar fragments for call-type classification. | |
US20030225579A1 (en) | Error-tolerant language understanding system and method | |
Golden et al. | Automatic topic identification for two-level call routing | |
US20090265162A1 (en) | Method for Retrieving Items Represented by Particles from an Information Database | |
Huang et al. | Extracting caller information from voicemail | |
MXPA98005525A (en) | Automated grouping of phrases with signific | |
MXPA97003425A (en) | System and automated method of calling telephone | |
Sheikh et al. | A fuzzy approach to mute sensitive information in noisy audio conversations | |
Kuroiwa et al. | Automatic prank call rejection system for home country direct based on speech recognition technology |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
EEER | Examination request | ||
MKEX | Expiry |
Effective date: 20180420 |