WO2002007003A2 - Natural language system - Google Patents

Natural language system Download PDF

Info

Publication number
WO2002007003A2
WO2002007003A2 PCT/GB2001/003194 GB0103194W WO0207003A2 WO 2002007003 A2 WO2002007003 A2 WO 2002007003A2 GB 0103194 W GB0103194 W GB 0103194W WO 0207003 A2 WO0207003 A2 WO 0207003A2
Authority
WO
WIPO (PCT)
Prior art keywords
module
phrase
word
natural language
language system
Prior art date
Application number
PCT/GB2001/003194
Other languages
French (fr)
Other versions
WO2002007003A9 (en
WO2002007003A3 (en
Inventor
John Gerald Taylor
Neill Richard Taylor
Original Assignee
King's College London
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by King's College London filed Critical King's College London
Priority to AU2001272640A priority Critical patent/AU2001272640A1/en
Publication of WO2002007003A2 publication Critical patent/WO2002007003A2/en
Publication of WO2002007003A3 publication Critical patent/WO2002007003A3/en
Publication of WO2002007003A9 publication Critical patent/WO2002007003A9/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F40/00—Handling natural language data
    • G06F40/20—Natural language analysis
    • G06F40/253—Grammatical analysis; Style critique
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/047—Probabilistic or stochastic networks

Definitions

  • the present invention relates to a natural language system and in particular to a natural language system that uses artificial neural networks.
  • natural language processing also known as computational linguistics
  • computer systems are programmed to respond to natural human language.
  • the aim of the system is to interpret input data in the form of a natural language so that further processing may be performed in accordance with the meaning of the input data.
  • One use of such a program is to make it possible to instruct a computer in ordinary language to cause it to perform the required operations.
  • a natural language system normally includes a parser responsible for transforming input data representing words, phrases and sentences into data of a form representing the meaning of word groupings that can be further processed.
  • a stored lexicon or dictionary is used to store the words the system can process, information on the word types (nouns, verbs, etc) and definitions of any grammatical rules.
  • the parser searches for specific words or sets of words. For each word or set, the rule base contains one or more associated sentence pattern that is used to determine command word groupings. For example, in the command "Please insert the block into machine A", the parser may be programmed to look for the word “insert”, having an associated rule "insert
  • Target into Destination Having identified "insert", the parser can inform the system that its target is the block and its destination is Machine A. Once verbs, nouns and adjectives can be identified, they can be translated into commands that the computer system can interpret and apply. However, such systems are limited to the words in the dictionary and to the rules held in the rule base.
  • a natural language system comprising a phrase module connected to a plurality of word modules, each word module being associated with a class of words and holding representations of words of the respective class, the word modules being configured to prime a representation upon presentation of a stimulus associated with the respective representation, wherein the phrase module includes a neural network configured to determine an excitation sequence in dependence on the primed word representations and to trigger the primed word representations in accordance with the excitation sequence.
  • the triggering of word representations may, for example, allow a sentence to be generated or an input to be recognised.
  • the neural network may comprise an ACTION neural network.
  • the ACTION network may include an initiator node configured to accept a stimulus and trigger the start of a sequence.
  • the ACTION network may include a transition node configured to track the triggering and trigger in order the primed word representations according to the excitation sequence.
  • the ACTION network may include a memory node configured to store information relating to the sequence.
  • the or each node may comprise an ACTION network or a portion of an ACTION network.
  • Each word module may include an ACTION neural network configured to accept the presented stimulus and to prime a representation associated with the stimulus by holding the stimulus in a portion of the network corresponding to that representation, wherein the portions of the network corresponding to the associated representation are linked to the phrase module to pass an indication of the stimulus to the phrase module.
  • Each portion of the network may have an associated stimulus threshold, the stimulus being held in the portion until it exceeds the threshold, thereby causing the ACTION network to fire for the respective word.
  • the stimulus threshold may be set greater than any presented stimulus, wherein the phase module is arranged to apply a further stimulus to the word module's ACTION network, thereby triggering the primed word representation, in accordance with the excitation sequence.
  • Each word module may be connected to an element representation module, the element representation module having a portion associated with each of a plurality of elements and being arranged to accept an input and to produce a stimulus from an associated portion is its element is present in the input, the element representation module being arranged to present the stimulus to the word module via the connection.
  • the system may further comprise a control module connected to the phrase module and being configured to apply a stimulus to the phrase analyser to trigger the primed word representations.
  • a class of a word module may be selected from the set of nouns, pronouns, adverbs, verbs, prepositions, adjectives, derivational morphemes (prefixes and suffixes) and inflectional morphemes.
  • a phrase module may be arranged to process one of the set of sentences, noun phrases, verb phrases, prepositional phrases, adjectival phrases, combination of morphemes, morpheme combination with suffixes and/or prefixes, conjugation and tense formation.
  • the system may further comprise an action generator connected to a verb word module and a noun word module wherein, upon triggering of a verb from the verb word module and a noun from the noun word module, the action generator is configured to trigger an appropriate action.
  • the system may further comprise a memory arranged to store a presented stimulus with the corresponding primed word representation.
  • Each phrase module may comprise a first level phrase module connected to a second level phrase module and a specifier module, the second level phrase module also being connected to a word module, the specifier and word modules comprising memories for words associated with the phrase class, wherein each connection is excitatory in both directions except for the connection from the second level phrase module to the first level phrase module which is inhibitory.
  • the phrase modules may include a noun phrase analyser, an infinitival phrase analyser, a verb phrase analyser and a complementiser phrase analyser.
  • the system may comprise a connection of a first connection class originating from the verb phrase analyser to the infinitival phrase analyser, from the complementiser phrase analyser to the noun phrase analyser, from the verb phrase analyser to the noun phrase analyser, and/or from the complementiser phrase analyser to the verb phrase analyser.
  • the first connection class comprises an excitatory connection from the originating phrase analyser's second level phrase module to the first ievel phrase module of the other phrase analyser.
  • the first connection class comprises an inhibitory connection to the originating phrase analyser's second level phrase module from the first level phrase module of the other phrase analyser
  • the memory may comprise a plurality of neurons, the system further comprising trace means for tracing synapses between neurons.
  • the system may further comprise learning means for training the neural network(s) by causal Hebbian and/or reinforcement training.
  • a natural language processing method comprising maintaining a phrase module connected to a plurality of word modules, each word module being associated with a class of words and holding representations of words of the respective class, priming a representation within a word module upon presentation of a stimulus associated with the respective representation, wherein the phrase module includes a neural network performing the steps of determining an excitation sequence in dependence on the primed word representations and triggering the primed word representations in accordance with the excitation sequence.
  • Each word module may be connected to an element representation module, the element representation module having a portion associated with each of a plurality of elements, wherein each element representation module performs the steps of accepting an input, producing a stimulus from an associated portion if its element is present in the input and presenting the stimulus to the word module.
  • a natural language system comprising a first memory encoding a phrase module and a further memory encoding a word module, the phrase module being linked to the word module, wherein the word module is associated with a class of words and holds representations of words of the respective class, wherein upon presentation of a stimulus associated with the respective representation the word modules is configured to prime a representation, wherein the phrase module includes a neural network configured to determine an excitation sequence in dependence on the primed word representations and to trigger the primed word representations in accordance with the excitation sequence.
  • semantics are defined initially by virtual actions to guide the construction of manipulatable symbol representations for objects and actions, in particular to obtain a model of syntactic processing.
  • Such representations can be later used to develop a semantic network in the form of a look-up memory.
  • Figure 1 is a schematic diagram of a neural network architecture in accordance with the present invention.
  • Figure 2 is a schematic diagram of an ACTION neural network
  • Figure 3 is the schematic diagram of Figure 1 illustrating selected features in more detail
  • Figure 4 is a schematic diagram of a neural network architecture in accordance with the present invention.
  • Figure 5 is a schematic diagram illustrating selected aspects of the neural network architecture in more detail;
  • Figure 6a-i are graphs plotting against time the excitation of parts of a neural architecture in accordance with the present invention;
  • Figure 7 is a schematic diagram of a neural network architecture in accordance with the present invention.
  • Figures 8a-8h are graphs plotting against time the excitation of parts of the neural network or Figure 7.
  • syntax and semantics of incoming (segmented, pre-processed) sound patterns are processed in parallel.
  • the system is programmed such that semantics guides the development of syntax, and that semantics of single words arise from actions possible on an object (for nouns) or actions that co ⁇ espond to the word (for verbs).
  • Semantics of object nouns are encoded in terms of sets of actions and associated sensory input patterns. Actions may also be associated with verbs.
  • Nouns ball, floor, door, table, target, box.
  • Figure 1 is a schematic diagram of a neural network architecture according to the present invention.
  • Objects are represented by Self Organising Feature Maps (SOFM) in an object representation module 10.
  • SOFM Self Organising Feature Maps
  • object representation module 10 Such a representation allows a simple localised interpretation. Nodes of the SOFMs are modelled as leaky integrators so as to form a buffer that acts as a working memory site. Object coding is by dedicated nodes for simplicity.
  • Actions are coded in an ACTION network 20 to allow for sequence learning.
  • the architecture of an ACTION network is described in detail below, with reference to Figure 2.
  • each action of the set specified above is coded as a single cortical node in an ACTION network.
  • Action sequences associated with a given object representation are linked to the object's dedicated node by link 30 using a Hebbian learning process, in which the relevant action nodes are activated (according to the actions being taken) by the particular active object representation.
  • the associated action representations in action module 20 will be activated via link 30.
  • intention module 70 the actions are not actually taken. Instead, the action is primed, as allowed by the connections in Figure 1.
  • semantics are initially encoded as "virtual actions”.
  • the verb phrase module 60 is activated by a signal from the willed intention module 70 to generate a verb, the signal is passed to the verb representation module 40 to select an appropriate verb (guided by the relevant action representation from the action representation module 20).
  • the verb phrase module 60 then activates a noun in the. noun representation module 50 (guided by the relevant object representation from the object representation module 10 activated by the input I).
  • Other inputs to the verb phrase analyser may include inputs from other phrase analysers or direct speech leading to speech recognition.
  • the noun representation module 50 is also an ACTION network. Connections between the relevant action representation and the verb representation modules, and between the object representation module 20 and the associated noun representation module 50, are achieved by Hebbian learning of the bi-directional connections between the object representation in module 10 and the noun representation module 50 and between the action representation module 20 and the verb representation module 40. Alternatively, these connections can be hard- wired.
  • FIG. 2 is a schematic diagram of an ACTION neural network architecture.
  • the architecture of an ACTION neural network is based on current understanding of the features of the frontal lobes of the Human brain.
  • the architecture is described in terms of the names of the parts of the Human brain for simplicity but can be implemented completely in a computer program or neural hardware, such as the pRAM (probabilistic RAM, details of which can be found in IEEE Transactions on Computers, Vol. 41, No. 12, December 1992 pl552-1561) chip.
  • pRAM program or neural hardware
  • Each of the boxes shown in Figure 2 represent a grouping of interconnected neurons forming a neural network capable of accepting an input and providing an output if the inputs cause the network to fire.
  • the main difference between an ACTION network and other types of neural networks is that there are feedback loops between certain groupings that model, in a simplistic level, current understanding of the operation of areas of the brain.
  • ACTION network It is possible in an ACTION network to achieve temporal sequence storage and generation (TSSG).
  • TSSG temporal sequence storage and generation
  • ACTION neural network architectures have been shown to achieve chunking (the composition of basic elements such, as words, into higher representations, such as sentences) of temporal sequences.
  • An ACTION network include, or may act as, initiator, transition or memory modules that have continued activity during delay periods.
  • Initiator modules that operate to initiate a particular sequence. For this function they hold a memory of the entire sequence. Transition modules are only active during the delay between two elements of a sequence. For this function they hold a memory .of the transition between the two elements. Memory modules have activity that continues throughout the whole of the sequence response, although this is greatest in between elements. It also persists throughout the period in which the element is active, but at a reduced level. This activity may be specific to a particular sequence or to sequences containing classes of words. Memory modules allow the continuation of the memory of the particular sequence throughout the sequence itself.
  • the initiator, transition and memory modules may be present in the cortex, thalamus, STN and STR as well as in a reflected form in the globus pallidi.
  • Neurons are modelled as mean firing rate leaky integrators. Hebbian reinforcement is used for learning. Sequence production is only allowed to continue if the action selected is correct.
  • excitatory connections are shown as open arrowheads whilst inhibitory connections are shown as closed arrows.
  • Data passes via an input (I) into the cortex 100 and then on to the striatum (STR) 110. From the striatum 110, data is passed to the globus pallidus (internal) (GPi) 120, then to the thalamus (TH) 130 and then back to the cortex 100. Thus, a feedback loop of activity occurs around this loop. There may also be similar feedback more rapidly around the cortex 100 to thalamus 130 loop (shown as two-way link 140). There is also an indirect loop, involving the globus pallidus external (GPe) 150 as well as the sub-thalamic nucleus (STN) 160. In certain cases, the action network may include an output node 170. This node is fed by activity from the cortex
  • the system can also produce an output from the cortex 100 or thalamus 130.
  • the following description provides a mathematical basis for the operation of an
  • Neurones are modelled as leaky integrators, with a mean firing-rate response given by the positive part of a sigmoidal function.
  • the network includes weighting factors to be applied to its connections including: w 1 - connections from the input layer to the cortex; w 2 - lateral connections in the cortex; w 3 - connections from the cortex to the STR; w 4 - lateral connections in the STR; w 5 - connections from TH to the cortex; w 6 - connections from the cortex to STN; w7 - connections from STN to GPi; w 8 - connections from STR to GPi; w 9 - connections from STN to GPe; w 10 - connections from STR to GPe; w - connections from the cortex to TH; w n - connections from the GPi to TH; w 13 - connections from GPe to STN;
  • the network's potentials are: s for the cortex; a for input; m for TH; n for STN; u for GPe; r for STR; and ⁇ for GPi.
  • the time constant of the neurones is indicated by ⁇ and is a value between 0 and 1.
  • n ⁇ t+l) ⁇ n(t) + w 6 */(s(f), ⁇ 4 ) - w 13 */( M (t), ⁇ 5 ) + T (2)
  • T is a tonic input;
  • r(t+l) ⁇ r(t) + w 3 */(s(t), ⁇ 4 ) + w 4 *Mt),Q 6 ) (3)
  • d ⁇ t+l) ⁇ d ⁇ t) + w 7 n ⁇ t), ⁇ 7 ) - w 8 * ( t), ⁇ 6 ) (4)
  • ⁇ is the threshold.
  • Other functions, sigmoidal or otherwise, may be used. Activity is held during the delay between sequence elements by the combination of the loop from cortex 100 to TH 130 to cortex 100 via the direct and indirect pathways as well as the reciprocal connections between cortex and TH 130.
  • the handing on of activity from one type of neuron to another, such as from the memory neuron to transition neuron, is mainly an effect of lateral connections in the cortex 100.
  • the memory neuron exceeds its learnt threshold for use of the lateral weights it excites the transition neuron.
  • the activity of the memory neuron is kept sub-threshold by lateral inhibition in the STR 110 from the initiator.
  • connections I to cortex 100 and cortex 100 to STR 110 are required always to be excitatory. Lateral STR connections are always inhibitory. The remaining connections (lateral cortex, cortex to OUT and TH to cortex) are allowed to switch during the learning process from excitatory to inhibitory, and vice versa.
  • the willed intentions module (WI) 70 is only activated when it is desired for an action to be being taken. Learning can then take place between the action representations module 20 and the associated object representation module 10.
  • Activity in the action representations during learning of the words associated to objects leads to learning not only of the connections between object representations and associated noun representations (as shown by the bi-directional line joining the relevant nodes in each of these modules in Figure 1) but also of the route through the verb phrase module 60 joining the noun representation module 50 and the set of relevant actions in the action representation module 20.
  • a direct route is also possible, as is shown by link 80 in Figure 1.
  • Figure 3 is a schematic diagram of the neural architecture of Figure 1 in which selected elements and connections are shown in more detail. In particular, the elements of the architecture of Figure 1 necessary for dealing with syntax for 2 word sentences are described in detail.
  • ACTION networks In order to produce such sequences, three new types of ACTION networks are introduced, initiator networks (IN), memory networks (MEM) and transition networks (TRANS).
  • IN initiator networks
  • MEM memory networks
  • TRANS transition networks
  • the verb and noun modules 40, 50 each include ACTION networks (41-42; 51- 52) of the type discussed with reference to Figure 2 having output nodes.
  • the verb phrase module 60 includes initiator ACTION networks 61, 62, memory ACTION networks 63, 64 and transition ACTION network 65, 66. These ACTION networks are of the type discussed with reference to Figure 2 without output nodes.
  • the ACTION networks (41, 42,51,52,61-66) are illustrated as a pair of horizontally linked nodes, the upper node being the cortex of the network whilst the lower node is the striatum (STR).
  • excitatory connections are shown as open arrowheads and inhibitory connections are shown as closed arrowheads.
  • FIG. 3 An example of the present invention will now be described with reference to Figure 3, in which the neural architecture is programmed to generate two verb phrases: lift box and shut door.
  • an object is presented throughout the task as an input to the object representation module 10. This has the effect of exciting the corresponding object representation.
  • a willed input signal is also applied by the willed input module 70 as a general input to the 2 ACTION networks 21, 22 of the 2 actions, "lift” and “shut", and at the IN networks 61, 62 for both verb phrases.
  • the presentation of the object leads to low-levels of activity in the associated noun representation and action representation. This in turn leads, in both the noun and action representation modules 20, 50, to inhibitory activity on the non-associated representations.
  • the low-level of activity that results in the action representation generates further low activation on the associated verb representation and on the initiator network for the desired verb phrase. This again leads to inhibition of non-associated verb representations and other verb phrase initiators.
  • the presentation of the willed intention signal leads to the excitation of the primed IN cortex neuron and MEM cortex neuron. Inhibition via an STR connection 67 from the IN to MEM network prevents initial high activity in the MEM network. Due to the low activation of the MEM cortical neuron the TRANS cortical neuron also has low activation during this initial period. As the IN network activity increases it excites the associated verb network in verb representation module 40 leading to the respective verb's ACTION network's OUT neuron generating an output when its threshold is exceeded. Feedback inhibits the verb network and corresponding IN network, reducing activity to a low level.
  • Figure 4 is a schematic diagram of the neural architecture of Figure 1, including further features, in accordance with a preferred aspect of the present invention.
  • the neural network architecture of Figure 1 is extended to include an adjectival phrase module 310 and an adjective representation module 300.
  • the adjectival phrase module 310 is configured to allow a number of adjectives to be produced prior to a noun.
  • Figure 5 is a schematic diagram illustrating selected features of the neural network architecture in more detail.
  • the architecture is extended to include an adjective module 300 an adjectival phrase module 310 and a feature representation module 330.
  • a feature is an attribute such as "red” or “heavy” and has a neural representation in the features module 330.
  • the adjective and noun modules 300, 50 each include ACTION networks (301-303; 51-53) of the type discussed with reference to Figure 2 having output nodes.
  • the adjectival phrase module 310 includes an initiator ACTION network 311 and a transition ACTION network 312 of the type discussed with reference to Figure 2 without output nodes.
  • the ACTION networks (51-53; 301-303; 311, 312) are illustrated as a pair of horizontally linked nodes, the upper node being the cortex of the network whilst the lower node is the striatum (STR).
  • excitatory connections are shown as open arrowheads and inhibitory connections are shown as closed arrowheads.
  • an input is applied to feature representation module 320 and object representation module 10.
  • Those representations corresponding to the input become excited and pass this excitation to their linked adjective and noun representations via the respective ACTION networks (301-303; 51-53).
  • a number of adjective representations and a noun representation are primed.
  • a signal from the willed intention module 70 to create a sentence is passed to the initiator network 311.
  • the willed intention module 70 causes inhibition to all links from the adjective representation module 300 to the transition network 312.
  • the willed intention module is arranged to monitor the feature representation module 320 and to only remove this inhibition when there are no features excited. .
  • the initiator network 311 passes on the willed intention signal by exciting all the Action networks 301-303 of the adjectival representations, whether they are primed or not. However, only those that are primed by excitation from feature representations 320 can produce sustained activity and excite their OUT neurone beyond the threshold value.
  • the striatal lateral connections 340 between adjective ACTION networks 301-303 also ensures that only one primed adjective ACTION network exceeds the threshold at a time. Hence, only one word can be generated at a time.
  • the transition network 312 is unable to be excited and the initiator network 311 remains active such that another adjective can be generated.
  • the willed intention signal that inhibits all inputs to the transition network 312 is removed if no features in the feature representation module 320 are active. Once the inhibition is removed, excitation to the transition network 312 is applied by the last generated adjective. The transition network 312 excites all noun representations via their respective noun ACTION networks 51-53.
  • Figure 6a-i shows the time-courses of some of the neurones involved in generating an adjectival phrase composed of 3 adjectives.
  • the excitation levels of the 3 active feature representations are plotted against time in Figures 6a to 6c, respectively.
  • Figures 6d to 6f are plots in time of cortical activities of the 3 adjective representations, each linked to one of the feature representations. A progression through time can be seen as the maximum activities are reached.
  • Figure 6g it is shown that the generated noun reaches its maximum activity after all the adjectives.
  • Figure 6h shows sustained activity of the initiator network whilst the adjectives are being generated.
  • Figure 6i shows the transition network becoming active once the inhibitory willed intention signal is removed.
  • Figure 7 is a schematic diagram of a neural network architecture in accordance with the present invention. Solid arrowheads represent inhibitory stimulus being applied in the direction of the arrow whilst open arrowheads represent excitatory stimulus.
  • the neural network architecture includes a noun-phrase analyser 400, a verb- phrase analyser 420, an infinitival-phrase analyser 430 and a complementiser phrase analyser 440. Analysers for adjectives and prepositions are not included but could be added following the same architecture pattern without difficulty.
  • Each analyser is made up of a number of modules that are each in turn ACTION neural networks which have been described in detail with reference to Figure 2.
  • a first level noun-phrase module 401 is connected to a second level noun-phrase module 402.
  • the two modules 401, 402 are. composed of initiators and transitions and the link is excitatory from the first level module 401 to the second level module 402 but inhibitory in reverse.
  • the first level module 401 is also linked to a specifier module 403 via excitatory links in both directions.
  • the second level module 402 is linked to a word module 404 via excitatory links in both directions.
  • the specifier module 403 and the word module 404 are storage regions for words of the appropriate type (nouns in the case of the noun-phrase analyser), the specifier module may store determiner type words (e.g. words such as 'when', 'who' and 'how' in the case of the noun-phrase analyser).
  • the initiator neurone of the first level noun-phrase module 401 is connected via excitatory weights to all words in the specifier module 403, which are in turn connected via excitatory weights back to the transition neurone of the first level noun-phrase module 401.
  • All transition neurons of the first and second level noun-phrase modules act in an inhibitory manner via lateral striatal connections on their associated initiator neurones.
  • the transition neurone of the first level noun-phrase module 401 is connected via excitatory weights to the initiator neurone of the second level noun-phrase module 402 which in turn is connected back to the transition neurone of the first level noun-phrase module 401 via inhibitory weights.
  • the initiator neurone of the second level noun-phrase module 402 is connected via excitatory weights to all words in the word module 404, which are in turn connected via excitatory weights back to the transition neurone of the second level noun-phrase module 402.
  • This connection architecture is mirrored for each analyser 400-440.
  • each first level module 401, 411, 421, 431, 441 is linked via inhibitory weights to selected ones of the other analyser's second level module 402, 412, 432, 442 transition neurones. There is also a return link via excitatory weights.
  • the inter-analyser connections can be seen in Figure 7, however in summary they are: noun-phrase to infinitival-phrase; infinitival-phrase to verb-phrase; verb-phrase to noun-phrase; complementiser-phrase to noun-phrase; . complementiser-phrase to infinitival-phrase; and, noun-phrase to complementiser-phrase.
  • the architecture only allows the initiator of a first or second level module to excite a word module or specifier module. Transition either projects to a second level module or to a first level module of another analyser. In this manner, words can only be generated when there is priming.
  • the word modules 403, 423, 433, 443 have output neurones connected to the cortical nodes of their ACTION neural networks. In this manner, when output nodes exceed a threshold the word is produced, at which point inhibition is introduced for, for example, 80 time steps for word representations and also associated representations (object representations for nouns, action representations for verbs etc.).
  • Training of the neural network is performed by use of both causal hebbian and reinforcement learning techniques, in which traces are included on synapses to give longer and smoother responses to help bridge temporal gaps between inputs. For example, a short movement occurs with auxiliary verbs when a statement changes to a question:
  • the neural network is taught the rule that "can", an infinitival word, is excited in its new position by the complementiser phrase analyser and that structures that excite it in its infinitival statement form are still active in the question form but do not lead to a second activation. This is caused either by lasting inhibition or lack of priming.
  • the excitation flow in the neural network to generate the original sentence is: 401 (initiator) - ⁇ 403 (no word generated) ⁇ ⁇ 401 (transition) - ⁇ 402 (initiator)
  • ⁇ ⁇ 404 (generates T) - ⁇ 402 (transition) ->431 (initiator) - ⁇ 433 (no word generated) ⁇ 431 (transition) ⁇ 432 (initiator) - 434 (generates 'can') - 432 (transition) ->421 (initiator) - 423 (no word generated) - ⁇ 421 (transition) ⁇ >422 (initiator) " ⁇ 424 (generates 'have') ⁇ ⁇ 422 (transition) HM01 (initiator) - 403 (generates 'a') ⁇ 401 (transition) -»402 (initiator) -»404 (generates 'Z') -»402 (transition).
  • Figures 8a-8h are graphs plotting against time the excitation of parts of the neural network or Figure 7.
  • Figures 8a to 8d show the activation pattern of the cortical neurones involved in the generation of the original sentence. These show the correct temporal order of activation of the words T (Fig 8a), 'can' (Fig 8b), and 'have' (Fig. 8d) as well as a single strong activation of the second level infinitival-phrase module initiator neurone (Fig. 8c), the second smaller activation being due to excitation of the first level infinitival- phrase module by the second level noun-phrase module before global inhibition takes effect.
  • Figures 8e to 8h show the same neurones as Figures 8a to 8d but activated in the moved sequence, 'can' (Fig. 8f), T (Fig. 8g), 'have' (Fig. 8h).
  • Fig. 8f the second level infinitival-phrase module initiator neurone
  • Fig. 8e the second level infinitival-phrase module initiator neurone
  • the neural network architectures operate to generate verb and noun based sentences.
  • the architecture could be trained or appropriately hard-wired to act as phrase analysers for sentence syntax and those that control morpheme combination.
  • the architecture may include sentence analysers, noun phrase analysers, verb phrase analyser, prepositional phrase analysers and adjectival phrase analysers.
  • Morphemic combination analysers may include derivational analysers (prefix and suffix connectable together to add a prefix and any number of suffixes to a morpheme stem) and inflectional analysers involved in verb formation, including conjugational analysers to provide the present tense forms, regular past tense analysers and irregular past tense analysers.
  • a semantic memory may be incrementally constructed for the nouns and verbs, etc., and for new words of a more abstract form but related to the earlier words.
  • This semantic memory may be an associative memory/dictionary look-up which can be constructed using the priming from the representations, especially using the feature representations associated with words, to guide clustering in a general feature space.
  • a more specific form of such a memory can be an associative neural network memory, learnt by a competitive and/or Hebbian form of learning, such as a Hopfield network or a self-organising feature map.
  • a hardware representation is possible in terms of RAM-based neural systems, such as the pRAM.
  • This semantic memory may be extended to new. word learning, and used to replace the meaning given to words by means of the more computationally intensive ACTION network systems described herein, so as to improve the speed of language processing in both recognition and production.
  • the control of the willed intention module and other control aspects of this system could be performed by an attentional control system such as that described in co-pending patent application (agents reference P15265GB).

Abstract

A natural language system and method is described. A phrase module is connected to a plurality of word modules, each word module being associated with a class of words and holding representations of words of the respective class. Word modules are configured to prime a representation upon presentation of a stimulus associated with the respective representation. The phrase module includes a neural network configured to determine an excitation sequence in dependence on the primed word representations and to trigger the primed word representations in accordance with the excitation sequence.

Description

Natural Language System Field of the invention
The present invention relates to a natural language system and in particular to a natural language system that uses artificial neural networks.
Background to the invention
In natural language processing, also known as computational linguistics, computer systems are programmed to respond to natural human language. The aim of the system is to interpret input data in the form of a natural language so that further processing may be performed in accordance with the meaning of the input data. One use of such a program is to make it possible to instruct a computer in ordinary language to cause it to perform the required operations.
A natural language system normally includes a parser responsible for transforming input data representing words, phrases and sentences into data of a form representing the meaning of word groupings that can be further processed. A stored lexicon or dictionary is used to store the words the system can process, information on the word types (nouns, verbs, etc) and definitions of any grammatical rules.
Most natural language systems are rule based. In such systems, the parser searches for specific words or sets of words. For each word or set, the rule base contains one or more associated sentence pattern that is used to determine command word groupings. For example, in the command "Please insert the block into machine A", the parser may be programmed to look for the word "insert", having an associated rule "insert
Target into Destination". Having identified "insert", the parser can inform the system that its target is the block and its destination is Machine A. Once verbs, nouns and adjectives can be identified, they can be translated into commands that the computer system can interpret and apply. However, such systems are limited to the words in the dictionary and to the rules held in the rule base.
An additional problem is that because computer systems do not process in terms of natural language, natural language commands must be translated into machine language that is comprehensible by the computer system by the natural language processor so that it can then be acted upon by the computer system. Any subsequent output must be translated back into natural language in order for the user to understand it. This process is computationally intensive in both directions and, due to the two-way translation, introduces many opportunities for errors. It is highly desirable for natural language systems to be used in user interfaces for human-computer-interaction. However, the limited adaptability of current natural language systems in combination with the level of computing resources necessary to operate such systems have meant that, as yet, natural language processing systems have not been successfully applied in conventional computer systems to any great degree.
Statement of invention
According to one aspect of the present invention, there is provided a natural language system comprising a phrase module connected to a plurality of word modules, each word module being associated with a class of words and holding representations of words of the respective class, the word modules being configured to prime a representation upon presentation of a stimulus associated with the respective representation, wherein the phrase module includes a neural network configured to determine an excitation sequence in dependence on the primed word representations and to trigger the primed word representations in accordance with the excitation sequence. The triggering of word representations may, for example, allow a sentence to be generated or an input to be recognised.
The neural network may comprise an ACTION neural network.
The ACTION network may include an initiator node configured to accept a stimulus and trigger the start of a sequence. The ACTION network may include a transition node configured to track the triggering and trigger in order the primed word representations according to the excitation sequence. The ACTION network may include a memory node configured to store information relating to the sequence. The or each node may comprise an ACTION network or a portion of an ACTION network. Each word module may include an ACTION neural network configured to accept the presented stimulus and to prime a representation associated with the stimulus by holding the stimulus in a portion of the network corresponding to that representation, wherein the portions of the network corresponding to the associated representation are linked to the phrase module to pass an indication of the stimulus to the phrase module. Each portion of the network may have an associated stimulus threshold, the stimulus being held in the portion until it exceeds the threshold, thereby causing the ACTION network to fire for the respective word.
The stimulus threshold may be set greater than any presented stimulus, wherein the phase module is arranged to apply a further stimulus to the word module's ACTION network, thereby triggering the primed word representation, in accordance with the excitation sequence.
Each word module may be connected to an element representation module, the element representation module having a portion associated with each of a plurality of elements and being arranged to accept an input and to produce a stimulus from an associated portion is its element is present in the input, the element representation module being arranged to present the stimulus to the word module via the connection.
The system may further comprise a control module connected to the phrase module and being configured to apply a stimulus to the phrase analyser to trigger the primed word representations.
A class of a word module may be selected from the set of nouns, pronouns, adverbs, verbs, prepositions, adjectives, derivational morphemes (prefixes and suffixes) and inflectional morphemes.
A phrase module may be arranged to process one of the set of sentences, noun phrases, verb phrases, prepositional phrases, adjectival phrases, combination of morphemes, morpheme combination with suffixes and/or prefixes, conjugation and tense formation.
The system may further comprise an action generator connected to a verb word module and a noun word module wherein, upon triggering of a verb from the verb word module and a noun from the noun word module, the action generator is configured to trigger an appropriate action.
The system may further comprise a memory arranged to store a presented stimulus with the corresponding primed word representation.
Each phrase module may comprise a first level phrase module connected to a second level phrase module and a specifier module, the second level phrase module also being connected to a word module, the specifier and word modules comprising memories for words associated with the phrase class, wherein each connection is excitatory in both directions except for the connection from the second level phrase module to the first level phrase module which is inhibitory. The phrase modules may include a noun phrase analyser, an infinitival phrase analyser, a verb phrase analyser and a complementiser phrase analyser. The system may comprise a connection of a first connection class originating from the verb phrase analyser to the infinitival phrase analyser, from the complementiser phrase analyser to the noun phrase analyser, from the verb phrase analyser to the noun phrase analyser, and/or from the complementiser phrase analyser to the verb phrase analyser.
Preferably, the first connection class comprises an excitatory connection from the originating phrase analyser's second level phrase module to the first ievel phrase module of the other phrase analyser.
Preferably, the first connection class comprises an inhibitory connection to the originating phrase analyser's second level phrase module from the first level phrase module of the other phrase analyser
The memory may comprise a plurality of neurons, the system further comprising trace means for tracing synapses between neurons.
The system may further comprise learning means for training the neural network(s) by causal Hebbian and/or reinforcement training.
According to another aspect of the present invention, there is provided a natural language processing method comprising maintaining a phrase module connected to a plurality of word modules, each word module being associated with a class of words and holding representations of words of the respective class, priming a representation within a word module upon presentation of a stimulus associated with the respective representation, wherein the phrase module includes a neural network performing the steps of determining an excitation sequence in dependence on the primed word representations and triggering the primed word representations in accordance with the excitation sequence.
Each word module may be connected to an element representation module, the element representation module having a portion associated with each of a plurality of elements, wherein each element representation module performs the steps of accepting an input, producing a stimulus from an associated portion if its element is present in the input and presenting the stimulus to the word module.
According to a further aspect of the present invention, there is provided a natural language system comprising a first memory encoding a phrase module and a further memory encoding a word module, the phrase module being linked to the word module, wherein the word module is associated with a class of words and holds representations of words of the respective class, wherein upon presentation of a stimulus associated with the respective representation the word modules is configured to prime a representation, wherein the phrase module includes a neural network configured to determine an excitation sequence in dependence on the primed word representations and to trigger the primed word representations in accordance with the excitation sequence.
In the present invention, semantics are defined initially by virtual actions to guide the construction of manipulatable symbol representations for objects and actions, in particular to obtain a model of syntactic processing. Such representations can be later used to develop a semantic network in the form of a look-up memory.
Brief description of the drawings
An example of the present invention will now be described in detail with reference to the accompanying drawings in which:
Figure 1 is a schematic diagram of a neural network architecture in accordance with the present invention;
Figure 2 is a schematic diagram of an ACTION neural network; Figure 3 is the schematic diagram of Figure 1 illustrating selected features in more detail;
Figure 4 is a schematic diagram of a neural network architecture in accordance with the present invention;
Figure 5 is a schematic diagram illustrating selected aspects of the neural network architecture in more detail; Figure 6a-i are graphs plotting against time the excitation of parts of a neural architecture in accordance with the present invention;
Figure 7 is a schematic diagram of a neural network architecture in accordance with the present invention; and,
Figures 8a-8h are graphs plotting against time the excitation of parts of the neural network or Figure 7.
Detailed description
Manipulatable symbols are what makes advanced thinking possible. In the neural architecture of the present invention, the following are used: • representations powerful enough to describe concepts;
• working memory structures for use during intelligent task solution;
• potentially recurrent architectures so as to allow for holding activity and its possible transformations; and, • creation of learnt schemas of action sequences, which can be used in an automatic manner to extend the powers of the neural architectures. In natural language, words represent concepts and it is these that must be represented and manipulated. Natural language uses symbols (phonemes, syllables, morphemes, words) that are encoded in such a manner as to allow an infinite number of transformations on them. The combination of syntax (the grammatical arrangement of words, showing their connection or relationship) with the surrounding context must be processed to determine the associated semantics (the meaning) for the words being heard, read or otherwise received as an input. In this manner, an appropriately programmed system is able to produce words as part of manipulatable and meaningful strings. In the present invention, syntax and semantics of incoming (segmented, pre-processed) sound patterns are processed in parallel. The system is programmed such that semantics guides the development of syntax, and that semantics of single words arise from actions possible on an object (for nouns) or actions that coπespond to the word (for verbs). Semantics of object nouns are encoded in terms of sets of actions and associated sensory input patterns. Actions may also be associated with verbs.
Taking words from, for example, the set: Nouns: ball, floor, door, table, target, box. Verbs: reach, grasp, release, push, shut, touch, lift. Sequences of actions taken on the objects are coupled to the appropriate actions for the verbs. In this way, a robotic system that has lifted a box, when told that the object it has in its hand is 'box' and when it learns it is 'lifting' that box, will associate this action with the natural language structure 'lift box' (or 'box lift' according to the language parameters determining ordering) in which the action verb functions as a verb and the object as the noun in the phrase structure analysis: verb phrase = (lift=V)(box=N).
Some of the sequences of possible actions that can be taken on each of the objects (we only consider a few representatives) are as follows: Box: reach hand^touch box*^ grasp box-^lift box^release box; Ball: reach hand-^touch ball-^push ball; Target: reach hand-^touch target-^grab target ■^lift target ^target on table.
Figure 1 is a schematic diagram of a neural network architecture according to the present invention.
Objects are represented by Self Organising Feature Maps (SOFM) in an object representation module 10. Such a representation allows a simple localised interpretation. Nodes of the SOFMs are modelled as leaky integrators so as to form a buffer that acts as a working memory site. Object coding is by dedicated nodes for simplicity.
Actions are coded in an ACTION network 20 to allow for sequence learning. The architecture of an ACTION network is described in detail below, with reference to Figure 2.
Due to their basis on the architecture of the Human brain, ACTION networks are described herein in terms of the names of those parts of the Human brain.
For actions, each action of the set specified above is coded as a single cortical node in an ACTION network. Action sequences associated with a given object representation are linked to the object's dedicated node by link 30 using a Hebbian learning process, in which the relevant action nodes are activated (according to the actions being taken) by the particular active object representation. There are explicit modules for verb and noun word representations 40, 50 and verb phrase representations 60. Upon future activation of an object representation in the object module 10, the associated action representations in action module 20 will be activated via link 30. However, where there is no source of 'willed intention' (normally an external control signal) from willed, intention module 70, the actions are not actually taken. Instead, the action is primed, as allowed by the connections in Figure 1. As such, semantics are initially encoded as "virtual actions".
If the verb phrase module 60 is activated by a signal from the willed intention module 70 to generate a verb, the signal is passed to the verb representation module 40 to select an appropriate verb (guided by the relevant action representation from the action representation module 20). The verb phrase module 60 then activates a noun in the. noun representation module 50 (guided by the relevant object representation from the object representation module 10 activated by the input I). Other inputs to the verb phrase analyser may include inputs from other phrase analysers or direct speech leading to speech recognition.
The noun representation module 50 is also an ACTION network. Connections between the relevant action representation and the verb representation modules, and between the object representation module 20 and the associated noun representation module 50, are achieved by Hebbian learning of the bi-directional connections between the object representation in module 10 and the noun representation module 50 and between the action representation module 20 and the verb representation module 40. Alternatively, these connections can be hard- wired.
Figure 2 is a schematic diagram of an ACTION neural network architecture. The architecture of an ACTION neural network is based on current understanding of the features of the frontal lobes of the Human brain. The architecture is described in terms of the names of the parts of the Human brain for simplicity but can be implemented completely in a computer program or neural hardware, such as the pRAM (probabilistic RAM, details of which can be found in IEEE Transactions on Computers, Vol. 41, No. 12, December 1992 pl552-1561) chip. Each of the boxes shown in Figure 2 represent a grouping of interconnected neurons forming a neural network capable of accepting an input and providing an output if the inputs cause the network to fire. The main difference between an ACTION network and other types of neural networks is that there are feedback loops between certain groupings that model, in a simplistic level, current understanding of the operation of areas of the brain.
It is possible in an ACTION network to achieve temporal sequence storage and generation (TSSG). Suitably hard-wired or trained ACTION neural network architectures have been shown to achieve chunking (the composition of basic elements such, as words, into higher representations, such as sentences) of temporal sequences. An ACTION network include, or may act as, initiator, transition or memory modules that have continued activity during delay periods.
Initiator modules that operate to initiate a particular sequence. For this function they hold a memory of the entire sequence. Transition modules are only active during the delay between two elements of a sequence. For this function they hold a memory .of the transition between the two elements. Memory modules have activity that continues throughout the whole of the sequence response, although this is greatest in between elements. It also persists throughout the period in which the element is active, but at a reduced level. This activity may be specific to a particular sequence or to sequences containing classes of words. Memory modules allow the continuation of the memory of the particular sequence throughout the sequence itself. The initiator, transition and memory modules may be present in the cortex, thalamus, STN and STR as well as in a reflected form in the globus pallidi.
Neurons are modelled as mean firing rate leaky integrators. Hebbian reinforcement is used for learning. Sequence production is only allowed to continue if the action selected is correct.
In Figure 2, excitatory connections are shown as open arrowheads whilst inhibitory connections are shown as closed arrows. Data passes via an input (I) into the cortex 100 and then on to the striatum (STR) 110. From the striatum 110, data is passed to the globus pallidus (internal) (GPi) 120, then to the thalamus (TH) 130 and then back to the cortex 100. Thus, a feedback loop of activity occurs around this loop. There may also be similar feedback more rapidly around the cortex 100 to thalamus 130 loop (shown as two-way link 140). There is also an indirect loop, involving the globus pallidus external (GPe) 150 as well as the sub-thalamic nucleus (STN) 160. In certain cases, the action network may include an output node 170. This node is fed by activity from the cortex
100 and only allows the action network to generate an output if neurones in the output node 170 exceed a predetermined threshold value. The system can also produce an output from the cortex 100 or thalamus 130. The following description provides a mathematical basis for the operation of an
ACTION network:
Neurones are modelled as leaky integrators, with a mean firing-rate response given by the positive part of a sigmoidal function.
The network includes weighting factors to be applied to its connections including: w1 - connections from the input layer to the cortex; w2 - lateral connections in the cortex; w3 - connections from the cortex to the STR; w4 - lateral connections in the STR; w5 - connections from TH to the cortex; w6 - connections from the cortex to STN; w7 - connections from STN to GPi; w8 - connections from STR to GPi; w9 - connections from STN to GPe; w10 - connections from STR to GPe; w - connections from the cortex to TH; wn - connections from the GPi to TH; w13 - connections from GPe to STN; The network's potentials are: s for the cortex; a for input; m for TH; n for STN; u for GPe; r for STR; and ά for GPi. The time constant of the neurones is indicated by λ and is a value between 0 and 1. The neurone equations are: s{t+l) = λs{t) + wl* a{t),Ql) + w2*fXs{t), θ2) + w5*flm{t), θ3) (1) n{t+l) = λn(t) + w6*/(s(f), θ4) - w13*/(M(t), θ5) + T (2) where T = is a tonic input; r(t+l) = λr(t) + w3*/(s(t), θ4) + w4*Mt),Q6) (3) d{t+l) = λd{t) + w7 n{t), θ7) - w8* ( t), θ6) (4)
«(t+l ) = λ«(t) + w9*/(«(t), θ7) - w10*/(r(t), θ6) (5) m{t+l) = λm{t) + wn*/ (t), θ4) - w12*/Mt), θ8) (6) where * indicates convolution for lateral connections in a given module and auditory input to the cortex, with appropriate kernels for the weights. The weights in equations (1) - (6) are introduced to achieve sustained recurrent activity. The output function /is defined as:
, - ftanh (x - a), if x - a > 0
10, otherwise (7)
where α is the threshold. Other functions, sigmoidal or otherwise, may be used. Activity is held during the delay between sequence elements by the combination of the loop from cortex 100 to TH 130 to cortex 100 via the direct and indirect pathways as well as the reciprocal connections between cortex and TH 130. The handing on of activity from one type of neuron to another, such as from the memory neuron to transition neuron, is mainly an effect of lateral connections in the cortex 100. When the memory neuron exceeds its learnt threshold for use of the lateral weights it excites the transition neuron. The activity of the memory neuron is kept sub-threshold by lateral inhibition in the STR 110 from the initiator.
Connections I to cortex 100 and cortex 100 to STR 110 are required always to be excitatory. Lateral STR connections are always inhibitory. The remaining connections (lateral cortex, cortex to OUT and TH to cortex) are allowed to switch during the learning process from excitatory to inhibitory, and vice versa. In Figure 1, the willed intentions module (WI) 70 is only activated when it is desired for an action to be being taken. Learning can then take place between the action representations module 20 and the associated object representation module 10. Activity in the action representations during learning of the words associated to objects (or the actions themselves) leads to learning not only of the connections between object representations and associated noun representations (as shown by the bi-directional line joining the relevant nodes in each of these modules in Figure 1) but also of the route through the verb phrase module 60 joining the noun representation module 50 and the set of relevant actions in the action representation module 20. A direct route is also possible, as is shown by link 80 in Figure 1. During later activation of objects via an input to the object representation module 10, if the willed intentions module 70 is not active, sufficient potential is not available for the associated action representations to fire. However, the potential is sufficient to prime the action representation, thus leading to guidance of the associated word representations by "virtual actions".
Figure 3 is a schematic diagram of the neural architecture of Figure 1 in which selected elements and connections are shown in more detail. In particular, the elements of the architecture of Figure 1 necessary for dealing with syntax for 2 word sentences are described in detail.
In English, a command such as 'push box' or 'open door' are verb phrases (VP=Verb+Noun). Such verb phrases must be given meaning by a system in order to implement them. Noun phrases such as 'door handle' or 'box lid' may also have to be understood. In addition to such phrases, it is necessary to comprehend other types of phrases and combinations of phrases. In all cases, the phrases can be broken down into sequences, or sequences of sequences, each of which can be processed semi- independently.
In order to produce such sequences, three new types of ACTION networks are introduced, initiator networks (IN), memory networks (MEM) and transition networks (TRANS).
The verb and noun modules 40, 50 each include ACTION networks (41-42; 51- 52) of the type discussed with reference to Figure 2 having output nodes. The verb phrase module 60 includes initiator ACTION networks 61, 62, memory ACTION networks 63, 64 and transition ACTION network 65, 66. These ACTION networks are of the type discussed with reference to Figure 2 without output nodes. In both cases, the ACTION networks (41, 42,51,52,61-66) are illustrated as a pair of horizontally linked nodes, the upper node being the cortex of the network whilst the lower node is the striatum (STR). In the illustrated connections, excitatory connections are shown as open arrowheads and inhibitory connections are shown as closed arrowheads.
An example of the present invention will now be described with reference to Figure 3, in which the neural architecture is programmed to generate two verb phrases: lift box and shut door. In order to produce the desired output, an object is presented throughout the task as an input to the object representation module 10. This has the effect of exciting the corresponding object representation. A willed input signal is also applied by the willed input module 70 as a general input to the 2 ACTION networks 21, 22 of the 2 actions, "lift" and "shut", and at the IN networks 61, 62 for both verb phrases. The presentation of the object leads to low-levels of activity in the associated noun representation and action representation. This in turn leads, in both the noun and action representation modules 20, 50, to inhibitory activity on the non-associated representations. The low-level of activity that results in the action representation generates further low activation on the associated verb representation and on the initiator network for the desired verb phrase. This again leads to inhibition of non-associated verb representations and other verb phrase initiators.
Thus, presentation of the object "box" results in the priming of the noun "box", the connected action "lift", verb "lift" and verb phrase initiator for "lift box". At the same time, there is inhibition of the noun "door", action "shut", verb "shut" and the verb phrase initiator for "shut door". When the willed intention signal is presented, this priming leads to the generation of the action "lift" on the object "box" along with generation of the verb phrase "lift box". Activity in the respective ACTION networks of the verb phrase module 60 upon presentation of the willed intention signal will now be discussed.
The presentation of the willed intention signal leads to the excitation of the primed IN cortex neuron and MEM cortex neuron. Inhibition via an STR connection 67 from the IN to MEM network prevents initial high activity in the MEM network. Due to the low activation of the MEM cortical neuron the TRANS cortical neuron also has low activation during this initial period. As the IN network activity increases it excites the associated verb network in verb representation module 40 leading to the respective verb's ACTION network's OUT neuron generating an output when its threshold is exceeded. Feedback inhibits the verb network and corresponding IN network, reducing activity to a low level. This releases the IN inhibitory action on the MEM network, allowing the activity of its cortical neuron to increase which, via the cortical connection 68, leads to the activation of the TRANS network. The TRANS cortical neuron then causes an increase in the activity of the corresponding noun network leading to the noun OUT neuron generating an output by exceeding its threshold. Feedback then inhibits the MEM, TRANS and noun networks. The non-zero activity seen to exist on some neurons after the verb phrase has been completed is due to the continued presence of the object. If the WI is turned on again the action and verb phrase will be generated once again. Initiator and transition networks need not be word specific. Instead, the neurons may be word class specific, such as for nouns representing man-made or natural objects. Presentation of speech sequences leads to similar activation without the use of the willed intention signal.
Figure 4 is a schematic diagram of the neural architecture of Figure 1, including further features, in accordance with a preferred aspect of the present invention.
In this example, the neural network architecture of Figure 1 is extended to include an adjectival phrase module 310 and an adjective representation module 300. The adjectival phrase module 310 is configured to allow a number of adjectives to be produced prior to a noun. Figure 5 is a schematic diagram illustrating selected features of the neural network architecture in more detail.
The architecture is extended to include an adjective module 300 an adjectival phrase module 310 and a feature representation module 330. A feature is an attribute such as "red" or "heavy" and has a neural representation in the features module 330. The adjective and noun modules 300, 50 each include ACTION networks (301-303; 51-53) of the type discussed with reference to Figure 2 having output nodes. The adjectival phrase module 310 includes an initiator ACTION network 311 and a transition ACTION network 312 of the type discussed with reference to Figure 2 without output nodes. In both cases, the ACTION networks (51-53; 301-303; 311, 312) are illustrated as a pair of horizontally linked nodes, the upper node being the cortex of the network whilst the lower node is the striatum (STR). In the illustrated connections, excitatory connections are shown as open arrowheads and inhibitory connections are shown as closed arrowheads.
In operation, an input is applied to feature representation module 320 and object representation module 10. Those representations corresponding to the input become excited and pass this excitation to their linked adjective and noun representations via the respective ACTION networks (301-303; 51-53). This produces a low level activity in those ACTION networks that in turn inhibits those ACTION networks of adjective and noun representations without excitation via lateral STR connections 340. Thus, a number of adjective representations and a noun representation are primed. A signal from the willed intention module 70 to create a sentence is passed to the initiator network 311. At the same time, the willed intention module 70 causes inhibition to all links from the adjective representation module 300 to the transition network 312. Preferably, the willed intention module is arranged to monitor the feature representation module 320 and to only remove this inhibition when there are no features excited. .
The initiator network 311 passes on the willed intention signal by exciting all the Action networks 301-303 of the adjectival representations, whether they are primed or not. However, only those that are primed by excitation from feature representations 320 can produce sustained activity and excite their OUT neurone beyond the threshold value. The striatal lateral connections 340 between adjective ACTION networks 301-303 also ensures that only one primed adjective ACTION network exceeds the threshold at a time. Hence, only one word can be generated at a time.
Once an adjective exceeds the threshold, that word is generated and inhibition turns off the feature. This in turn reduces the excitement to the adjective ACTION network 301-303 associated with the word, preventing it from firing again. If the inhibitory willed intention signal is still active then the transition network 312 is unable to be excited and the initiator network 311 remains active such that another adjective can be generated. As mentioned above, the willed intention signal that inhibits all inputs to the transition network 312 is removed if no features in the feature representation module 320 are active. Once the inhibition is removed, excitation to the transition network 312 is applied by the last generated adjective. The transition network 312 excites all noun representations via their respective noun ACTION networks 51-53. However, only the primed noun ACTION network, having been previously excited by an excited object representation, can exceed threshold set at its OUT node and generate a noun. Once the noun is generated the whole model is inhibited, turning off object and noun representations, the transition network and any un-generated features and adjectives.
Figure 6a-i shows the time-courses of some of the neurones involved in generating an adjectival phrase composed of 3 adjectives. The excitation levels of the 3 active feature representations are plotted against time in Figures 6a to 6c, respectively. Figures 6d to 6f are plots in time of cortical activities of the 3 adjective representations, each linked to one of the feature representations. A progression through time can be seen as the maximum activities are reached. In Figure 6g, it is shown that the generated noun reaches its maximum activity after all the adjectives. Figure 6h shows sustained activity of the initiator network whilst the adjectives are being generated. Finally, Figure 6i shows the transition network becoming active once the inhibitory willed intention signal is removed. Figure 7 is a schematic diagram of a neural network architecture in accordance with the present invention. Solid arrowheads represent inhibitory stimulus being applied in the direction of the arrow whilst open arrowheads represent excitatory stimulus.
The neural network architecture includes a noun-phrase analyser 400, a verb- phrase analyser 420, an infinitival-phrase analyser 430 and a complementiser phrase analyser 440. Analysers for adjectives and prepositions are not included but could be added following the same architecture pattern without difficulty.
Each analyser is made up of a number of modules that are each in turn ACTION neural networks which have been described in detail with reference to Figure 2. Taking as an example the noun-phrase analyser 400, a first level noun-phrase module 401 is connected to a second level noun-phrase module 402. The two modules 401, 402 are. composed of initiators and transitions and the link is excitatory from the first level module 401 to the second level module 402 but inhibitory in reverse. The first level module 401 is also linked to a specifier module 403 via excitatory links in both directions. The second level module 402 is linked to a word module 404 via excitatory links in both directions. The specifier module 403 and the word module 404 are storage regions for words of the appropriate type (nouns in the case of the noun-phrase analyser), the specifier module may store determiner type words (e.g. words such as 'when', 'who' and 'how' in the case of the noun-phrase analyser).
More specifically, the initiator neurone of the first level noun-phrase module 401 is connected via excitatory weights to all words in the specifier module 403, which are in turn connected via excitatory weights back to the transition neurone of the first level noun-phrase module 401. All transition neurons of the first and second level noun-phrase modules act in an inhibitory manner via lateral striatal connections on their associated initiator neurones. The transition neurone of the first level noun-phrase module 401 is connected via excitatory weights to the initiator neurone of the second level noun-phrase module 402 which in turn is connected back to the transition neurone of the first level noun-phrase module 401 via inhibitory weights. The initiator neurone of the second level noun-phrase module 402 is connected via excitatory weights to all words in the word module 404, which are in turn connected via excitatory weights back to the transition neurone of the second level noun-phrase module 402. This connection architecture is mirrored for each analyser 400-440.
In addition, the initiator of each first level module 401, 411, 421, 431, 441 is linked via inhibitory weights to selected ones of the other analyser's second level module 402, 412, 432, 442 transition neurones. There is also a return link via excitatory weights. The inter-analyser connections can be seen in Figure 7, however in summary they are: noun-phrase to infinitival-phrase; infinitival-phrase to verb-phrase; verb-phrase to noun-phrase; complementiser-phrase to noun-phrase; . complementiser-phrase to infinitival-phrase; and, noun-phrase to complementiser-phrase.
If the architecture included a prepositional-phrase analyser it would be connected to the complementiser phrase analyser.
The architecture only allows the initiator of a first or second level module to excite a word module or specifier module. Transition either projects to a second level module or to a first level module of another analyser. In this manner, words can only be generated when there is priming.
The word modules 403, 423, 433, 443 have output neurones connected to the cortical nodes of their ACTION neural networks. In this manner, when output nodes exceed a threshold the word is produced, at which point inhibition is introduced for, for example, 80 time steps for word representations and also associated representations (object representations for nouns, action representations for verbs etc.). In order to preserve knowledge of input activation, weight traces are introduced as a weight modulation: w(t + 1) = λw{f) + (l - Λ)lnput
Using this architecture grammatically sensible components of sentences can be moved whilst still making sense, therefore permitting understanding of phrases with changes in word order from those previously learnt and also generation of phrases that more closely fit real speech. Training of the neural network is performed by use of both causal hebbian and reinforcement learning techniques, in which traces are included on synapses to give longer and smoother responses to help bridge temporal gaps between inputs. For example, a short movement occurs with auxiliary verbs when a statement changes to a question:
"I can have a Z" changes to "Can I have a Z"
For this case, the neural network is taught the rule that "can", an infinitival word, is excited in its new position by the complementiser phrase analyser and that structures that excite it in its infinitival statement form are still active in the question form but do not lead to a second activation. This is caused either by lasting inhibition or lack of priming.
Referring to elements by their reference numerals, the excitation flow in the neural network to generate the original sentence is: 401 (initiator) -^403 (no word generated) ■■►401 (transition) -^402 (initiator)
■^404 (generates T) -^402 (transition) ->431 (initiator) -^433 (no word generated) ■ 431 (transition) ■ 432 (initiator) - 434 (generates 'can') - 432 (transition) ->421 (initiator) - 423 (no word generated) -^421 (transition) τ>422 (initiator) "^424 (generates 'have') ■■►422 (transition) HM01 (initiator) - 403 (generates 'a') ^401 (transition) -»402 (initiator) -»404 (generates 'Z') -»402 (transition).
Once the noun 'Z' is generated, the sentence is completed leading to global inhibition from a higher order module (not shown).
For the moved sentence, the excitation flow is:
441 (initiator) -^443 (no word generated) -^441 (transition) -^432 (initiator) ■^434 (generates 'can') τ*432 (transition) -^401 (initiator) *^403 (no word generated) -»401 (transition) - 402 (initiator) -»404 (generates T) -»402 (transition) -»431 (initiator) -i>433 (no word generated) -^431 (transition) -^432 (initiator) -^434 (no word generated as 'can' already produced) -^432 (transition) τ>421 (transition) -^422 (initiator) -^424 (generates 'have') -^422 (transition) H>401 (initiator) -i>403 (generates 'a') -»401 (transition) -»402 (initiator) -»404 (generates 'Z') - 402 (transition).
Again, when 'Z' is generated there is global inhibition.
Figures 8a-8h are graphs plotting against time the excitation of parts of the neural network or Figure 7. Figures 8a to 8d show the activation pattern of the cortical neurones involved in the generation of the original sentence. These show the correct temporal order of activation of the words T (Fig 8a), 'can' (Fig 8b), and 'have' (Fig. 8d) as well as a single strong activation of the second level infinitival-phrase module initiator neurone (Fig. 8c), the second smaller activation being due to excitation of the first level infinitival- phrase module by the second level noun-phrase module before global inhibition takes effect. Figures 8e to 8h show the same neurones as Figures 8a to 8d but activated in the moved sequence, 'can' (Fig. 8f), T (Fig. 8g), 'have' (Fig. 8h). In this case there are two strong activations of the second level infinitival-phrase module initiator neurone (Fig. 8e), but only one activation of the word 'can' that leads to generation, the second activation leading only to low levels of activity, as can be seen at around time 200 in Figure 8f.
The above description has been concerned primarily with neural networks that have been trained or suitably hard-wired to achieve the described architecture. However, it will be apparent to the skilled reader that such an architecture could also be achieved by training an action network that has sufficient neurons to permit chunking. In the examples, the neural network architectures operate to generate verb and noun based sentences. The architecture could be trained or appropriately hard-wired to act as phrase analysers for sentence syntax and those that control morpheme combination. The architecture may include sentence analysers, noun phrase analysers, verb phrase analyser, prepositional phrase analysers and adjectival phrase analysers. Morphemic combination analysers may include derivational analysers (prefix and suffix connectable together to add a prefix and any number of suffixes to a morpheme stem) and inflectional analysers involved in verb formation, including conjugational analysers to provide the present tense forms, regular past tense analysers and irregular past tense analysers.
After initial learning of nouns, verbs etc., as well as the development of phrase structure analysers using the above described architectures, a semantic memory may be incrementally constructed for the nouns and verbs, etc., and for new words of a more abstract form but related to the earlier words. This semantic memory may be an associative memory/dictionary look-up which can be constructed using the priming from the representations, especially using the feature representations associated with words, to guide clustering in a general feature space. A more specific form of such a memory can be an associative neural network memory, learnt by a competitive and/or Hebbian form of learning, such as a Hopfield network or a self-organising feature map. A hardware representation is possible in terms of RAM-based neural systems, such as the pRAM. This semantic memory may be extended to new. word learning, and used to replace the meaning given to words by means of the more computationally intensive ACTION network systems described herein, so as to improve the speed of language processing in both recognition and production. The control of the willed intention module and other control aspects of this system could be performed by an attentional control system such as that described in co-pending patent application (agents reference P15265GB).

Claims

Claims
1. A natural language system comprising a phrase module connected to a plurality of word modules, each word module being associated with a class of words and holding representations of words of the respective class, the word modules being configured to prime a representation upon presentation of a stimulus associated with the respective representation, wherein the phrase module includes a neural network configured to determine an excitation sequence in dependence on the primed word representations and to trigger the primed word representations in accordance with the excitation sequence.
2. A natural language system according to any preceding claim, in which each word module is connected to an element representation module, the element representation module having a portion associated with each of a plurality of elements and being aπanged to accept an input and to produce a stimulus from an associated portion .if its element is present in the input, the element representation module being arranged to present the stimulus to the word module via the connection.
3. A natural language system according to claim 1 or 2, in which a class of a word module is selected from the set of nouns, pronouns, adverbs, verbs, prepositions, adjectives, derivational morphemes and inflectional morphemes.
4. A natural language system according to any preceding claim, further comprising an action generator connected to a verb word module and a noun word module wherein, upon triggering of a verb from the verb word module and a noun from the noun word module, the action generator is configured to trigger an appropriate action.
5. A natural language system according to claim 1, further comprising a plurality of phrase modules, each phrase module being associated with a class of phrase.
6. A natural language system according to claim 5, wherein each phrase module comprises a first level phrase module connected to a second level phrase module and a specifier module, the second level phrase module also being connected to a word module, the specifier and word modules comprising memories for words associated with the phrase class, wherein each connection is excitatory in both directions except for the connection from the second level phrase module to the first level phrase module which is inhibitory.
7. A natural language system according to any preceding claim, in which a phrase module is arranged to process one of the set of sentences, noun phrases, verb phrases, prepositional phrases, adjectival phrases, morpheme combination with morphemes, prefixes or suffixes, conjugation and tense formation.
8. A natural language system according to claim 6, in which the phrase modules include a noun phrase analyser, an infinitival phrase analyser, a verb phrase analyser and a complementiser phrase analyser.
9. A natural language system according to claim 8, comprising a connection of a first connection class originating from the verb phrase analyser to the infinitival phrase analyser.
10. A natural language system according to claim 8 or 9, comprising a connection of a first connection class originating from the complementiser phrase analyser to the noun phrase analyser.
11. A natural language system according to claim 8, 9 or 10, comprising a connection of a first connection class originating from the verb phrase analyser to the noun phrase analyser.
12. A natural language system according to claim 8, 9, 10 or 11, comprising a connection of a first connection class originating from the complementiser phrase analyser to the verb phrase analyser.
13. A natural language system according to any of claims 8 to 12, wherein the first connection class comprises an excitatory connection from the originating phrase analyser's second level phrase module to the first level phrase module of the other phrase analyser.
14. A natural language system according to claim 13, wherein the first connection class comprises an inhibitory connection to the originating phrase analyser's second level phrase module from the first level phrase module of the other phrase analyser
21. A natural language system according to claim 6 or any of claims 8 to 14, wherein each module comprises an ACTION network.
22. A natural language system according to claim 1, in which the neural network comprises an ACTION neural network.
23. A natural language system according to claim 21 or 22, in which the ACTION network includes an initiator node configured to accept a stimulus and trigger the start of a sequence.
24. A natural language system according to claim 21, 22 or 23, in which the ACTION network includes a transition node configured to track the triggering and trigger in order the primed word representations according to the excitation sequence.
25. A natural language system according to claim 21, 22, 23 or 24, in which the ACTION network includes a memory node configured to store information relating to the sequence.
26. A natural language system according to claim 22, 23, 24 or 25, in which the or each node comprises an ACTION network.
27. A natural language system according to claim 22, 23, 24 or 25, in which the or each node comprises a portion of an ACTION network.
28. A natural language system according to any preceding claim, in which each word module includes an ACTION neural network configured to accept the presented stimulus and to prime a representation associated with the stimulus by holding the stimulus in a portion of the network corresponding to that representation, wherein the portions of the network corresponding to the associated representation are linked to the phrase module to pass an indication of the stimulus to the phrase module.
29. A natural language system according to claim 28, in which each portion of the network has an associated stimulus threshold, the stimulus being held in the portion until it exceeds the threshold, thereby causing the ACTION network to fire for the respective word.
30. A natural language system according to claim 29, in which the stimulus threshold is set greater than any presented stimulus, wherein the phase module is arranged to apply a further stimulus to the word module's ACTION network, thereby triggering the primed word representation, in accordance with the excitation sequence.
31. A natural language system according to any preceding claim, further comprising a control module connected to the phrase module and being configured to apply a stimulus to the phrase analyser to trigger the primed word representations.
32. A natural language system according to any preceding claim, further comprising a memory arranged to store a presented stimulus with the corresponding primed word representation.
33. A natural language system according to claim 32, in which the memory comprises a plurality of neurons, the system further comprising trace means for tracing synapses between neurons.
34. A natural language system according to claim 32 or 33, further comprising learning means for training the neural network(s) by causal Hebbian and/or reinforcement training.
35. A natural language processing method comprising: maintaining a phrase module connected to a plurality of word modules, each word module being associated with a class of words and holding representations of words of the respective class; priming a representation within a word module upon presentation of a stimulus associated with the respective representation, wherein the phrase module includes a neural network performing the steps of determining an excitation sequence in dependence on the primed word representations and triggering the primed word representations in accordance with the excitation sequence.
36. A method according to claim 35, in which each word module is connected to an element representation module, the element representation module having a portion associated with each of a plurality of elements, wherein each element representation module performs the steps of accepting an input, producing a stimulus from an associated portion if its element is present in the input and presenting the stimulus to the word module.
37. A computer program comprising computer executable code for operating the system of any of claims 1 to 34.
38. A computer program comprising computer program code means for executing the steps of claim 35 or 36.
39. A pRAM neural chip configured to execute the computer program of claim 37 or 38.
40. A natural language system comprising a first memory encoding a phrase module and a further memory encoding a word module, the phrase module being linked to the word module, wherein the word module is associated with a class of words and holds representations of words of the respective class, wherein upon presentation of a stimulus associated with the respective representation the word modules is configured to prime a representation, wherein the phrase module includes a neural network configured to determine an excitation sequence in dependence on the primed word representations and to trigger the primed word representations in accordance with the excitation sequence.
PCT/GB2001/003194 2000-07-17 2001-07-17 Natural language system WO2002007003A2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
AU2001272640A AU2001272640A1 (en) 2000-07-17 2001-07-17 Natural language system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
GB0017524.0 2000-07-17
GBGB0017524.0A GB0017524D0 (en) 2000-07-17 2000-07-17 Natural language system

Publications (3)

Publication Number Publication Date
WO2002007003A2 true WO2002007003A2 (en) 2002-01-24
WO2002007003A3 WO2002007003A3 (en) 2003-10-02
WO2002007003A9 WO2002007003A9 (en) 2003-11-27

Family

ID=9895815

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/GB2001/003194 WO2002007003A2 (en) 2000-07-17 2001-07-17 Natural language system

Country Status (3)

Country Link
AU (1) AU2001272640A1 (en)
GB (1) GB0017524D0 (en)
WO (1) WO2002007003A2 (en)

Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5794050A (en) * 1995-01-04 1998-08-11 Intelligent Text Processing, Inc. Natural language understanding system

Patent Citations (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5794050A (en) * 1995-01-04 1998-08-11 Intelligent Text Processing, Inc. Natural language understanding system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
TAYLOR N R ET AL: "Learning to generate temporal sequences by models of frontal lobes" IJCNN'99. INTERNATIONAL JOINT CONFERENCE ON NEURAL NETWORKS. PROCEEDINGS (CAT. NO.99CH36339), PROCEEDINGS OF INTERNATIONAL CONFERENCE ON NEURAL NETWORKS, WASHINGTON, DC, USA, 10-16 JULY 1999, pages 38-41 vol.1, XP002227642 1999, Piscataway, NJ, USA, IEEE, USA ISBN: 0-7803-5529-6 *

Also Published As

Publication number Publication date
WO2002007003A9 (en) 2003-11-27
GB0017524D0 (en) 2000-08-30
WO2002007003A3 (en) 2003-10-02
AU2001272640A1 (en) 2002-01-30

Similar Documents

Publication Publication Date Title
Christiansen et al. Connectionist natural language processing: The state of the art
Feldman et al. L 0—the first five years of an automated language acquisition project
Miikkulainen Subsymbolic case-role analysis of sentences with embedded clauses
JP2764343B2 (en) Clause / phrase boundary extraction method
Huyck A psycholinguistic model of natural language parsing implemented in simulated neurons
Thompson et al. Learning to parse natural language database queries into logical form
Tozzo et al. Neural event extraction from movies description
Bryson Embodiment versus memetics
Reilly A connectionist model of some aspects of anaphor resolution
Miikkulainen Subsymbolic parsing of embedded structures
WO2002007003A2 (en) Natural language system
Collier An historical overview of natural language processing systems that learn
Corston-Oliver et al. Combining decision trees and transformation-based learning to correct transferred linguistic representations
Adorni et al. Natural language input for scene generation
Stabler et al. The learning and emergence of mildly context sensitive languages
Wellington et al. Scalable purely-discriminative training for word and tree transducers
Rooth Finite state intensional semantics
Smith et al. A software toolkit for modeling human sentence parsing: An approach using continuous-time, discrete-state stochastic dynamical systems
Kaumanns Assessment and analysis of the applicability of recurrent neural networks to natural language understanding with a focus on the problem of coreference resolution
Zhang et al. CMLM-CSE: Based on Conditional MLM Contrastive Learning for Sentence Embeddings
Prasad et al. SPAWNing Structural Priming Predictions from a Cognitively Motivated Parser
Björkman Machine Learning Evaluation of Natural Language to Computational Thinking: On the possibilities of coding without syntax
Reilly Boundary effects in the linguistic representations of simple recurrent networks
Pease et al. Towards an automatic translation of medical terminology and texts into Arabic
Okediya et al. Building Ontologyfor Yorùbá Language

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A2

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NO NZ PL PT RO RU SD SE SG SI SK SL TJ TM TR TT TZ UA UG US UZ VN YU ZA ZW

AL Designated countries for regional patents

Kind code of ref document: A2

Designated state(s): GH GM KE LS MW MZ SD SL SZ TZ UG ZW AM AZ BY KG KZ MD RU TJ TM AT BE CH CY DE DK ES FI FR GB GR IE IT LU MC NL PT SE TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application
REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

122 Ep: pct application non-entry in european phase
COP Corrected version of pamphlet

Free format text: PAGES 20-24, CLAIMS, REPLACED BY NEW PAGES 20-24; DUE TO LATE TRANSMITTAL BY THE RECEIVING OFFICE

NENP Non-entry into the national phase

Ref country code: JP