US20020055950A1 - Synchronizing audio and text of multimedia segments - Google Patents

Synchronizing audio and text of multimedia segments Download PDF

Info

Publication number
US20020055950A1
US20020055950A1 US09/841,327 US84132701A US2002055950A1 US 20020055950 A1 US20020055950 A1 US 20020055950A1 US 84132701 A US84132701 A US 84132701A US 2002055950 A1 US2002055950 A1 US 2002055950A1
Authority
US
United States
Prior art keywords
audio
text
information
data
component
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Abandoned
Application number
US09/841,327
Inventor
Bradley Witteman
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Great Elm Group Inc
Original Assignee
Arabesque Communications Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Arabesque Communications Inc filed Critical Arabesque Communications Inc
Priority to US09/841,327 priority Critical patent/US20020055950A1/en
Publication of US20020055950A1 publication Critical patent/US20020055950A1/en
Assigned to OPENWAVE SYSTEMS INC. reassignment OPENWAVE SYSTEMS INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ATMOTION INC.
Assigned to ARABESQUE COMMUNICATIONS, INC. reassignment ARABESQUE COMMUNICATIONS, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: WITTEMAN, BRADLEY J.
Assigned to ATMOTION INC. reassignment ATMOTION INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: ARABESQUE COMMUNICATIONS, INC.
Abandoned legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • G06F16/48Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/40Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/06Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice
    • GPHYSICS
    • G10MUSICAL INSTRUMENTS; ACOUSTICS
    • G10LSPEECH ANALYSIS OR SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00Speech recognition
    • G10L15/08Speech classification or search
    • G10L2015/088Word spotting

Definitions

  • the invention relates to searching and retrieving multimedia information.
  • a method for retrieving a multimedia segment from a signal stream having an audio component and a closed caption component. The method includes separating the audio component and the closed caption component from the signal stream; generating an audio pattern representative of the start of the multimedia segment; locating the audio pattern in the audio component; and temporally aligning the text with the audio pattern in the audio component.
  • Implementations of the invention include one or more of the following.
  • the method includes retrieving text from the closed caption component; and comparing the text against one or more keywords delimiting the multimedia segment.
  • the keywords may delimit the start and the end of the multimedia segment, or may be stored in a dictionary.
  • the method may compare the text against one or more key characters delimiting the multimedia segment.
  • the key characters may be stored in a dictionary.
  • the method may include generating a concluding audio pattern representative of the end of the multimedia segment; and locating the concluding audio pattern in the audio component.
  • the method may identify the multimedia segment between the audio patterns.
  • the multimedia segment and text may be indexed in a database.
  • the multimedia segment may be played on-demand.
  • an apparatus retrieves a multimedia segment from a signal stream, the signal stream having an audio component and a closed caption component.
  • the apparatus includes means for separating the audio component and the closed caption component from the signal stream; means for generating an audio pattern representative of the start of the multimedia segment; means for locating the audio pattern in the audio component; and means for temporally aligning the text with the audio pattern in the audio component.
  • Implementations of the invention include one or more of the following.
  • the apparatus may include means for retrieving text from the closed caption component; and means for comparing the text against one or more keywords delimiting the multimedia segment.
  • the keywords may delimit the start and the end of the multimedia segment.
  • the keywords may be stored in a dictionary.
  • a means for comparing the text against one or more key characters delimiting the multimedia segment may be provided.
  • the key characters may be stored in a dictionary.
  • the apparatus may include means for generating a concluding audio pattern representative of the end of the multimedia segment; and means for locating the concluding audio pattern in the audio component.
  • a means for identifying the multimedia segment between the audio patterns may be used.
  • the multimedia segment and text may be indexed in a database.
  • the apparatus may include a means for playing the multimedia segment on-demand.
  • an apparatus retrieves a multimedia segment from a signal stream, the signal stream having an audio component and a text component.
  • the apparatus includes a decoder for separating the audio component and the text component from the signal stream; an audio synthesizer coupled to the decoder for generating an audio pattern representative of the start of the multimedia segment; a pattern recognizer coupled to the decoder and to the audio synthesizer for locating the audio pattern in the audio component; and an aligner coupled to the pattern recognizer and to the decoder for temporally aligning the text with the audio pattern in the audio component.
  • Implementations of the invention include one or more of the following.
  • the apparatus may include an extractor for retrieving text from the closed caption component; and a comparator for checking the text against one or more keywords delimiting the multimedia segment.
  • the keywords may delimit the start and the end of the multimedia segment.
  • a dictionary may store the keywords.
  • the text may be compared against one or more key characters delimiting the multimedia segment.
  • a character dictionary may store the key characters.
  • the synthesizer may generate a concluding audio pattern representative of the end of the multimedia segment and the pattern recognizer locates the concluding audio pattern in the audio component.
  • a database may store the multimedia segment between the audio patterns and the text.
  • a method for retrieving a multimedia segment from a signal stream, the signal stream having an audio component and a text component includes generating audio patterns representative of the start and the end of the multimedia segment; locating the audio patterns in the audio component; and delimiting a portion of the audio component between the audio patterns as the multimedia segment.
  • Implementations of this aspect may include temporally aligning the text with the start of the delimited audio component portion; and storing the text and the delimited audio component portion in a database.
  • a computer system includes a plurality of interdependent processors, each interdependent processor executing an independent operating system image without sharing file system state information.
  • Each interdependent processor has a network access card with a first network connection and a second network connection; and each interdependent processor has a decoder for separating the audio component and the text component from the signal stream; an audio synthesizer coupled to the decoder for generating an audio pattern representative of the start of the multimedia segment; a pattern recognizer coupled to the decoder and to the audio synthesizer for locating the audio pattern in the audio component; an aligner coupled to the pattern recognizer and to the decoder for temporally aligning the text with the audio pattern in the audio component; and a database coupled to the aligner for storing the audio component with the aligned text.
  • Each processor is connected to a first active backplane and to a second active backplane, the second active backplane operating in lieu of the first active backplane in case of a fail-over; and one or more peripherals connected to each of the first and second active backplanes and responsive to data requests transmitted over the first and second active backplanes.
  • Advantages of the invention include one or more of the following.
  • the invention allows value-added information such as news on business, sports, current events and entertainment to be accessible from the user's computer or telephone.
  • the response to the user's request is a natural sounding playback, in contrast to conventional text-to-speech (TTS) technologies whose outputs are stilted, mechanical and unnatural sounding.
  • TTS text-to-speech
  • the invention allows time intensive information, or information that requires the user to listen for extended periods of time, to be delivered smoothly and without interruptions. Further, the invention requires little computational power and is inexpensive to implement.
  • the invention also provides scalability and fault tolerance in delivering the value-added information.
  • the invention allows many servers to perform the same task in an active/active scalable manner.
  • the invention also supports load balancing among a pool of like servers. By providing a client process with access to a pool of like servers which are load balanced, the invention keeps the response time for each request to a minimum.
  • the invention supports high data availability, fast access to shared data, and low administrative costs through data consolidation. Additionally, the invention may be built using standard off-the-shelf components to reduce overall system cost.
  • FIG. 1 is a flow chart illustrating a process for processing a multimedia segment with reference to closed captions.
  • FIG. 2A is a diagram illustrating components of the multimedia segment.
  • FIG. 2B is an example of a multimedia signal containing closed-caption character data.
  • FIG. 3 is a flow chart illustrating a process for aligning the closed-caption character data with a multimedia segment.
  • FIG. 4 is a block diagram illustrating a first computer system for processing multimedia segments.
  • FIG. 5 is a block diagram illustrating a second computer system for processing multimedia segments.
  • FIG. 6 is a block diagram illustrating a computer node of the computer system of the present invention.
  • FIG. 7 is a block diagram illustrating a third computer system for processing multimedia segments.
  • FIG. 8 is a block diagram illustrating a fourth computer system for processing multimedia segments.
  • FIG. 9 is a diagram of software supported by the computer system of the present invention.
  • a process 400 for finding cue points delineating a start and an end of a multimedia segment is shown.
  • the process 400 separates an audio component and a closed caption component from the signal stream.
  • An audio pattern representative of the start of the multimedia segment is generated and the audio pattern is located in the audio component.
  • the process then temporally aligns the text with the audio pattern in the audio component.
  • the multimedia segment is delivered over a television feed.
  • a radio feed, a cable feed, or a computer generated multimedia feed can be used in lieu of the television feed.
  • the television feed is received (step 402 ).
  • the television feed includes an audio feed and a text feed.
  • the text feed can be a caption, or a written representation of the audio feed.
  • the caption is similar to a subtitle and is designed to allow hearing-impaired listeners to enjoy the program. In this case, the caption is used to determine the start and the end of the multimedia segment.
  • a signal from the television feed is analyzed and closed caption data is extracted (step 412 ).
  • Closed caption data is hidden in line 21 of a vertical blanking interval (VBI), which is the part between video frames and includes a number of lines of video.
  • VBI vertical blanking interval
  • Captions 1 and 2 and texts 1 and 2 are stored in line 21 of the first field in each frame, while the second field's line 21 can be used for captions 3 and 4 and texts 3 and 4 .
  • the closed caption text feed is then separated (step 414 ).
  • the text feed is compared to a dictionary of key words and phrases (step 416 ).
  • the keyword or the phrase is sent to a speech recognizer (step 418 ).
  • the speech recognizer searches recent audio feeds for the same keyword or phrase (step 420 ).
  • the audio feed is separated (step 422 ). From step 422 , the audio feed is provided to the speech recognizer of step 420 . The process 400 then determines a start of the audio block, indexes the audio block and sends the audio block to an information store (step 424 ). Finally, the information store synchronizes the catalogs audio and text blocks (step 426 ). Steps 412 - 424 collectively form a block 410 which is processed inside a computer such as the computer of FIG. 6.
  • the television feed enters the system and is separated into audio and text feeds.
  • the text feed is compared to a dictionary of known marker words and phrases such as “CNN Headlines Sports”, for example.
  • the word or phrase is sent to the speech recognizer to search recent audio feeds for that word or phrase.
  • the audio block is indexed and sent to the information store.
  • the speech recognizer system is then instructed to look for the key word or phrase that signals the end of the audio feed.
  • the text and audio feeds are then synchronized, indexed and stored in the information store awaiting user recall or system use.
  • the process of FIG. 1 tracks the closed caption content while trying to match text to a dictionary of key words and phrases.
  • the matched key words and phrases are extracted and sent to the speech recognizer. Sound information associated with the key words or phrases is compared against a buffered version of the audio feed. The sound feed and text feed are then aligned.
  • the result is a synchronized audio and text streams, even if the original audio and text (closed-caption) information were originally out of synch. Since the text feed is searchable and aligned with the audio feed, the user can search for the item of interest and can either read the text feed or listen to the audio feed.
  • the speech recognizer of step 418 is discussed next. This recognizer applies the current text feed from the closed caption information with the two dictionaries mentioned above. The recognizer performs the following functions:
  • the recognizer looks for an intersection between the current text string and entries in the dictionary of significant words, phrases and characters. When there is an intersection, it creates an open entry state or a close entry state.
  • the open entry state is entered when the intersection exists with the matching record in the information store being closed.
  • the close entry state is entered when the intersection exists with the current record being open.
  • Pseudo code for the open entry state is as follows:
  • Speech recognizer finds the target word or phrase identified by the search software in the audio feed
  • Pseudo code for the close entry state is as follows:
  • the speech recognizer finds the target word or phrase identified by the search software in the audio feed.
  • Delivery software that is able to search and retrieve a specific data block by the data block's unique identifier which will reflect the content type, and the date and time of the data block. The delivery agent will then send this data block to the user's phone or their SMS/Alphanumeric pager at the user's request or according to a preset schedule.
  • the content type of the text information and corresponding audio data is identified by various words/phrases or significant characters recognized in the dictionaries.
  • the dictionary of significant characters include those characters that are used to delineate the closed caption feed such as a triple carat (>>>) to signify the beginning of a story.
  • the word/phrase dictionary includes words and/or phrases commonly used to separate programming segments. Examples from CNN Headline News include “CNN Headline Sports” which relates to breaking sports news; “CNN Dollars and Sense” which relates to breaking financial news; and “CNN Headline News” which relates to breaking current events.
  • the recognizer when it encounters a match, creates an entry into the database that uniquely identifies the data by the content type, date encountered and time encountered. Thus, the recognizer continually compares the textual information from the closed caption feed to the dictionary of words and phrases, as well as the dictionary of significant characters.
  • the user can request on demand information using a voice user interface on a telephone.
  • the delivery agent When the delivery agent is instructed by the user to deliver information to the user, the agent looks for the most recent data block associated with the requested content type. The delivery agent may also send this information to the user at specific times according to a user defined profile.
  • on demand information can be determined from a user defined profile.
  • the user can configure their agent to send them specific information in text format to either a SMS phone or an alphanumeric pager at scheduled times.
  • on demand information can be requested from a web interface.
  • the user of the system can also access the text information stored within the information store via the web page and personalized agentry.
  • FIG. 2A A breakdown of various components contained in a clip of multimedia presentation such as television is shown in FIG. 2A. Since video information is bandwidth intensive, a block 440 containing visual picture information consumes the bulk of the available bandwidth represented in FIG. 2A. A block 442 representing audio information, a block 444 representing closed caption information and a block 446 representing SAP information collectively require less bandwidth than the visual picture information block 480 .
  • the process of FIG. 1 isolates the audio information block 442 and the closed caption information block 444 and reproduces an audio information block 448 and a closed caption information block 450 .
  • FIG. 2B shows a representative multimedia signal such as a television signal containing closed-caption character data.
  • Closed caption data is preceded by a sine wave analogous to color burst (called the “Clock Run-In”) and three “start bits” that are always “0”, “0”, and “1”.
  • the sine wave typically lasts for about seven clock cycles. Two bytes of data, using seven bits, odd parity format, are possible on a given line.
  • Using only Line 21 , Field 1 , of the VBI a delivery rate of about 3600 characters or 500 words per minute can be achieved.
  • FIG. 3 shows a diagram illustrating the operation of an alignment process 440 .
  • the audio information block 448 and the closed caption information block 450 of FIG. 2A are captured (step 442 ).
  • the audio information is isolated (step 444 ).
  • the closed caption information is also separated (step 446 ).
  • the audio information of step 444 and the closed caption information of step 446 may be out of sync.
  • a slight delay generally exists between the audio and closed caption feeds in live shows such as CNN Headline News and ESPN, among others. This artifact is due to real time transcription of these events.
  • the audio information may correspond to a current sentence “This is CNN Headline Sports” while the closed caption still references a portion of the previous sentence.
  • the closed caption is delayed and shifted out of alignment at time 4 (step 448 ).
  • the closed caption information is used as a “flag” for a speech recognizer that will look for a specific block of dialog. For example, the “Headline Sports” lead in and lead out from each sports news segment. This audio marker allows the process of FIG. 1 to automatically capture the audio segment without cropping the beginning or end of it because of the closed caption delay.
  • the closed caption is watched by an agent software which compares the caption text to a defined dictionary of words and/or phrases.
  • the agent looks for specific characters, such as “>>>” delineating the start of a new program segment and/or specific words or phrases such as “CNN Headline Sports,” for example. These specific characters or words and phrases trigger the speech recognizer to watch for that word in the audio information (step 450 ).
  • the speech recognizer After the speech recognizer has been instructed to watch for the appropriate words in the stream of audio information, the speech recognizer searches approximately before and after the time code where it encountered the specific characters, keyword or phrase (step 452 ). In this particular example, the speech recognizer starts to look for audio segments that correspond to the target text stream “CNN Headline Sports” at about time 3 . From step 452 , the speech recognizer then finds the related audio segment at time 1 (step 454 ).
  • the related audio segment may be determined by creating an index entry for the audio file, sending the synchronized audio and text data stream to the information store, restarting the speech recognizer to look for the ending queue and, when that queue is defined, the data block is complete. The data block is then indexed and inserted into the information store. This audio block is cached on the information store for subsequent use.
  • the text associated with the closed caption data can be pushed as a text message to the user's SMS service or alphanumeric pager.
  • the text message can be added to previous news text messages, the current text message may override the previous day's text message to prevent filling the user's mail box with stale, outdated news items.
  • multimedia information such as an audio segment associated with the closed caption data can be pushed as an audio message to the user's telephone or sound board if he or she is using a computer.
  • the text or audio message can be added to previous news messages, the current text or audio message may override the previous day's messages to prevent filling the user's mail box with stale, outdated news items. Moreover, if the user has configured the system to have the news refreshed on a periodic basis such as monthly, weekly, hourly or on demand based on a trigger event, the text block is automatically refreshed.
  • FIG. 4 shows a computer system in accordance with the present invention.
  • Telephone calls from a plain old telephone service (POTS) network are received by a cellular switch 100 .
  • the cellular switch 100 transfers the telephone calls to a plurality of computers 102 and 104 over multiple T 1 line pairs 101 & 103 and 105 & 107 .
  • POTS plain old telephone service
  • the interface to the cellular switch 100 at each of T 1 line pairs 101 & 103 or 105 & 107 receives analog speech signal from the network, bandpass-filters and conditions the signal by the line interface and then applies the signal to a CODER/DECoder (CODEC) circuit.
  • CODEC CODER/DECoder
  • the CODEC filters, samples, and digitizes the inbound analog audio signal and passes this digitized audio signal for subsequent digital signal processing (DSP).
  • DSP digital signal processing
  • the interface expands stored, compressed audio data for playback; adjusts the volume and rate of speed of playback upon application or user request; and generates tones such as DTMF, MF, or any application-defined general-purpose tone as needed.
  • Each of computers 102 and 104 communicates with each other over an active backplane 110 . Additionally, multiple data storage devices 106 and 108 are also connected to the active backplane 110 .
  • the active backplane 110 may be an Ethernet switch such as a CISCO® 2900 switch, available from Cisco Systems, Inc. of San Jose, Calif.
  • the active backplane 110 optimizes the Ethernet network by dividing it into multiple smaller, more efficient collision domains, or segments. Individual segments created by the active backplane 110 can consist of one computer, like a file server, a group of computers, or even the entire business network.
  • the active backplane 110 provides point to point bidirectional communication among many nodes such that each node has a full bandwidth access to the data storage device 106 or 108 .
  • Each of data storage devices 106 and 108 is a network Redundant Array of Independent Disk (RAID) data storage system.
  • Each network RAID data storage system 106 or 108 connects directly to the active backplane 110 .
  • the network RAID data storage system may be a RAID array available from Network Appliances, Inc. which attaches directly to an Ethernet backplane 110 such as a 10 Mbit, a 10 Mbit, or a 1 Gbit Ethernet backplane 110 .
  • the network RAID data storage devices 106 or 108 includes a redundant RAID controller and is fully fault tolerant. As such, the network RAID data storage device 106 or 108 is a collection of disks under hardware or software control such that a single drive failure does not bring the system of FIG. 4 down. The network RAID data storage device 106 or 108 also guards against fan and power failures and provides remote failure notification.
  • Each of data storage devices 106 and 108 may be a RAID-1 system, in which every disk has a mirror image of its data stored on another disk.
  • the data storage devices 106 and 108 may be a RAID-2 or RAID-3 sub-system which stripes user data across a group of data drives (typically four or eight drives per group).
  • the data storage devices 106 and 108 may also be a RAID-4 or RAID-5 sub-system which stripes block (or sometimes groups of blocks) of data and stores the data entirely on an individual disk.
  • the operating system running on each data storage device 106 or 108 is optimized for file operations and has a thread-based non-preemptive multitasking environment.
  • Data requests are transmitted over the active backplane 110 in accordance with a file sharing protocol, which is a special set of rules for communicating between a requester running on the computer 102 or 104 and the network RAID data storage system 106 or 108 so that client applications in a computer may read and write to files on the network RAID data storage system 106 or 108 .
  • applications may request services from server programs in a computer network.
  • an application can access files at a remote server as well as other resources, including printers, mailslots, and named pipes, to read, create, and update files on the remote server.
  • the protocol can also communicate with any server program that is set up to receive a client request.
  • the file sharing protocol includes the Common Internet File System (CIFS) for Microsoft-based systems or the Network File System (NFS) for Unix-based systems.
  • the file sharing protocols may be the Server Message Block (SMB) protocol, which is used over the Internet on top of its TCP/IP protocol or on top of other network protocols such as IPX or NetBEUI.
  • SMB Server Message Block
  • the file sharing protocol supported by the RAID data storage device 106 or 108 provides a locking facility which may be a file locking facility or a byte-range locking facility. The locking facility enhances data integrity for the file sharing environment of FIG. 4. Locking can be used to coordinate concurrent access to a file by multiple applications and users. It can prevent concurrent readers and writers of shared data from reading “stale” data (i.e., data currently in the process of being updated by another application) and/or overwriting each others' updates.
  • file locking the application requests that all other applications/users be denied read access to the file while the application holds the file open. If another application should attempt to try to open the file for reading (or reading and writing) while the file is already open with this deny-mode, the attempt fails.
  • byte-range locking may be used to restrict other applications'access to sections of an open file, usually while the holder of the byte-range lock is intending to read or write the locked section. In certain implementations, byte-range locks can be obtained only on already open files.
  • a Messaging system For interprocessor communications, a Messaging system is provided with a unified mailbox containing message pointer lists and reference counts.
  • the CIFS file system lock and application real-time programming processes such as open, lock, update, close are provided for each application requiring access to the mailbox.
  • Lock management and disk allocation strategy is handled by the CIFS file system running on the RAID controller 106 or 108 . In this way, each application telephony server and electronic mail server can operate on the messaging system directly without the need of a single threaded file or database server. Thus, maximum scalability is supported.
  • the CIFS protocol automatically frees locks if a network connection is lost after some programmable period of time.
  • Any computer 102 or 104 in the system illustrated in FIG. 4 can fail regardless of the locks it has outstanding and not affect the operation of any other computer that wants to operate on the same data.
  • each computer or node in the cluster can operate on the entire file system independent of the state or operation of other nodes.
  • FIG. 5 shows a second computer system with redundant backplanes.
  • speech data from the POTS network is presented to a cellular switch 140 .
  • the cellular switch 140 in turn distributes the incoming calls to a plurality of computers 142 and 144 .
  • the cellular switch 140 communicates with each of computers 142 and 144 over multiple T 1 lines 141 & 143 and 145 & 147 , respectively.
  • the computers 142 and 144 in turn communicate with each other and with other peripheral devices over a redundant backplane with a first active backplane 146 and a second active backplane 148 .
  • the first and second active backplanes 146 and 148 are connected together.
  • Each of the active backplanes 146 and 148 may be an Ethernet switch such as the Cisco 2900 switch.
  • the backplanes 146 and 148 divide the Ethernet network into multiple smaller, more efficient collision domains, or segments.
  • Each backplane 146 or 148 provides point-to-point bidirectional communication among computers 142 - 144 such that each computer has a full bandwidth access to the shared peripherals such as data storage devices 154 and 156 . Instead of transmitting information to every device on the network and creating unnecessary traffic, each backplane 146 or 148 determines the exact destination for the data and transmits it only to the port where that device is attached.
  • Each backplane 146 or 148 can also handle multiple transmissions at one time. In the system illustrated in FIG.
  • the backplane 146 or 148 determines exactly where the data needs to go and creates a temporary connection between the two computers. This connection is a private link that is established only for the time needed to complete the data transmission. Also, unlike shared Ethernet hubs, the backplane 146 or 148 allows multiple simultaneous telephone calls to be handled, thus increasing overall network performance.
  • the backplanes 146 and 148 also provide redundancy in bus access. The use of the redundant backplanes 146 and 148 thus provides scalability for both processor-to-processor communication (as in node-to-node communication) and for processor-to-I/O communication (as in the case of a network-attached RAID device).
  • the processor-to-memory communication bandwidth limitation is obviated by the fact that nodes in a cluster provide processor-to-memory access within the node and do not require LAN access.
  • servers 150 and 152 are Also connected to the active backplanes 146 and 148 .
  • the servers 150 and 152 in turn are connected to the Internet 160 .
  • multiple network data storage devices 154 and 156 are also connected to the active backplanes 146 and 148 to satisfy data requests from computers 142 and 144 or servers 150 and 152 .
  • the dual backplanes 146 and 148 provide scalability and fault tolerance. Further, multiple servers may perform the same task in an active/active scalable manner. By providing a client process with access to a pool of like servers which are load balanced, the response time for each request is optimized.
  • FIG. 6 illustrates the computer 142 in more detail.
  • the computer 142 has a processor 172 which is connected to a processor bus 174 .
  • the processor 172 communicates with a random memory access (RAM) 176 and a read-only memory (ROM) 178 over the CPU bus 174 .
  • the processor 172 communicates with one or more peripheral devices through an I/O controller 180 which is connected to the CPU bus 174 .
  • the I/O controller 180 in turn provides an I/O bus 172 for communication with a display device 184 , a data storage device 186 , a telephone interface card 187 , and a network interface system 188 .
  • the closed caption decoder 183 is connected to a receiver-tuner 182 , which can receive over-the-air signals using an optional antenna 181 or can directly receive programs from a wired connection.
  • the receiver-tuner 182 may receive wireless transmissions directed at a direct broadcast satellite (DBS) receiver, a television receiver or a radio receiver, or alternatively, may receive wired transmissions from sources such as cable or a multimedia source.
  • the multimedia source can be a personal computer, a laser disc player, a videotape player, a digital video disk (DVD) player, and optical disk or a magnetic data storage device such as a hard disk.
  • the multimedia source can be a network source or a remote source connected to the computer of FIG. 6 using an IEEE-1394 bus (FireWire).
  • the closed caption decoder 183 may be a Model PDR-885, which is a Closed Caption Encoder with an integral Decoder for inserting caption and text data on Line 21 of the NTSC signal and which is available from Computer Prompting & Captioning Co. of Rockville, Md. Caption and Text data may be inserted in either field on lines 10 though 25 of the Vertical Blanking Interval (VBI). In addition, data on the incoming video signal may be recovered from either field, lines 10 through 25 of the VBI. This incoming data may be displayed as Open Captions, using the built-in Closed Caption Decoder, and captured by a computer utilizing the PDR-885's serial data port. Each of these functions, Encoding, Decoding, and Data Recovery may be done simultaneously.
  • the telephone interface card 187 may be a D/21H or a D/41H voice board, available from Dialogic Corporation of Parsippany, N.J. and which may support digital SS 7 , digital E&M, digital loop start, digital DID, analog E&M, analog loop start, and analog DID.
  • Each of the two (D/21H) or four (D/41H) loop start interfaces receives speech and telephony signaling information from the telephone network.
  • analog signal from the cellular switch 100 or 140 is bandpass-filtered and conditioned by the line interface and then applied to a CODER/DECoder (CODEC) circuit.
  • CODER/DECoder CODER/DECoder
  • the CODEC filters, samples, and digitizes the inbound analog signal and passes this digitized signal to a digital signal processor (DSP).
  • DSP digital signal processor
  • the DSP uses Automatic Gain Control to compensate for variations in the level of the incoming analog signal, applies an adaptive differential pulse code modulation (ADPCM) or pulse code modulation (PCM) algorithm to compress the signal and save disk storage space.
  • ADPCM adaptive differential pulse code modulation
  • PCM pulse code modulation
  • the DSP also detects the presence of tones—DTMF, MF, or an application defined single-or dual-frequency tone.
  • inbound telephony signaling ring detection and loop current detection
  • the DSP responds to these signals, informs the application of telephony signaling status, and instructs the line interface to transmit outbound signaling (on-hook/off-hook) to the telephone network.
  • the DSP can use different digitizing rates from 24 to 64 kilobits per second (Kb/s) as selected by the application for the best speech quality and most efficient storage.
  • the digitizing rate is selected on a channel-by-channel basis and can be changed each time a record or play function is initiated.
  • Outbound processing is the reverse of inbound processing.
  • the DSP processed speech is transmitted to the host processor for storage purposes.
  • speech information is received from the host processor and sent to the DSP, which converts the file into digitized speech.
  • the DSP sends the digitized speech to the CODEC to be converted into analog speech and then to the line interface for transmission to the telephone network.
  • the DSP expands stored, compressed audio data for playback; adjusts the volume and rate of speed of playback upon application or user request; and generates tones—DTMF, MF, or any application-defined general-purpose tone.
  • the network interface system 188 has a plurality of network interface cards 189 - 190 .
  • the network interface system 188 may be a NETELLIGENT® 10/100 TX PCI INTEL® UTP Controller, available from Compaq Computers Corporation of Houston, Tex.
  • the Netelligent 10/100 unit can operate at 10 or 100 megabits per second (Mb/s) and automatically negotiates to the highest common speed when connected to a network. It has a network fault tolerance feature which enables its Controller to keep an active link by automatically switching to a backup network interface if a network connection fails.
  • Software components executing on the computer of FIG. 6 includes a decoder for separating the audio component and the text component from the signal stream; an audio synthesizer coupled to the decoder for generating an audio pattern representative of the start of the multimedia segment; a pattern recognizer coupled to the decoder and to the audio synthesizer for locating the audio pattern in the audio component; and an aligner coupled to the pattern recognizer and to the decoder for temporally aligning the text with the audio pattern in the audio component.
  • FIG. 7 another computer system is shown.
  • a plurality of computers 200 and 202 and a disk 209 communicate over a dual bus with a first bus 204 and a second bus 206 .
  • a first Ethernet switch 210 is connected to the first bus 204 .
  • the Ethernet switch 210 may be the Cisco 2900 switch.
  • the Ethernet switch 210 in turn is connected to an Internet pool of Web servers 214 . Servers 214 support Web content retrieval, email, database management, and system management.
  • the Ethernet switch 210 is also connected to a first director 216 as well as a second director 220 .
  • the first director 216 in turn communicates with an Internet router 222 .
  • the Internet router 222 is also connected to a hub 226 and to the Internet 160 .
  • a second Ethernet switch 212 is connected to the second bus 206 .
  • the second Ethernet switch 212 which may be the Cisco 2900 switch, is also connected to an Internet Web server 218 and to the second director 220 .
  • the second director 220 in turn is connected to an Internet router 224 which is connected to the internet 160 .
  • the Internet router 224 is also connected to the hub 226 .
  • Directors recognize a Universal Resource Locator (URL) or Internet Protocol (IP) address as being associated with a pool of servers. If a server becomes unavailable, the server request is simply put in a slow poll mode and server requests are not sent to it until it starts responding. Directors provide various load-balancing algorithms to even out the load among a pool of servers. These devices assure high availability and scalability. By using directors 216 and 220 within such a clustered system, provisioning active/active pools of servers can be provided using off the shelf components to assure scalable, load balanced, fault tolerant access of clients to all server resources.
  • URL Universal Resource Locator
  • IP Internet Protocol
  • FIG. 8 shows yet another embodiment of the invention.
  • Incoming data from the POTS network is received by a cellular switch 240 .
  • the cellular switch 240 in turn provides POTS data to computers 242 and 244 over multiple T 1 connections for each computer.
  • Each of computers 242 and 244 in turn is connected to a dual backplane system which is made up of a first active backplane 246 and a second active backplane 248 .
  • a plurality of servers 270 and 280 are also connected to the first active backplane 246 and second active backplane 248 .
  • a plurality of data storage devices 250 and 252 are connected to the first active backplane 246 and the second active backplane 248 .
  • multiple directors 254 and 260 are also connected to the first active backplane 246 and the second active backplane 248 .
  • Each of the directors 254 and 260 is connected to each of routers 256 and 262 .
  • the routers 256 and 262 communicate with each other over a hub 258 .
  • the routers 256 and 262 are also connected to the Internet 160 .
  • the deployment of the dual backplanes 246 and 248 and directors 254 and 260 allows many servers to perform the same task in an active/active scalable manner. By providing a client process with access to a pool of like servers which are load balanced, the system minimizes the response time for each request.
  • FIG. 9 an overview of processes executing on the computer systems described above are shown. These processes include a telephony process 300 which is a collection one or more telephony application modules and an automatic speech recognition (ASR) server module.
  • ASR automatic speech recognition
  • multiple client application modules communicate with one or more automatic speech recognition (ASR) client module, one of which is assigned per communication port.
  • ASR client module in turn communicates with an ASR server which handles the speech recognition task.
  • the ASR server software may be the NUANCE-6® software, available from Nuance Communications of Menlo Park, Calif.
  • the ASR server enables speech recognition applications that allow users to talk to computers as if they were speaking with human agents.
  • the ASR server may utilize linguistic and statistical models to interpret and understand natural human speech to provide high recognition accuracy across a range of devices, languages, applications and vocabularies.
  • the scalable client/server architecture of the system handles call volumes and vocabularies of all sizes while providing high reliability. Additionally, each client module also communicates over a telephony protocol which supports facsimile, text, automatic speech recognition and speech processing.
  • Telephony application servers use the same file store methods as the electronic mail server. Users send and receive voice and fax messages and convert the electronic mail and Web text to speech using the telephony application. Electronic mail is sent to and from an IMAP4 electronic mail server which uses the same message store as the telephony application. In this way, the user is presented with the abstraction of a unified mailbox and advanced Web services both from the phone as well as the desktop.
  • the telephony process 300 communicates with an internal services process 320 via client/server communications.
  • a file sharing protocol handler processes requests from client applications so that they can read and write to files or request services from server programs in the computer network.
  • the file sharing protocol handler is a CIFS handler.
  • CIFS locking strategies system wide resources and queues can be processed locally and in parallel with peer servers such as resource managers, application managers, pager/outdialer managers, and fax managers.
  • the lock technique improves parallelism and scalability while reducing load balancing and fail-over complexity.
  • the CIFS file protocol handler implements requests from one or more application commands such as allocate and lock.
  • the application manager allocates one or more universal ports, eliminates redundant inbound/outbound requests, and allocates and deallocates universal ports to set up various conference bridges as necessary.
  • the application manager also communicates with a fax delivery manager and a pager delivery manager.
  • the fax delivery manager maintains fax queues and is aware of the availability of dedicated and/or dynamically allocated dial-out ports.
  • the fax delivery manager is also responsible for retrying fax delivery if necessary.
  • the fax delivery manager communicates with the CIFS software information regarding the fax queue management as well as the fax retrieval process.
  • the pager delivery manager maintains the pager queues and is aware of the dedicated and/or dynamically allocated dial-out ports. It also manages the retry of the pager delivery process.
  • the internal services module 320 also contains a resource manager which allocates the available conference bridges.
  • the internal service module 320 also has a OA&M (Operations, Administration and Maintenance) manager which supports mailbox, port and disk utilities.
  • the internal services process 320 is a collection of software such as a CIFS application module, an application manager module, a facsimile manager module, a pager and message delivery manager module, a resource manager module, and an operation and administration (OA&M) manager module.
  • the external services process 340 is a collection of services which handle tasks such as Internet Message Access Protocol 4 (IMAP4) /Post Office Protocol 3 (POP3) electronic mail, Short Message System (SMS) server, Web agent server, network and system management module, personal information management/electronic mail synchronization server module, and Web Messaging server module.
  • IMAP4 Internet Message Access Protocol 4
  • POP3 Post Office Protocol 3
  • SMS Short Message System
  • Web agent server Web agent server
  • network and system management module personal information management/electronic mail synchronization server module
  • Web Messaging server module Web Messaging server module.
  • the electronic mail server sends and receives messages using IMAP4 and POP3 SMTP protocols.
  • the electronic mail server uses the same file store methods as the telephony application servers, thus supporting the abstraction of a unified mailbox to the phone as well as to the desktop.
  • the Web server implements the Web user interface for message access, service management, system management, PIM synchronization, and Internet data collection.
  • ACTIVE-X® and JAVA® applets or scripts unified messages can be accessed over the Internet using Web browsers such as Microsoft Internet Explorer and Netscape Navigator.
  • the user can use a Web browser to access his or her calendar, address book, and to-do list.
  • the user can also access the Web server from the Internet to add or change services and configure service parameters. With the help of forms and menus, the user may interact with configuration parameters. Further, limited access to service parameters is supported over the telephone user interface.
  • the Personal Information Manager module manages a user's address book, calendar, and to-do list.
  • An address book may contain entries with fields for name, title, company address, business phone, mobile phone, home phone, fax, electronic mail, Web page address and notes.
  • a calendar may keep track of appointments, with features such as subject/agenda, location, date/time, and appointment length.
  • a to-do list keeps track of things that need to be done.
  • the to-do list entry may provide fields for subject/agenda, priority, due date/time, and reminder notes.
  • the system can page, call or send electronic mail with to-do list content.
  • PIM synchronization over the Web is provided to make information on palm-top computers available to the user at any time anywhere by telephone or Web client access.
  • the Web server is the gateway to handle this synchronization.
  • the external services module 340 contains an IMAP4/POP3® electronic mail server.
  • the server handles electronic mail consolidation and unifies the mailbox access to the Web.
  • the IMAP4/POP3 server 342 communicates using an LDAP (Lightweight Directory Access Protocol) to an X.500 directory server.
  • LDAP Lightweight Directory Access Protocol
  • X.500 is a standard produced by the ISO/ITU defining the protocols and information model for a global directory service that is independent of computing application and network platform.
  • the X.500 standard defines a specification for a rich, global, distributed directory based on hierarchically named information objects (directory entries) that users can browse and search.
  • X.500 uses a model of a set of Directory Servers (DSAs), each holding a portion of the global Directory Information Base (DIB).
  • DSAs co-operate to provide a directory service to user applications in a way which means these applications need not be aware of the location of the information they are accessing.
  • the IMAP4/POP3 server and the X.500 directory server in turn communicates over the Internet 319 . Further, the IMAP/POP3 server communicates with an SMS server. Additionally, the X.500 directory communicates with a personal information management (PIM)-electronic mail synchronization system. The PIM-electronic mail synchronizer in turn communicates with the Internet 319 such that a user can synchronize his PIM and electronic mail over the Internet 319 .
  • PIM personal information management
  • the external services module 340 contains a network management module which acts as an interface to the CIFS message store system.
  • the network management module also communicates over the Internet 319 .
  • a system administrator operating the network management module is provided with secure tools to manage the system over the network. Server management, network management, resource management and mailbox management can be all accessed from the Web.
  • a Web Messaging module can be provided to act as a service provider for various message content presentations.
  • the Web Messaging software also handles user preference configurations stored in the profile for system management and system configuration.
  • the external services module 340 includes a Web agent server which operates in conjunction with a phone agent to process spoken requests from the user for searching or operating applications over the Internet 319 .
  • the phone agent in turn provides commands to a Web methods interpreter.
  • the Web methods interpreter translates the verbal request or command to an Internet 319 search and accesses the Internet 319 appropriately.
  • the user can access Internet 319 data such as stock quotes and weather reports from a cellular phone.
  • the above processes in conjunction with the computer systems described above, operate in a fault-tolerant and scalable manner. Further, a pool of servers may execute these processes in an active/active scalable manner. The directors provide load balancing among the pool of like servers. By providing the above processes with access to the pool of like servers which are load balanced, the system provides fast response time for each request.
  • Each of the above described modules or programs can be implemented in a high level procedural or object-oriented programming language to operate in conjunction with a computer system.
  • the programs can be implemented in assembly or machine language, if desired.
  • the language may be a compiled or interpreted language.
  • Each such computer program can be stored on a storage medium or device (e.g., CD-ROM, hard disk or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the procedures described.
  • a storage medium or device e.g., CD-ROM, hard disk or magnetic diskette
  • the system also may be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner.
  • ATV Advanced Television
  • DTV Digital Television
  • HDTV High-Definition Television
  • SDTV Standard Digital Television
  • ATV supports new caption features such as multiple caption streams (enabling viewers to choose between different languages or different reading levels), a wider range of character sizes, fonts, and colors, and increased flexibility regarding caption placement.
  • multiple language audio streams can be searched and played in response to a user request.

Abstract

Methods and apparatus, including computer program products, for synchronizing segments in a signal stream having an audio component and a text component. In one aspect, a search is made in the text component for a marker text string consisting of at least one word, character or phrase; an audio pattern representative of the marker text string is generated; the audio pattern is located in the audio component; and, the audio and text components are aligned temporally. In another aspect, cue points delineating a start and/or end of a multimedia segment are found. In another aspect, a user can search for a multimedia segment and either read the text feed or listen to the audio feed.

Description

    CROSS REFERENCE TO RELATED APPLICATIONS
  • This application is a continuation application of and claims priority to U.S. application Ser. No. 09/221,334, filed on Dec. 23, 1998, the entire disclosure of which is incorporated by reference.[0001]
  • BACKGROUND
  • The invention relates to searching and retrieving multimedia information. [0002]
  • Rapid advances in computer technology have resulted in an explosion of electronically accessible information. For instance, virtually any information imaginable is available on information sources such as specialized databases and servers. The information can be accessed from the convenience of the user's computer, or may be accessed using a telephone which can relay information and allow their users to respond quickly to various situations. The advent of wireless telephones has further allowed users access to information regardless of the user's proximity to a telephone jack. [0003]
  • The availability of these information sources renders information, formerly hard to manage and retrieve, readily accessible. To date, the advantages associated with the widespread availability of electronic data have not been fully realized due to a variety of reasons. For instance, not all information is stored or available in the same format so that the information can be easily searched. This situation occurs when information is stored variously as video clips, sound files and text files. More particularly, information such as sound cannot be searched using conventional text search commands. Moreover, certain perishable and high value-added information such as news on business, sports, current events and entertainment are best presented in audio-visual form and multimedia form rather than text form. Yet, such sources of audio-visual/multimedia information can be prolific and can strain the processing and data storage capacity of a computer hosting such information. Examples of prolific sources of audio-visual/multimedia information include television feeds, cable feeds, radio feeds, and computer generated multimedia feeds. [0004]
  • SUMMARY OF THE INVENTION
  • In one aspect, a method is disclosed for retrieving a multimedia segment from a signal stream having an audio component and a closed caption component. The method includes separating the audio component and the closed caption component from the signal stream; generating an audio pattern representative of the start of the multimedia segment; locating the audio pattern in the audio component; and temporally aligning the text with the audio pattern in the audio component. [0005]
  • Implementations of the invention include one or more of the following. The method includes retrieving text from the closed caption component; and comparing the text against one or more keywords delimiting the multimedia segment. The keywords may delimit the start and the end of the multimedia segment, or may be stored in a dictionary. The method may compare the text against one or more key characters delimiting the multimedia segment. The key characters may be stored in a dictionary. The method may include generating a concluding audio pattern representative of the end of the multimedia segment; and locating the concluding audio pattern in the audio component. [0006]
  • The method may identify the multimedia segment between the audio patterns. The multimedia segment and text may be indexed in a database. The multimedia segment may be played on-demand. [0007]
  • In another aspect, an apparatus retrieves a multimedia segment from a signal stream, the signal stream having an audio component and a closed caption component. The apparatus includes means for separating the audio component and the closed caption component from the signal stream; means for generating an audio pattern representative of the start of the multimedia segment; means for locating the audio pattern in the audio component; and means for temporally aligning the text with the audio pattern in the audio component. [0008]
  • Implementations of the invention include one or more of the following. The apparatus may include means for retrieving text from the closed caption component; and means for comparing the text against one or more keywords delimiting the multimedia segment. The keywords may delimit the start and the end of the multimedia segment. The keywords may be stored in a dictionary. A means for comparing the text against one or more key characters delimiting the multimedia segment may be provided. The key characters may be stored in a dictionary. The apparatus may include means for generating a concluding audio pattern representative of the end of the multimedia segment; and means for locating the concluding audio pattern in the audio component. A means for identifying the multimedia segment between the audio patterns may be used. The multimedia segment and text may be indexed in a database. The apparatus may include a means for playing the multimedia segment on-demand. [0009]
  • In a third aspect, an apparatus retrieves a multimedia segment from a signal stream, the signal stream having an audio component and a text component. The apparatus includes a decoder for separating the audio component and the text component from the signal stream; an audio synthesizer coupled to the decoder for generating an audio pattern representative of the start of the multimedia segment; a pattern recognizer coupled to the decoder and to the audio synthesizer for locating the audio pattern in the audio component; and an aligner coupled to the pattern recognizer and to the decoder for temporally aligning the text with the audio pattern in the audio component. [0010]
  • Implementations of the invention include one or more of the following. The apparatus may include an extractor for retrieving text from the closed caption component; and a comparator for checking the text against one or more keywords delimiting the multimedia segment. The keywords may delimit the start and the end of the multimedia segment. A dictionary may store the keywords. The text may be compared against one or more key characters delimiting the multimedia segment. A character dictionary may store the key characters. The synthesizer may generate a concluding audio pattern representative of the end of the multimedia segment and the pattern recognizer locates the concluding audio pattern in the audio component. A database may store the multimedia segment between the audio patterns and the text. [0011]
  • In another aspect, a method for retrieving a multimedia segment from a signal stream, the signal stream having an audio component and a text component, includes generating audio patterns representative of the start and the end of the multimedia segment; locating the audio patterns in the audio component; and delimiting a portion of the audio component between the audio patterns as the multimedia segment. [0012]
  • Implementations of this aspect may include temporally aligning the text with the start of the delimited audio component portion; and storing the text and the delimited audio component portion in a database. [0013]
  • In another aspect, a computer system includes a plurality of interdependent processors, each interdependent processor executing an independent operating system image without sharing file system state information. Each interdependent processor has a network access card with a first network connection and a second network connection; and each interdependent processor has a decoder for separating the audio component and the text component from the signal stream; an audio synthesizer coupled to the decoder for generating an audio pattern representative of the start of the multimedia segment; a pattern recognizer coupled to the decoder and to the audio synthesizer for locating the audio pattern in the audio component; an aligner coupled to the pattern recognizer and to the decoder for temporally aligning the text with the audio pattern in the audio component; and a database coupled to the aligner for storing the audio component with the aligned text. Each processor is connected to a first active backplane and to a second active backplane, the second active backplane operating in lieu of the first active backplane in case of a fail-over; and one or more peripherals connected to each of the first and second active backplanes and responsive to data requests transmitted over the first and second active backplanes. [0014]
  • Advantages of the invention include one or more of the following. The invention allows value-added information such as news on business, sports, current events and entertainment to be accessible from the user's computer or telephone. The response to the user's request is a natural sounding playback, in contrast to conventional text-to-speech (TTS) technologies whose outputs are stilted, mechanical and unnatural sounding. Moreover, the invention allows time intensive information, or information that requires the user to listen for extended periods of time, to be delivered smoothly and without interruptions. Further, the invention requires little computational power and is inexpensive to implement. [0015]
  • The invention also provides scalability and fault tolerance in delivering the value-added information. The invention allows many servers to perform the same task in an active/active scalable manner. The invention also supports load balancing among a pool of like servers. By providing a client process with access to a pool of like servers which are load balanced, the invention keeps the response time for each request to a minimum. Thus, the invention supports high data availability, fast access to shared data, and low administrative costs through data consolidation. Additionally, the invention may be built using standard off-the-shelf components to reduce overall system cost.[0016]
  • BRIEF DESCRIPTION OF THE DRAWINGS
  • FIG. 1 is a flow chart illustrating a process for processing a multimedia segment with reference to closed captions. [0017]
  • FIG. 2A is a diagram illustrating components of the multimedia segment. [0018]
  • FIG. 2B is an example of a multimedia signal containing closed-caption character data. [0019]
  • FIG. 3 is a flow chart illustrating a process for aligning the closed-caption character data with a multimedia segment. [0020]
  • FIG. 4 is a block diagram illustrating a first computer system for processing multimedia segments. [0021]
  • FIG. 5 is a block diagram illustrating a second computer system for processing multimedia segments. [0022]
  • FIG. 6 is a block diagram illustrating a computer node of the computer system of the present invention. [0023]
  • FIG. 7 is a block diagram illustrating a third computer system for processing multimedia segments. [0024]
  • FIG. 8 is a block diagram illustrating a fourth computer system for processing multimedia segments. [0025]
  • FIG. 9 is a diagram of software supported by the computer system of the present invention. [0026]
  • DESCRIPTION
  • Referring now to FIG. 1, a [0027] process 400 for finding cue points delineating a start and an end of a multimedia segment is shown. Generally, the process 400 separates an audio component and a closed caption component from the signal stream. An audio pattern representative of the start of the multimedia segment is generated and the audio pattern is located in the audio component. The process then temporally aligns the text with the audio pattern in the audio component.
  • In the embodiment of FIG. 1, the multimedia segment is delivered over a television feed. Alternatively, a radio feed, a cable feed, or a computer generated multimedia feed can be used in lieu of the television feed. [0028]
  • Initially, the television feed is received (step [0029] 402). The television feed includes an audio feed and a text feed. The text feed can be a caption, or a written representation of the audio feed. The caption is similar to a subtitle and is designed to allow hearing-impaired listeners to enjoy the program. In this case, the caption is used to determine the start and the end of the multimedia segment.
  • A signal from the television feed, an example of which is shown in FIG. 2B, is analyzed and closed caption data is extracted (step [0030] 412). Closed caption data is hidden in line 21 of a vertical blanking interval (VBI), which is the part between video frames and includes a number of lines of video. Captions 1 and 2 and texts 1 and 2 are stored in line 21 of the first field in each frame, while the second field's line 21 can be used for captions 3 and 4 and texts 3 and 4.
  • The closed caption text feed is then separated (step [0031] 414). The text feed is compared to a dictionary of key words and phrases (step 416). When a keyword or phrase is found, the keyword or the phrase is sent to a speech recognizer (step 418). The speech recognizer then searches recent audio feeds for the same keyword or phrase (step 420).
  • Concurrently, the audio feed is separated (step [0032] 422). From step 422, the audio feed is provided to the speech recognizer of step 420. The process 400 then determines a start of the audio block, indexes the audio block and sends the audio block to an information store (step 424). Finally, the information store synchronizes the catalogs audio and text blocks (step 426). Steps 412-424 collectively form a block 410 which is processed inside a computer such as the computer of FIG. 6.
  • In sum, the television feed enters the system and is separated into audio and text feeds. The text feed is compared to a dictionary of known marker words and phrases such as “CNN Headlines Sports”, for example. Once a word or phrase has been identified, the word or phrase is sent to the speech recognizer to search recent audio feeds for that word or phrase. When the word or phrase is detected in the audio feed, the audio block is indexed and sent to the information store. The speech recognizer system is then instructed to look for the key word or phrase that signals the end of the audio feed. The text and audio feeds are then synchronized, indexed and stored in the information store awaiting user recall or system use. [0033]
  • In this manner, the process of FIG. 1 tracks the closed caption content while trying to match text to a dictionary of key words and phrases. The matched key words and phrases are extracted and sent to the speech recognizer. Sound information associated with the key words or phrases is compared against a buffered version of the audio feed. The sound feed and text feed are then aligned. The result is a synchronized audio and text streams, even if the original audio and text (closed-caption) information were originally out of synch. Since the text feed is searchable and aligned with the audio feed, the user can search for the item of interest and can either read the text feed or listen to the audio feed. [0034]
  • The speech recognizer of [0035] step 418 is discussed next. This recognizer applies the current text feed from the closed caption information with the two dictionaries mentioned above. The recognizer performs the following functions:
  • Recognize intersections between the closed caption information and the dictionaries; [0036]
  • Process the text key words/phrases and find the corresponding audio block; and [0037]
  • Store the corresponding text key words/phrases and audio block in the information store. [0038]
  • The recognizer looks for an intersection between the current text string and entries in the dictionary of significant words, phrases and characters. When there is an intersection, it creates an open entry state or a close entry state. The open entry state is entered when the intersection exists with the matching record in the information store being closed. The close entry state is entered when the intersection exists with the current record being open. [0039]
  • Pseudo code for the open entry state is as follows: [0040]
  • i. Speech recognizer finds the target word or phrase identified by the search software in the audio feed; [0041]
  • ii. Index entry into the database underlying the information store uniquely identifying this data block; [0042]
  • iii. Record the audio and associated text information into the information store; [0043]
  • iv. Reset the speech recognizer to look for the expected closing entry (or if there is no closing entry that is related to the opening entry, the system looks for indicators such as dead air, commercial break, and another intersection between text feed and dictionary, among others. [0044]
  • Pseudo code for the close entry state is as follows: [0045]
  • i. The audio and related text information is recorded into the information store; [0046]
  • ii. The speech recognizer finds the target word or phrase identified by the search software in the audio feed; and [0047]
  • iii. A closing entry is made into the information store to signify the end of the current data block. [0048]
  • Delivery software that is able to search and retrieve a specific data block by the data block's unique identifier which will reflect the content type, and the date and time of the data block. The delivery agent will then send this data block to the user's phone or their SMS/Alphanumeric pager at the user's request or according to a preset schedule. [0049]
  • The content type of the text information and corresponding audio data is identified by various words/phrases or significant characters recognized in the dictionaries. The dictionary of significant characters include those characters that are used to delineate the closed caption feed such as a triple carat (>>>) to signify the beginning of a story. The word/phrase dictionary includes words and/or phrases commonly used to separate programming segments. Examples from CNN Headline News include “CNN Headline Sports” which relates to breaking sports news; “CNN Dollars and Sense” which relates to breaking financial news; and “CNN Headline News” which relates to breaking current events. The recognizer, when it encounters a match, creates an entry into the database that uniquely identifies the data by the content type, date encountered and time encountered. Thus, the recognizer continually compares the textual information from the closed caption feed to the dictionary of words and phrases, as well as the dictionary of significant characters. [0050]
  • Various information delivery modes are discussed next. The user can request on demand information using a voice user interface on a telephone. When the delivery agent is instructed by the user to deliver information to the user, the agent looks for the most recent data block associated with the requested content type. The delivery agent may also send this information to the user at specific times according to a user defined profile. [0051]
  • When the user verbally requests specific information (i.e. “Read me the sports news ”) the system will search the information store for the most recent data block that corresponds with that definition. When it encounters the correct information, the responsive audio content is streamed to the telephone for the user to listen. [0052]
  • Alternatively, on demand information can be determined from a user defined profile. The user can configure their agent to send them specific information in text format to either a SMS phone or an alphanumeric pager at scheduled times. Further, on demand information can be requested from a web interface. The user of the system can also access the text information stored within the information store via the web page and personalized agentry. [0053]
  • A breakdown of various components contained in a clip of multimedia presentation such as television is shown in FIG. 2A. Since video information is bandwidth intensive, a [0054] block 440 containing visual picture information consumes the bulk of the available bandwidth represented in FIG. 2A. A block 442 representing audio information, a block 444 representing closed caption information and a block 446 representing SAP information collectively require less bandwidth than the visual picture information block 480.
  • As shown at the bottom of FIG. 2A, the process of FIG. 1 isolates the audio information block [0055] 442 and the closed caption information block 444 and reproduces an audio information block 448 and a closed caption information block 450.
  • FIG. 2B shows a representative multimedia signal such as a television signal containing closed-caption character data. Closed caption data is preceded by a sine wave analogous to color burst (called the “Clock Run-In”) and three “start bits” that are always “0”, “0”, and “1”. The sine wave typically lasts for about seven clock cycles. Two bytes of data, using seven bits, odd parity format, are possible on a given line. Using only Line [0056] 21, Field 1, of the VBI, a delivery rate of about 3600 characters or 500 words per minute can be achieved.
  • FIG. 3 shows a diagram illustrating the operation of an [0057] alignment process 440. Initially, the audio information block 448 and the closed caption information block 450 of FIG. 2A are captured (step 442). Next, the audio information is isolated (step 444). The closed caption information is also separated (step 446). At this stage, the audio information of step 444 and the closed caption information of step 446 may be out of sync. A slight delay generally exists between the audio and closed caption feeds in live shows such as CNN Headline News and ESPN, among others. This artifact is due to real time transcription of these events. In this example, the audio information may correspond to a current sentence “This is CNN Headline Sports” while the closed caption still references a portion of the previous sentence. Thus, the closed caption is delayed and shifted out of alignment at time 4 (step 448).
  • To help compensate for this delay between the audio and text feeds, the closed caption information is used as a “flag” for a speech recognizer that will look for a specific block of dialog. For example, the “Headline Sports” lead in and lead out from each sports news segment. This audio marker allows the process of FIG. 1 to automatically capture the audio segment without cropping the beginning or end of it because of the closed caption delay. [0058]
  • The closed caption is watched by an agent software which compares the caption text to a defined dictionary of words and/or phrases. The agent looks for specific characters, such as “>>>” delineating the start of a new program segment and/or specific words or phrases such as “CNN Headline Sports,” for example. These specific characters or words and phrases trigger the speech recognizer to watch for that word in the audio information (step [0059] 450).
  • After the speech recognizer has been instructed to watch for the appropriate words in the stream of audio information, the speech recognizer searches approximately before and after the time code where it encountered the specific characters, keyword or phrase (step [0060] 452). In this particular example, the speech recognizer starts to look for audio segments that correspond to the target text stream “CNN Headline Sports” at about time 3. From step 452, the speech recognizer then finds the related audio segment at time 1 (step 454).
  • The related audio segment may be determined by creating an index entry for the audio file, sending the synchronized audio and text data stream to the information store, restarting the speech recognizer to look for the ending queue and, when that queue is defined, the data block is complete. The data block is then indexed and inserted into the information store. This audio block is cached on the information store for subsequent use. [0061]
  • Generally, when the user requests news to be delivered to him or her for reading, the text associated with the closed caption data can be pushed as a text message to the user's SMS service or alphanumeric pager. Although the text message can be added to previous news text messages, the current text message may override the previous day's text message to prevent filling the user's mail box with stale, outdated news items. [0062]
  • Alternatively, when the user requests news to be played for him or her, multimedia information such as an audio segment associated with the closed caption data can be pushed as an audio message to the user's telephone or sound board if he or she is using a computer. [0063]
  • Although the text or audio message can be added to previous news messages, the current text or audio message may override the previous day's messages to prevent filling the user's mail box with stale, outdated news items. Moreover, if the user has configured the system to have the news refreshed on a periodic basis such as monthly, weekly, hourly or on demand based on a trigger event, the text block is automatically refreshed. [0064]
  • FIG. 4 shows a computer system in accordance with the present invention. Telephone calls from a plain old telephone service (POTS) network are received by a [0065] cellular switch 100. The cellular switch 100 transfers the telephone calls to a plurality of computers 102 and 104 over multiple T1 line pairs 101 &103 and 105 &107.
  • The interface to the [0066] cellular switch 100 at each of T1 line pairs 101 &103 or 105 &107 receives analog speech signal from the network, bandpass-filters and conditions the signal by the line interface and then applies the signal to a CODER/DECoder (CODEC) circuit. The CODEC filters, samples, and digitizes the inbound analog audio signal and passes this digitized audio signal for subsequent digital signal processing (DSP). For outbound data, the interface expands stored, compressed audio data for playback; adjusts the volume and rate of speed of playback upon application or user request; and generates tones such as DTMF, MF, or any application-defined general-purpose tone as needed.
  • Each of [0067] computers 102 and 104 communicates with each other over an active backplane 110. Additionally, multiple data storage devices 106 and 108 are also connected to the active backplane 110. The active backplane 110 may be an Ethernet switch such as a CISCO® 2900 switch, available from Cisco Systems, Inc. of San Jose, Calif. The active backplane 110 optimizes the Ethernet network by dividing it into multiple smaller, more efficient collision domains, or segments. Individual segments created by the active backplane 110 can consist of one computer, like a file server, a group of computers, or even the entire business network. The active backplane 110 provides point to point bidirectional communication among many nodes such that each node has a full bandwidth access to the data storage device 106 or 108. Each of data storage devices 106 and 108 is a network Redundant Array of Independent Disk (RAID) data storage system. Each network RAID data storage system 106 or 108 connects directly to the active backplane 110. The network RAID data storage system may be a RAID array available from Network Appliances, Inc. which attaches directly to an Ethernet backplane 110 such as a 10 Mbit, a 10 Mbit, or a 1 Gbit Ethernet backplane 110.
  • The network RAID [0068] data storage devices 106 or 108 includes a redundant RAID controller and is fully fault tolerant. As such, the network RAID data storage device 106 or 108 is a collection of disks under hardware or software control such that a single drive failure does not bring the system of FIG. 4 down. The network RAID data storage device 106 or 108 also guards against fan and power failures and provides remote failure notification.
  • Each of [0069] data storage devices 106 and 108 may be a RAID-1 system, in which every disk has a mirror image of its data stored on another disk. Alternatively, the data storage devices 106 and 108 may be a RAID-2 or RAID-3 sub-system which stripes user data across a group of data drives (typically four or eight drives per group). The data storage devices 106 and 108 may also be a RAID-4 or RAID-5 sub-system which stripes block (or sometimes groups of blocks) of data and stores the data entirely on an individual disk. Additionally, the operating system running on each data storage device 106 or 108 is optimized for file operations and has a thread-based non-preemptive multitasking environment. Data requests are transmitted over the active backplane 110 in accordance with a file sharing protocol, which is a special set of rules for communicating between a requester running on the computer 102 or 104 and the network RAID data storage system 106 or 108 so that client applications in a computer may read and write to files on the network RAID data storage system 106 or 108. Further, applications may request services from server programs in a computer network. Through the file sharing protocol, an application can access files at a remote server as well as other resources, including printers, mailslots, and named pipes, to read, create, and update files on the remote server. The protocol can also communicate with any server program that is set up to receive a client request.
  • The file sharing protocol includes the Common Internet File System (CIFS) for Microsoft-based systems or the Network File System (NFS) for Unix-based systems. Alternatively, the file sharing protocols may be the Server Message Block (SMB) protocol, which is used over the Internet on top of its TCP/IP protocol or on top of other network protocols such as IPX or NetBEUI. The file sharing protocol supported by the RAID [0070] data storage device 106 or 108 provides a locking facility which may be a file locking facility or a byte-range locking facility. The locking facility enhances data integrity for the file sharing environment of FIG. 4. Locking can be used to coordinate concurrent access to a file by multiple applications and users. It can prevent concurrent readers and writers of shared data from reading “stale” data (i.e., data currently in the process of being updated by another application) and/or overwriting each others' updates.
  • In file locking, the application requests that all other applications/users be denied read access to the file while the application holds the file open. If another application should attempt to try to open the file for reading (or reading and writing) while the file is already open with this deny-mode, the attempt fails. Alternatively, byte-range locking may be used to restrict other applications'access to sections of an open file, usually while the holder of the byte-range lock is intending to read or write the locked section. In certain implementations, byte-range locks can be obtained only on already open files. [0071]
  • For interprocessor communications, a Messaging system is provided with a unified mailbox containing message pointer lists and reference counts. The CIFS file system lock and application real-time programming processes such as open, lock, update, close are provided for each application requiring access to the mailbox. Lock management and disk allocation strategy is handled by the CIFS file system running on the [0072] RAID controller 106 or 108. In this way, each application telephony server and electronic mail server can operate on the messaging system directly without the need of a single threaded file or database server. Thus, maximum scalability is supported.
  • Moreover, the CIFS protocol automatically frees locks if a network connection is lost after some programmable period of time. Any [0073] computer 102 or 104 in the system illustrated in FIG. 4 can fail regardless of the locks it has outstanding and not affect the operation of any other computer that wants to operate on the same data. Hence, each computer or node in the cluster can operate on the entire file system independent of the state or operation of other nodes.
  • FIG. 5 shows a second computer system with redundant backplanes. As in the system of FIG. 4, speech data from the POTS network is presented to a [0074] cellular switch 140. The cellular switch 140 in turn distributes the incoming calls to a plurality of computers 142 and 144. The cellular switch 140 communicates with each of computers 142 and 144 over multiple T1 lines 141 &143 and 145 &147, respectively. The computers 142 and 144 in turn communicate with each other and with other peripheral devices over a redundant backplane with a first active backplane 146 and a second active backplane 148. The first and second active backplanes 146 and 148 are connected together.
  • Each of the [0075] active backplanes 146 and 148 may be an Ethernet switch such as the Cisco 2900 switch. The backplanes 146 and 148 divide the Ethernet network into multiple smaller, more efficient collision domains, or segments. Each backplane 146 or 148 provides point-to-point bidirectional communication among computers 142-144 such that each computer has a full bandwidth access to the shared peripherals such as data storage devices 154 and 156. Instead of transmitting information to every device on the network and creating unnecessary traffic, each backplane 146 or 148 determines the exact destination for the data and transmits it only to the port where that device is attached. Each backplane 146 or 148 can also handle multiple transmissions at one time. In the system illustrated in FIG. 5, when one computer sends information destined for another device on the network, the backplane 146 or 148 determines exactly where the data needs to go and creates a temporary connection between the two computers. This connection is a private link that is established only for the time needed to complete the data transmission. Also, unlike shared Ethernet hubs, the backplane 146 or 148 allows multiple simultaneous telephone calls to be handled, thus increasing overall network performance. The backplanes 146 and 148 also provide redundancy in bus access. The use of the redundant backplanes 146 and 148 thus provides scalability for both processor-to-processor communication (as in node-to-node communication) and for processor-to-I/O communication (as in the case of a network-attached RAID device). The processor-to-memory communication bandwidth limitation is obviated by the fact that nodes in a cluster provide processor-to-memory access within the node and do not require LAN access.
  • Also connected to the [0076] active backplanes 146 and 148 are servers 150 and 152. The servers 150 and 152 in turn are connected to the Internet 160. Additionally, multiple network data storage devices 154 and 156 are also connected to the active backplanes 146 and 148 to satisfy data requests from computers 142 and 144 or servers 150 and 152.
  • The [0077] dual backplanes 146 and 148 provide scalability and fault tolerance. Further, multiple servers may perform the same task in an active/active scalable manner. By providing a client process with access to a pool of like servers which are load balanced, the response time for each request is optimized.
  • FIG. 6 illustrates the [0078] computer 142 in more detail. The computer 142 has a processor 172 which is connected to a processor bus 174. The processor 172 communicates with a random memory access (RAM) 176 and a read-only memory (ROM) 178 over the CPU bus 174. Additionally, the processor 172 communicates with one or more peripheral devices through an I/O controller 180 which is connected to the CPU bus 174. The I/O controller 180 in turn provides an I/O bus 172 for communication with a display device 184, a data storage device 186, a telephone interface card 187, and a network interface system 188.
  • Also connected to the I/[0079] O controller 180 is a closed caption decoder 183. The closed caption decoder 183 is connected to a receiver-tuner 182, which can receive over-the-air signals using an optional antenna 181 or can directly receive programs from a wired connection. The receiver-tuner 182 may receive wireless transmissions directed at a direct broadcast satellite (DBS) receiver, a television receiver or a radio receiver, or alternatively, may receive wired transmissions from sources such as cable or a multimedia source. For example, the multimedia source can be a personal computer, a laser disc player, a videotape player, a digital video disk (DVD) player, and optical disk or a magnetic data storage device such as a hard disk. Further, the multimedia source can be a network source or a remote source connected to the computer of FIG. 6 using an IEEE-1394 bus (FireWire). The closed caption decoder 183 may be a Model PDR-885, which is a Closed Caption Encoder with an integral Decoder for inserting caption and text data on Line 21 of the NTSC signal and which is available from Computer Prompting & Captioning Co. of Rockville, Md. Caption and Text data may be inserted in either field on lines 10 though 25 of the Vertical Blanking Interval (VBI). In addition, data on the incoming video signal may be recovered from either field, lines 10 through 25 of the VBI. This incoming data may be displayed as Open Captions, using the built-in Closed Caption Decoder, and captured by a computer utilizing the PDR-885's serial data port. Each of these functions, Encoding, Decoding, and Data Recovery may be done simultaneously.
  • The telephone interface card [0080] 187 may be a D/21H or a D/41H voice board, available from Dialogic Corporation of Parsippany, N.J. and which may support digital SS7, digital E&M, digital loop start, digital DID, analog E&M, analog loop start, and analog DID. Each of the two (D/21H) or four (D/41H) loop start interfaces receives speech and telephony signaling information from the telephone network. Generally, using the interface, analog signal from the cellular switch 100 or 140 is bandpass-filtered and conditioned by the line interface and then applied to a CODER/DECoder (CODEC) circuit. The CODEC filters, samples, and digitizes the inbound analog signal and passes this digitized signal to a digital signal processor (DSP). The DSP then uses Automatic Gain Control to compensate for variations in the level of the incoming analog signal, applies an adaptive differential pulse code modulation (ADPCM) or pulse code modulation (PCM) algorithm to compress the signal and save disk storage space. The DSP also detects the presence of tones—DTMF, MF, or an application defined single-or dual-frequency tone. Thus, inbound telephony signaling (ring detection and loop current detection) are conditioned by the line interface and routed over a control bus to the DSP. The DSP responds to these signals, informs the application of telephony signaling status, and instructs the line interface to transmit outbound signaling (on-hook/off-hook) to the telephone network. When recording speech, the DSP can use different digitizing rates from 24 to 64 kilobits per second (Kb/s) as selected by the application for the best speech quality and most efficient storage. The digitizing rate is selected on a channel-by-channel basis and can be changed each time a record or play function is initiated.
  • Outbound processing is the reverse of inbound processing. The DSP processed speech is transmitted to the host processor for storage purposes. When playing back a stored file, speech information is received from the host processor and sent to the DSP, which converts the file into digitized speech. The DSP sends the digitized speech to the CODEC to be converted into analog speech and then to the line interface for transmission to the telephone network. For outbound data, the DSP expands stored, compressed audio data for playback; adjusts the volume and rate of speed of playback upon application or user request; and generates tones—DTMF, MF, or any application-defined general-purpose tone. [0081]
  • The [0082] network interface system 188 has a plurality of network interface cards 189-190. The network interface system 188 may be a NETELLIGENT® 10/100 TX PCI INTEL® UTP Controller, available from Compaq Computers Corporation of Houston, Tex. The Netelligent 10/100 unit can operate at 10 or 100 megabits per second (Mb/s) and automatically negotiates to the highest common speed when connected to a network. It has a network fault tolerance feature which enables its Controller to keep an active link by automatically switching to a backup network interface if a network connection fails.
  • Software components executing on the computer of FIG. 6 includes a decoder for separating the audio component and the text component from the signal stream; an audio synthesizer coupled to the decoder for generating an audio pattern representative of the start of the multimedia segment; a pattern recognizer coupled to the decoder and to the audio synthesizer for locating the audio pattern in the audio component; and an aligner coupled to the pattern recognizer and to the decoder for temporally aligning the text with the audio pattern in the audio component. [0083]
  • In FIG. 7, another computer system is shown. A plurality of [0084] computers 200 and 202 and a disk 209 communicate over a dual bus with a first bus 204 and a second bus 206. Additionally, a first Ethernet switch 210 is connected to the first bus 204. The Ethernet switch 210 may be the Cisco 2900 switch. The Ethernet switch 210 in turn is connected to an Internet pool of Web servers 214. Servers 214 support Web content retrieval, email, database management, and system management. The Ethernet switch 210 is also connected to a first director 216 as well as a second director 220. The first director 216 in turn communicates with an Internet router 222. The Internet router 222 is also connected to a hub 226 and to the Internet 160.
  • Correspondingly, a [0085] second Ethernet switch 212 is connected to the second bus 206. The second Ethernet switch 212, which may be the Cisco 2900 switch, is also connected to an Internet Web server 218 and to the second director 220. The second director 220 in turn is connected to an Internet router 224 which is connected to the internet 160. The Internet router 224 is also connected to the hub 226.
  • Directors recognize a Universal Resource Locator (URL) or Internet Protocol (IP) address as being associated with a pool of servers. If a server becomes unavailable, the server request is simply put in a slow poll mode and server requests are not sent to it until it starts responding. Directors provide various load-balancing algorithms to even out the load among a pool of servers. These devices assure high availability and scalability. By using [0086] directors 216 and 220 within such a clustered system, provisioning active/active pools of servers can be provided using off the shelf components to assure scalable, load balanced, fault tolerant access of clients to all server resources.
  • FIG. 8 shows yet another embodiment of the invention. Incoming data from the POTS network is received by a [0087] cellular switch 240. The cellular switch 240 in turn provides POTS data to computers 242 and 244 over multiple T1 connections for each computer. Each of computers 242 and 244 in turn is connected to a dual backplane system which is made up of a first active backplane 246 and a second active backplane 248. A plurality of servers 270 and 280 are also connected to the first active backplane 246 and second active backplane 248. Similarly, a plurality of data storage devices 250 and 252 are connected to the first active backplane 246 and the second active backplane 248.
  • Additionally, [0088] multiple directors 254 and 260 are also connected to the first active backplane 246 and the second active backplane 248. Each of the directors 254 and 260 is connected to each of routers 256 and 262. The routers 256 and 262 communicate with each other over a hub 258. The routers 256 and 262 are also connected to the Internet 160.
  • The deployment of the [0089] dual backplanes 246 and 248 and directors 254 and 260 allows many servers to perform the same task in an active/active scalable manner. By providing a client process with access to a pool of like servers which are load balanced, the system minimizes the response time for each request.
  • Turning now to FIG. 9, an overview of processes executing on the computer systems described above are shown. These processes include a [0090] telephony process 300 which is a collection one or more telephony application modules and an automatic speech recognition (ASR) server module. In the telephony process 300, multiple client application modules communicate with one or more automatic speech recognition (ASR) client module, one of which is assigned per communication port. The ASR client module in turn communicates with an ASR server which handles the speech recognition task. The ASR server software may be the NUANCE-6® software, available from Nuance Communications of Menlo Park, Calif. The ASR server enables speech recognition applications that allow users to talk to computers as if they were speaking with human agents. The ASR server may utilize linguistic and statistical models to interpret and understand natural human speech to provide high recognition accuracy across a range of devices, languages, applications and vocabularies. The scalable client/server architecture of the system handles call volumes and vocabularies of all sizes while providing high reliability. Additionally, each client module also communicates over a telephony protocol which supports facsimile, text, automatic speech recognition and speech processing.
  • Telephony application servers use the same file store methods as the electronic mail server. Users send and receive voice and fax messages and convert the electronic mail and Web text to speech using the telephony application. Electronic mail is sent to and from an IMAP4 electronic mail server which uses the same message store as the telephony application. In this way, the user is presented with the abstraction of a unified mailbox and advanced Web services both from the phone as well as the desktop. [0091]
  • When a server for telephony application goes down, all the ports in that node are lost. Since no shared services are implemented in the application servers, no fail-over is required. All ongoing calls on that server are dropped. The [0092] bus 204 or 206 will route new calls to the remaining telephony application servers.
  • The [0093] telephony process 300 communicates with an internal services process 320 via client/server communications. In the internal services module 320, a file sharing protocol handler processes requests from client applications so that they can read and write to files or request services from server programs in the computer network. In this case, the file sharing protocol handler is a CIFS handler. By using CIFS locking strategies, system wide resources and queues can be processed locally and in parallel with peer servers such as resource managers, application managers, pager/outdialer managers, and fax managers. The lock technique improves parallelism and scalability while reducing load balancing and fail-over complexity.
  • The CIFS file protocol handler implements requests from one or more application commands such as allocate and lock. The application manager allocates one or more universal ports, eliminates redundant inbound/outbound requests, and allocates and deallocates universal ports to set up various conference bridges as necessary. [0094]
  • The application manager also communicates with a fax delivery manager and a pager delivery manager. The fax delivery manager maintains fax queues and is aware of the availability of dedicated and/or dynamically allocated dial-out ports. The fax delivery manager is also responsible for retrying fax delivery if necessary. The fax delivery manager communicates with the CIFS software information regarding the fax queue management as well as the fax retrieval process. The pager delivery manager maintains the pager queues and is aware of the dedicated and/or dynamically allocated dial-out ports. It also manages the retry of the pager delivery process. The [0095] internal services module 320 also contains a resource manager which allocates the available conference bridges. The internal service module 320 also has a OA&M (Operations, Administration and Maintenance) manager which supports mailbox, port and disk utilities.
  • The [0096] internal services process 320 is a collection of software such as a CIFS application module, an application manager module, a facsimile manager module, a pager and message delivery manager module, a resource manager module, and an operation and administration (OA&M) manager module. The external services process 340 is a collection of services which handle tasks such as Internet Message Access Protocol 4 (IMAP4) /Post Office Protocol 3 (POP3) electronic mail, Short Message System (SMS) server, Web agent server, network and system management module, personal information management/electronic mail synchronization server module, and Web Messaging server module.
  • The electronic mail server sends and receives messages using IMAP4 and POP3 SMTP protocols. The electronic mail server uses the same file store methods as the telephony application servers, thus supporting the abstraction of a unified mailbox to the phone as well as to the desktop. The Web server implements the Web user interface for message access, service management, system management, PIM synchronization, and Internet data collection. Through ACTIVE-X® and JAVA® applets or scripts, unified messages can be accessed over the Internet using Web browsers such as Microsoft Internet Explorer and Netscape Navigator. The user can use a Web browser to access his or her calendar, address book, and to-do list. The user can also access the Web server from the Internet to add or change services and configure service parameters. With the help of forms and menus, the user may interact with configuration parameters. Further, limited access to service parameters is supported over the telephone user interface. [0097]
  • The Personal Information Manager module manages a user's address book, calendar, and to-do list. An address book may contain entries with fields for name, title, company address, business phone, mobile phone, home phone, fax, electronic mail, Web page address and notes. A calendar may keep track of appointments, with features such as subject/agenda, location, date/time, and appointment length. A to-do list keeps track of things that need to be done. The to-do list entry may provide fields for subject/agenda, priority, due date/time, and reminder notes. The system can page, call or send electronic mail with to-do list content. PIM synchronization over the Web is provided to make information on palm-top computers available to the user at any time anywhere by telephone or Web client access. The Web server is the gateway to handle this synchronization. [0098]
  • The [0099] external services module 340 contains an IMAP4/POP3® electronic mail server. The server handles electronic mail consolidation and unifies the mailbox access to the Web. The IMAP4/POP3 server 342 communicates using an LDAP (Lightweight Directory Access Protocol) to an X.500 directory server. LDAP is a standard capable of providing open access to directory services on the Internet, as well as integrating directories and providing a global directory service. X.500 is a standard produced by the ISO/ITU defining the protocols and information model for a global directory service that is independent of computing application and network platform. The X.500 standard defines a specification for a rich, global, distributed directory based on hierarchically named information objects (directory entries) that users can browse and search. X.500 uses a model of a set of Directory Servers (DSAs), each holding a portion of the global Directory Information Base (DIB). The DSAs co-operate to provide a directory service to user applications in a way which means these applications need not be aware of the location of the information they are accessing.
  • The IMAP4/POP3 server and the X.500 directory server in turn communicates over the Internet [0100] 319. Further, the IMAP/POP3 server communicates with an SMS server. Additionally, the X.500 directory communicates with a personal information management (PIM)-electronic mail synchronization system. The PIM-electronic mail synchronizer in turn communicates with the Internet 319 such that a user can synchronize his PIM and electronic mail over the Internet 319.
  • Additionally, the [0101] external services module 340 contains a network management module which acts as an interface to the CIFS message store system. The network management module also communicates over the Internet 319. A system administrator operating the network management module is provided with secure tools to manage the system over the network. Server management, network management, resource management and mailbox management can be all accessed from the Web.
  • A Web Messaging module can be provided to act as a service provider for various message content presentations. The Web Messaging software also handles user preference configurations stored in the profile for system management and system configuration. Further, the [0102] external services module 340 includes a Web agent server which operates in conjunction with a phone agent to process spoken requests from the user for searching or operating applications over the Internet 319. The phone agent in turn provides commands to a Web methods interpreter. The Web methods interpreter translates the verbal request or command to an Internet 319 search and accesses the Internet 319 appropriately. The user can access Internet 319 data such as stock quotes and weather reports from a cellular phone.
  • The above processes, in conjunction with the computer systems described above, operate in a fault-tolerant and scalable manner. Further, a pool of servers may execute these processes in an active/active scalable manner. The directors provide load balancing among the pool of like servers. By providing the above processes with access to the pool of like servers which are load balanced, the system provides fast response time for each request. [0103]
  • Each of the above described modules or programs can be implemented in a high level procedural or object-oriented programming language to operate in conjunction with a computer system. However, the programs can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language. [0104]
  • Each such computer program can be stored on a storage medium or device (e.g., CD-ROM, hard disk or magnetic diskette) that is readable by a general or special purpose programmable computer for configuring and operating the computer when the storage medium or device is read by the computer to perform the procedures described. The system also may be implemented as a computer-readable storage medium, configured with a computer program, where the storage medium so configured causes a computer to operate in a specific and predefined manner. [0105]
  • Although a computer system with one or more active backplanes have been disclosed, other suitable network architectures may be used. Further, although the present invention has been described with reference to analog television, the invention will also work with Advanced Television (ATV), also known as Digital Television (DTV). ATV, which incorporates the technologies known as High-Definition Television (HDTV) and Standard Digital Television (SDTV). ATV supports new caption features such as multiple caption streams (enabling viewers to choose between different languages or different reading levels), a wider range of character sizes, fonts, and colors, and increased flexibility regarding caption placement. In such an embodiment, multiple language audio streams can be searched and played in response to a user request. [0106]
  • While the invention has been shown and described with reference to particular embodiments, those skilled in the art will understand that the above and other changes in form and detail may be made without departing from the spirit and scope of the following claims.[0107]

Claims (1)

What is claimed is:
1. A method for synchronizing a multimedia segment of a signal stream, the signal stream having an audio component and a closed caption text component, the method comprising:
locating in the text component a marker text string, the marker text string being one of a set of text strings, each text string in the set made up of at least one word, phrase, or character;
generating an audio pattern representative of the located marker text string;
locating the audio pattern in the audio component; and
temporally aligning the closed caption text component with the audio pattern in the audio component.
US09/841,327 1998-12-23 2001-04-23 Synchronizing audio and text of multimedia segments Abandoned US20020055950A1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US09/841,327 US20020055950A1 (en) 1998-12-23 2001-04-23 Synchronizing audio and text of multimedia segments

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US09/221,334 US6243676B1 (en) 1998-12-23 1998-12-23 Searching and retrieving multimedia information
US09/841,327 US20020055950A1 (en) 1998-12-23 2001-04-23 Synchronizing audio and text of multimedia segments

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US09/221,334 Continuation US6243676B1 (en) 1998-12-23 1998-12-23 Searching and retrieving multimedia information

Publications (1)

Publication Number Publication Date
US20020055950A1 true US20020055950A1 (en) 2002-05-09

Family

ID=22827382

Family Applications (2)

Application Number Title Priority Date Filing Date
US09/221,334 Expired - Lifetime US6243676B1 (en) 1998-12-23 1998-12-23 Searching and retrieving multimedia information
US09/841,327 Abandoned US20020055950A1 (en) 1998-12-23 2001-04-23 Synchronizing audio and text of multimedia segments

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US09/221,334 Expired - Lifetime US6243676B1 (en) 1998-12-23 1998-12-23 Searching and retrieving multimedia information

Country Status (1)

Country Link
US (2) US6243676B1 (en)

Cited By (63)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030018662A1 (en) * 2001-07-19 2003-01-23 Sheng Li Synchronizing multimedia data
US20030177171A1 (en) * 2002-01-22 2003-09-18 Brown Bruce Loring Electronic mail retrieval
US20040044532A1 (en) * 2002-09-03 2004-03-04 International Business Machines Corporation System and method for remote audio caption visualizations
US6717690B1 (en) * 1999-11-29 2004-04-06 Xerox Corporation Method and apparatus for managing job contention for system resources in an electrographic reproduction system where images are multibanded
US6718327B1 (en) * 2001-08-31 2004-04-06 Openwave Systems Inc. Fault-tolerant queue with autonomous client operation
US6762857B1 (en) * 1999-11-29 2004-07-13 Xerox Corporation Method and apparatus to enable processing multiple capabilities for a sub-job when using a set of commonly shared resources
WO2004093059A1 (en) * 2003-04-18 2004-10-28 Unisay Sdn. Bhd. Phoneme extraction system
US20050027522A1 (en) * 2003-07-30 2005-02-03 Koichi Yamamoto Speech recognition method and apparatus therefor
US20050080631A1 (en) * 2003-08-15 2005-04-14 Kazuhiko Abe Information processing apparatus and method therefor
US20050114133A1 (en) * 2003-08-22 2005-05-26 Lawrence Mark System for and method of automated quality monitoring
US20070117544A1 (en) * 2003-04-22 2007-05-24 Spinvox Limited Method of providing voicemails to a wireless information device
US20070127688A1 (en) * 2006-02-10 2007-06-07 Spinvox Limited Mass-Scale, User-Independent, Device-Independent Voice Messaging System
US20070188657A1 (en) * 2006-02-15 2007-08-16 Basson Sara H Synchronizing method and system
US20080005656A1 (en) * 2006-06-28 2008-01-03 Shu Fan Stephen Pang Apparatus, method, and file format for text with synchronized audio
US20080049908A1 (en) * 2006-02-10 2008-02-28 Spinvox Limited Mass-Scale, User-Independent, Device-Independent Voice Messaging System
US20080235384A1 (en) * 2007-03-20 2008-09-25 Microsoft Corporation Web service for coordinating actions of clients
US7685358B1 (en) * 2003-03-03 2010-03-23 Netapp, Inc. System and method for coordinating cluster state information
US20100073559A1 (en) * 2008-09-22 2010-03-25 Basson Sara H Verbal description method and system
US20110069230A1 (en) * 2009-09-22 2011-03-24 Caption Colorado L.L.C. Caption and/or Metadata Synchronization for Replay of Previously or Simultaneously Recorded Live Programs
US20110125499A1 (en) * 2009-11-24 2011-05-26 Nexidia Inc. Speech recognition
US8055503B2 (en) 2002-10-18 2011-11-08 Siemens Enterprise Communications, Inc. Methods and apparatus for audio data analysis and data mining using speech recognition
US20120143606A1 (en) * 2010-12-01 2012-06-07 At&T Intellectual Property I, L.P. Method and system for testing closed caption content of video assets
US20130101217A1 (en) * 2011-10-25 2013-04-25 Samsung Electronics Co., Ltd. Method and apparatus for making personalized contents
US8442197B1 (en) * 2006-03-30 2013-05-14 Avaya Inc. Telephone-based user interface for participating simultaneously in more than one teleconference
WO2013152682A1 (en) * 2012-04-12 2013-10-17 天脉聚源(北京)传媒科技有限公司 Method for tagging news video subtitles
US8621011B2 (en) 2009-05-12 2013-12-31 Avaya Inc. Treatment of web feeds as work assignment in a contact center
USRE44954E1 (en) * 1996-01-24 2014-06-17 Sony Corporation Audio transmission, recording and reproducing system
US8989713B2 (en) 2007-01-09 2015-03-24 Nuance Communications, Inc. Selection of a link in a received message for speaking reply, which is converted into text form for delivery
US20150278872A1 (en) * 2014-03-29 2015-10-01 Google Technology Holdings LLC Method and Electronic Device for Distributing Advertisements
US20160293160A1 (en) * 2013-04-02 2016-10-06 Igal NIR Automatic Generation Of A Database For Speech Recognition From Video Captions
US9628852B2 (en) 2000-10-23 2017-04-18 Clearplay Inc. Delivery of navigation data for playback of audio and video content
US9741343B1 (en) * 2013-12-19 2017-08-22 Amazon Technologies, Inc. Voice interaction application selection
US11032620B1 (en) * 2020-02-14 2021-06-08 Sling Media Pvt Ltd Methods, systems, and apparatuses to respond to voice requests to play desired video clips in streamed media based on matched close caption and sub-title text
US20210295848A1 (en) * 2018-09-25 2021-09-23 Sonos, Inc. Voice detection optimization based on selected voice assistant service
US11223878B2 (en) * 2017-10-31 2022-01-11 Samsung Electronics Co., Ltd. Electronic device, speech recognition method, and recording medium
US11270123B2 (en) * 2019-10-22 2022-03-08 Palo Alto Research Center Incorporated System and method for generating localized contextual video annotation
US11432045B2 (en) * 2018-02-19 2022-08-30 Samsung Electronics Co., Ltd. Apparatus and system for providing content based on user utterance
US11432043B2 (en) 2004-10-20 2022-08-30 Clearplay, Inc. Media player configured to receive playback filters from alternative storage mediums
US11445266B2 (en) * 2018-09-13 2022-09-13 Ichannel.Io Ltd. System and computerized method for subtitles synchronization of audiovisual content using the human voice detection for synchronization
US20220417588A1 (en) * 2021-06-29 2022-12-29 The Nielsen Company (Us), Llc Methods and apparatus to determine the speed-up of media programs using speech recognition
US11615818B2 (en) 2005-04-18 2023-03-28 Clearplay, Inc. Apparatus, system and method for associating one or more filter files with a particular multimedia presentation
US20230124847A1 (en) * 2021-10-15 2023-04-20 Rovi Guides, Inc. Interactive pronunciation learning system
US20230127120A1 (en) * 2021-10-27 2023-04-27 Microsoft Technology Licensing, Llc Machine learning driven teleprompter
US20230300399A1 (en) * 2022-03-18 2023-09-21 Comcast Cable Communications, Llc Methods and systems for synchronization of closed captions with content output
US11790911B2 (en) 2018-09-28 2023-10-17 Sonos, Inc. Systems and methods for selective wake word detection using neural network models
US11792590B2 (en) 2018-05-25 2023-10-17 Sonos, Inc. Determining and adapting to changes in microphone performance of playback devices
US11790937B2 (en) 2018-09-21 2023-10-17 Sonos, Inc. Voice detection optimization using sound metadata
US11797263B2 (en) 2018-05-10 2023-10-24 Sonos, Inc. Systems and methods for voice-assisted media content selection
US11798553B2 (en) 2019-05-03 2023-10-24 Sonos, Inc. Voice assistant persistence across multiple network microphone devices
US11816393B2 (en) 2017-09-08 2023-11-14 Sonos, Inc. Dynamic computation of system response volume
US11817083B2 (en) 2018-12-13 2023-11-14 Sonos, Inc. Networked microphone devices, systems, and methods of localized arbitration
US11817076B2 (en) 2017-09-28 2023-11-14 Sonos, Inc. Multi-channel acoustic echo cancellation
US11832068B2 (en) 2016-02-22 2023-11-28 Sonos, Inc. Music service selection
US11862161B2 (en) 2019-10-22 2024-01-02 Sonos, Inc. VAS toggle based on device orientation
US11863593B2 (en) 2016-02-22 2024-01-02 Sonos, Inc. Networked microphone device control
US11869503B2 (en) 2019-12-20 2024-01-09 Sonos, Inc. Offline voice control
US11881223B2 (en) 2018-12-07 2024-01-23 Sonos, Inc. Systems and methods of operating media playback systems having multiple voice assistant services
US11881222B2 (en) 2020-05-20 2024-01-23 Sonos, Inc Command keywords with input detection windowing
US11887598B2 (en) 2020-01-07 2024-01-30 Sonos, Inc. Voice verification for media playback
US11893308B2 (en) 2017-09-29 2024-02-06 Sonos, Inc. Media playback system with concurrent voice assistance
US11900937B2 (en) 2017-08-07 2024-02-13 Sonos, Inc. Wake-word detection suppression
US11899519B2 (en) 2018-10-23 2024-02-13 Sonos, Inc. Multiple stage network microphone device with reduced power consumption and processing load
US11934742B2 (en) 2022-12-14 2024-03-19 Sonos, Inc. Playback device supporting concurrent voice assistants

Families Citing this family (147)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6356921B1 (en) * 1998-06-20 2002-03-12 International Business Machines Corporation Framework for progressive hierarchial and adaptive delivery rich media presentations and associated meta data
US6301667B1 (en) * 1998-10-08 2001-10-09 At&T Corporation Method and system for secure network management of high-speed internet access CPE
US6320623B1 (en) * 1998-11-13 2001-11-20 Philips Electronics North America Corporation Method and device for detecting an event in a program of a video and/ or audio signal and for providing the program to a display upon detection of the event
US6473778B1 (en) * 1998-12-24 2002-10-29 At&T Corporation Generating hypermedia documents from transcriptions of television programs using parallel text alignment
US6480819B1 (en) * 1999-02-25 2002-11-12 Matsushita Electric Industrial Co., Ltd. Automatic search of audio channels by matching viewer-spoken words against closed-caption/audio content for interactive television
JP2000278290A (en) * 1999-03-29 2000-10-06 Matsushita Electric Ind Co Ltd Network managing system
US6266094B1 (en) * 1999-06-14 2001-07-24 Medialink Worldwide Incorporated Method and apparatus for the aggregation and selective retrieval of television closed caption word content originating from multiple geographic locations
US7009657B2 (en) * 1999-06-14 2006-03-07 Medialink Worldwide Incorporated Method and system for the automatic collection and conditioning of closed caption text originating from multiple geographic locations
US7518657B2 (en) * 1999-06-14 2009-04-14 Medialink Worldwide Incorporated Method and system for the automatic collection and transmission of closed caption text
US7268823B2 (en) * 1999-06-14 2007-09-11 Medialink Worldwide Incorporated Method and system for the automatic collection and conditioning of closed caption text originating from multiple geographic locations, and resulting databases produced thereby
US6922733B1 (en) * 1999-06-30 2005-07-26 International Business Machines Corporation Method for coordinating visual and speech web browsers
US6519564B1 (en) * 1999-07-01 2003-02-11 Koninklijke Philips Electronics N.V. Content-driven speech-or audio-browser
US6618710B1 (en) 1999-07-07 2003-09-09 International Business Machines Corporation Apparatus and method for intelligent routing of electronic messages to pagers and computers
US6476871B1 (en) * 1999-08-25 2002-11-05 Sharp Laboratories Of America, Inc. Text display on remote device
US6912691B1 (en) * 1999-09-03 2005-06-28 Cisco Technology, Inc. Delivering voice portal services using an XML voice-enabled web server
US6598074B1 (en) * 1999-09-23 2003-07-22 Rocket Network, Inc. System and method for enabling multimedia production collaboration over a network
US6456973B1 (en) * 1999-10-12 2002-09-24 International Business Machines Corp. Task automation user interface with text-to-speech output
US6675205B2 (en) 1999-10-14 2004-01-06 Arcessa, Inc. Peer-to-peer automated anonymous asynchronous file sharing
CN1329861C (en) * 1999-10-28 2007-08-01 佳能株式会社 Pattern matching method and apparatus
US6996775B1 (en) * 1999-10-29 2006-02-07 Verizon Laboratories Inc. Hypervideo: information retrieval using time-related multimedia:
US6757866B1 (en) * 1999-10-29 2004-06-29 Verizon Laboratories Inc. Hyper video: information retrieval using text from multimedia
US6569206B1 (en) * 1999-10-29 2003-05-27 Verizon Laboratories Inc. Facilitation of hypervideo by automatic IR techniques in response to user requests
US7412643B1 (en) * 1999-11-23 2008-08-12 International Business Machines Corporation Method and apparatus for linking representation and realization data
US6978475B1 (en) * 1999-11-24 2005-12-20 Ecable, Llc Method and apparatus for internet TV
US6408306B1 (en) * 1999-12-14 2002-06-18 International Business Machines Corporation Method and system for automated distinguished name lookup
US7386450B1 (en) * 1999-12-14 2008-06-10 International Business Machines Corporation Generating multimedia information from text information using customized dictionaries
US6785234B1 (en) * 1999-12-22 2004-08-31 Cisco Technology, Inc. Method and apparatus for providing user control of audio quality
US6697796B2 (en) * 2000-01-13 2004-02-24 Agere Systems Inc. Voice clip search
US6925436B1 (en) * 2000-01-28 2005-08-02 International Business Machines Corporation Indexing with translation model for feature regularization
US6857008B1 (en) * 2000-04-19 2005-02-15 Cisco Technology, Inc. Arrangement for accessing an IP-based messaging server by telephone for management of stored messages
US20020112081A1 (en) * 2000-05-15 2002-08-15 Armstrong Donald E. Method and system for creating pervasive computing environments
JP2002062889A (en) * 2000-08-14 2002-02-28 Pioneer Electronic Corp Speech synthesizing method
US7349946B2 (en) * 2000-10-02 2008-03-25 Canon Kabushiki Kaisha Information processing system
KR100664012B1 (en) * 2000-10-14 2007-01-03 엘지전자 주식회사 Output language display method for digital television
US7975021B2 (en) 2000-10-23 2011-07-05 Clearplay, Inc. Method and user interface for downloading audio and video content filters to a media player
US6898799B1 (en) * 2000-10-23 2005-05-24 Clearplay, Inc. Multimedia content navigation and playback
JP2002152259A (en) * 2000-11-13 2002-05-24 Yozan Inc Communication terminal and charging processing unit
US20020133520A1 (en) * 2001-03-15 2002-09-19 Matthew Tanner Method of preparing a multimedia recording of a live presentation
US20020138280A1 (en) * 2001-03-23 2002-09-26 Drabo David William Method and system for transcribing recorded information and delivering transcriptions
US6885989B2 (en) * 2001-04-02 2005-04-26 International Business Machines Corporation Method and system for collaborative speech recognition for small-area network
FR2823343B1 (en) * 2001-04-05 2003-07-04 Cit Alcatel DEVICE FOR GENERATING A MULTIMEDIA FILE FOR A TELECOMMUNICATIONS TERMINAL AND ASSOCIATED MULTIMEDIA FILE
US20030163815A1 (en) * 2001-04-06 2003-08-28 Lee Begeja Method and system for personalized multimedia delivery service
US20030088687A1 (en) 2001-12-28 2003-05-08 Lee Begeja Method and apparatus for automatically converting source video into electronic mail messages
US8060906B2 (en) * 2001-04-06 2011-11-15 At&T Intellectual Property Ii, L.P. Method and apparatus for interactively retrieving content related to previous query results
DE10117367B4 (en) * 2001-04-06 2005-08-18 Siemens Ag Method and system for automatically converting text messages into voice messages
US6903779B2 (en) * 2001-05-16 2005-06-07 Yahoo! Inc. Method and system for displaying related components of a media stream that has been transmitted over a computer network
US7020663B2 (en) * 2001-05-30 2006-03-28 George M. Hay System and method for the delivery of electronic books
US7225126B2 (en) 2001-06-12 2007-05-29 At&T Corp. System and method for processing speech files
US20030018657A1 (en) * 2001-07-18 2003-01-23 Imation Corp. Backup of data on a network
US7599434B2 (en) * 2001-09-26 2009-10-06 Reynolds Jodie L System and method for compressing portions of a media signal using different codecs
US7302102B2 (en) * 2001-09-26 2007-11-27 Reynolds Jodie L System and method for dynamically switching quality settings of a codec to maintain a target data rate
US7457358B2 (en) * 2001-09-26 2008-11-25 Interact Devices, Inc. Polymorphic codec system and method
AU2002334720B8 (en) * 2001-09-26 2006-08-10 Interact Devices, Inc. System and method for communicating media signals
US7457359B2 (en) * 2001-09-26 2008-11-25 Mabey Danny L Systems, devices and methods for securely distributing highly-compressed multimedia content
KR100493674B1 (en) * 2001-12-29 2005-06-03 엘지전자 주식회사 Multimedia data searching and browsing system
US20030131362A1 (en) * 2002-01-09 2003-07-10 Koninklijke Philips Electronics N.V. Method and apparatus for multimodal story segmentation for linking multimedia content
US20030142653A1 (en) * 2002-01-25 2003-07-31 Ming-Ru Jiang Internal code control system and method for wireless data download
US7668901B2 (en) * 2002-04-15 2010-02-23 Avid Technology, Inc. Methods and system using a local proxy server to process media data for local area users
EP1530769A1 (en) * 2002-07-31 2005-05-18 Truecontext Corporation Contextual computing system
US7539086B2 (en) * 2002-10-23 2009-05-26 J2 Global Communications, Inc. System and method for the secure, real-time, high accuracy conversion of general-quality speech into text
US6816834B2 (en) * 2002-10-23 2004-11-09 Jon Jaroker System and method for secure real-time high accuracy speech to text conversion of general quality speech
US7716312B2 (en) 2002-11-13 2010-05-11 Avid Technology, Inc. Method and system for transferring large data files over parallel connections
US6980257B2 (en) * 2002-11-18 2005-12-27 Sharp Laboratories Of America, Inc. Plural-source, selectable, combined image/text single-screen display
US7646705B2 (en) * 2003-04-11 2010-01-12 International Business Machines Corporation Minimizing data loss chances during controller switching
CA2536260A1 (en) * 2003-08-26 2005-03-03 Clearplay, Inc. Method and apparatus for controlling play of an audio signal
US7499531B2 (en) * 2003-09-05 2009-03-03 Emc Corporation Method and system for information lifecycle management
US8103873B2 (en) 2003-09-05 2012-01-24 Emc Corporation Method and system for processing auditory communications
US8209185B2 (en) * 2003-09-05 2012-06-26 Emc Corporation Interface for management of auditory communications
US7457396B2 (en) * 2003-09-05 2008-11-25 Emc Corporation Automated call management
EP1692629B1 (en) * 2003-12-05 2011-06-08 Koninklijke Philips Electronics N.V. System & method for integrative analysis of intrinsic and extrinsic audio-visual data
US7792701B2 (en) * 2004-02-13 2010-09-07 International Business Machines Corporation Method and computer program product for providing accessibility services on demand
US20050234961A1 (en) * 2004-04-16 2005-10-20 Pinnacle Systems, Inc. Systems and Methods for providing a proxy for a shared file system
US8180742B2 (en) * 2004-07-01 2012-05-15 Emc Corporation Policy-based information management
US8229904B2 (en) * 2004-07-01 2012-07-24 Emc Corporation Storage pools for information management
US8180743B2 (en) * 2004-07-01 2012-05-15 Emc Corporation Information management
US20060004579A1 (en) * 2004-07-01 2006-01-05 Claudatos Christopher H Flexible video surveillance
US7707037B2 (en) * 2004-07-01 2010-04-27 Emc Corporation Archiving of surveillance data
US9268780B2 (en) * 2004-07-01 2016-02-23 Emc Corporation Content-driven information lifecycle management
US7444287B2 (en) * 2004-07-01 2008-10-28 Emc Corporation Efficient monitoring system and method
US20060004818A1 (en) * 2004-07-01 2006-01-05 Claudatos Christopher H Efficient information management
US8244542B2 (en) * 2004-07-01 2012-08-14 Emc Corporation Video surveillance
US8626514B2 (en) * 2004-08-31 2014-01-07 Emc Corporation Interface for management of multiple auditory communications
US8090844B2 (en) * 2004-10-08 2012-01-03 Truecontext Corporation Content management across shared, mobile file systems
US8799242B2 (en) 2004-10-08 2014-08-05 Truecontext Corporation Distributed scalable policy based content management
JP2006126482A (en) * 2004-10-28 2006-05-18 Seiko Epson Corp Audio data processor
WO2006077481A1 (en) * 2005-01-19 2006-07-27 Truecontext Corporation Policy-driven mobile forms applications
US7676026B1 (en) * 2005-03-08 2010-03-09 Baxtech Asia Pte Ltd Desktop telephony system
KR20060118876A (en) * 2005-05-17 2006-11-24 삼성전자주식회사 Display apparatus and control method thereof
TWI270052B (en) * 2005-08-09 2007-01-01 Delta Electronics Inc System for selecting audio content by using speech recognition and method therefor
US8020102B2 (en) * 2005-08-11 2011-09-13 Enhanced Personal Audiovisual Technology, Llc System and method of adjusting audiovisual content to improve hearing
US8160160B2 (en) * 2005-09-09 2012-04-17 Broadcast International, Inc. Bit-rate reduction for multimedia data streams
US20070139189A1 (en) * 2005-12-05 2007-06-21 Helmig Kevin S Multi-platform monitoring system and method
NO325191B1 (en) * 2005-12-30 2008-02-18 Tandberg Telecom As Sociable multimedia stream
US8872975B2 (en) * 2006-08-08 2014-10-28 Sony Corporation Receiving device, display controlling method, and program
US20080088735A1 (en) * 2006-09-29 2008-04-17 Bryan Biniak Social media platform and method
US20080201158A1 (en) 2007-02-15 2008-08-21 Johnson Mark D System and method for visitation management in a controlled-access environment
US8542802B2 (en) 2007-02-15 2013-09-24 Global Tel*Link Corporation System and method for three-way call detection
US7739596B2 (en) 2007-04-06 2010-06-15 Yahoo! Inc. Method and system for displaying contextual advertisements with media
US8856002B2 (en) * 2007-04-12 2014-10-07 International Business Machines Corporation Distance metrics for universal pattern processing tasks
US9405823B2 (en) * 2007-07-23 2016-08-02 Nuance Communications, Inc. Spoken document retrieval using multiple speech transcription indices
US8831946B2 (en) * 2007-07-23 2014-09-09 Nuance Communications, Inc. Method and system of indexing speech data
US8806320B1 (en) * 2008-07-28 2014-08-12 Cut2It, Inc. System and method for dynamic and automatic synchronization and manipulation of real-time and on-line streaming media
US8165450B2 (en) * 2007-11-19 2012-04-24 Echostar Technologies L.L.C. Methods and apparatus for filtering content in a video stream using text data
US8165451B2 (en) 2007-11-20 2012-04-24 Echostar Technologies L.L.C. Methods and apparatus for displaying information regarding interstitials of a video stream
US8136140B2 (en) * 2007-11-20 2012-03-13 Dish Network L.L.C. Methods and apparatus for generating metadata utilized to filter content from a video stream using text data
US8606085B2 (en) 2008-03-20 2013-12-10 Dish Network L.L.C. Method and apparatus for replacement of audio data in recorded audio/video stream
US8145482B2 (en) * 2008-05-25 2012-03-27 Ezra Daya Enhancing analysis of test key phrases from acoustic sources with key phrase training models
US8156520B2 (en) 2008-05-30 2012-04-10 EchoStar Technologies, L.L.C. Methods and apparatus for presenting substitute content in an audio/video stream using text data
US20090307741A1 (en) * 2008-06-09 2009-12-10 Echostar Technologies L.L.C. Methods and apparatus for dividing an audio/video stream into multiple segments using text data
CN101635804B (en) * 2008-07-23 2011-07-06 晨星软件研发(深圳)有限公司 Subtitle window output method and related device applied to television system
US8259082B2 (en) 2008-09-12 2012-09-04 At&T Intellectual Property I, L.P. Multimodal portable communication interface for accessing video content
US8588579B2 (en) 2008-12-24 2013-11-19 Echostar Technologies L.L.C. Methods and apparatus for filtering and inserting content into a presentation stream using signature data
US8407735B2 (en) 2008-12-24 2013-03-26 Echostar Technologies L.L.C. Methods and apparatus for identifying segments of content in a presentation stream using signature data
US8510771B2 (en) 2008-12-24 2013-08-13 Echostar Technologies L.L.C. Methods and apparatus for filtering content from a presentation stream using signature data
US8630726B2 (en) * 2009-02-12 2014-01-14 Value-Added Communications, Inc. System and method for detecting three-way call circumvention attempts
US9225838B2 (en) 2009-02-12 2015-12-29 Value-Added Communications, Inc. System and method for detecting three-way call circumvention attempts
US8437617B2 (en) 2009-06-17 2013-05-07 Echostar Technologies L.L.C. Method and apparatus for modifying the presentation of content
US20110197224A1 (en) * 2010-02-09 2011-08-11 Echostar Global B.V. Methods and Apparatus For Selecting Advertisements For Output By A Television Receiver Based on Social Network Profile Data
US8934758B2 (en) * 2010-02-09 2015-01-13 Echostar Global B.V. Methods and apparatus for presenting supplemental content in association with recorded content
WO2011140221A1 (en) * 2010-05-04 2011-11-10 Shazam Entertainment Ltd. Methods and systems for synchronizing media
US8468453B2 (en) 2010-08-12 2013-06-18 Echostar Technologies L.L.C. User-selected media content blocking
US8825661B2 (en) 2010-11-30 2014-09-02 International Business Machines Corporation Systems and methods for two stream indexing of audio content
US20120296652A1 (en) * 2011-05-18 2012-11-22 Sony Corporation Obtaining information on audio video program using voice recognition of soundtrack
US20130110832A1 (en) * 2011-10-27 2013-05-02 Microsoft Corporation Techniques to determine network addressing for sharing media files
US9547665B2 (en) 2011-10-27 2017-01-17 Microsoft Technology Licensing, Llc Techniques to determine network storage for sharing media files
US10733151B2 (en) 2011-10-27 2020-08-04 Microsoft Technology Licensing, Llc Techniques to share media files
US8804035B1 (en) * 2012-09-25 2014-08-12 The Directv Group, Inc. Method and system for communicating descriptive data in a television broadcast system
US10395642B1 (en) * 2012-11-19 2019-08-27 Cox Communications, Inc. Caption data fishing
CN103971687B (en) * 2013-02-01 2016-06-29 腾讯科技(深圳)有限公司 Implementation of load balancing in a kind of speech recognition system and device
CN104240710B (en) * 2013-06-06 2019-01-08 腾讯科技(深圳)有限公司 A kind of method, system and the terminal device of information transmission
US11051075B2 (en) 2014-10-03 2021-06-29 Dish Network L.L.C. Systems and methods for providing bookmarking data
US9854329B2 (en) * 2015-02-19 2017-12-26 Tribune Broadcasting Company, Llc Use of a program schedule to modify an electronic dictionary of a closed-captioning generator
US10289677B2 (en) 2015-02-19 2019-05-14 Tribune Broadcasting Company, Llc Systems and methods for using a program schedule to facilitate modifying closed-captioning text
US20180158469A1 (en) * 2015-05-25 2018-06-07 Guangzhou Kugou Computer Technology Co., Ltd. Audio processing method and apparatus, and terminal
CN104978961B (en) * 2015-05-25 2019-10-15 广州酷狗计算机科技有限公司 A kind of audio-frequency processing method, device and terminal
US10572961B2 (en) 2016-03-15 2020-02-25 Global Tel*Link Corporation Detection and prevention of inmate to inmate message relay
US9609121B1 (en) 2016-04-07 2017-03-28 Global Tel*Link Corporation System and method for third party monitoring of voice and video calls
US10341744B2 (en) * 2016-06-30 2019-07-02 Nbcuniversal Media, Llc System and method for controlling related video content based on domain specific language models
US10635828B2 (en) 2016-09-23 2020-04-28 Microsoft Technology Licensing, Llc Tokenized links with granular permissions
US10056083B2 (en) * 2016-10-18 2018-08-21 Yen4Ken, Inc. Method and system for processing multimedia content to dynamically generate text transcript
US10027797B1 (en) 2017-05-10 2018-07-17 Global Tel*Link Corporation Alarm control for inmate call monitoring
US10225396B2 (en) 2017-05-18 2019-03-05 Global Tel*Link Corporation Third party monitoring of a activity within a monitoring platform
US10860786B2 (en) 2017-06-01 2020-12-08 Global Tel*Link Corporation System and method for analyzing and investigating communication data from a controlled environment
US9930088B1 (en) 2017-06-22 2018-03-27 Global Tel*Link Corporation Utilizing VoIP codec negotiation during a controlled environment call
US10885903B1 (en) * 2018-12-10 2021-01-05 Amazon Technologies, Inc. Generating transcription information based on context keywords
US11172269B2 (en) 2020-03-04 2021-11-09 Dish Network L.L.C. Automated commercial content shifting in a video streaming system
CN112599130B (en) * 2020-12-03 2022-08-19 安徽宝信信息科技有限公司 Intelligent conference system based on intelligent screen

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5500919A (en) * 1992-11-18 1996-03-19 Canon Information Systems, Inc. Graphics user interface for controlling text-to-speech conversion
US5555343A (en) * 1992-11-18 1996-09-10 Canon Information Systems, Inc. Text parser for use with a text-to-speech converter
US5703655A (en) * 1995-03-24 1997-12-30 U S West Technologies, Inc. Video programming retrieval using extracted closed caption data which has been partitioned and stored to facilitate a search and retrieval process
US5809471A (en) * 1996-03-07 1998-09-15 Ibm Corporation Retrieval of additional information not found in interactive TV or telephony signal by application using dynamically extracted vocabulary
US5914719A (en) * 1996-12-03 1999-06-22 S3 Incorporated Index and storage system for data provided in the vertical blanking interval
US6263507B1 (en) * 1996-12-05 2001-07-17 Interval Research Corporation Browser for use in navigating a body of information, with particular application to browsing information represented by audiovisual data
US6489979B1 (en) * 1998-10-30 2002-12-03 International Business Machines Corporation Non-computer interface to a database and digital library

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5664060A (en) * 1994-01-25 1997-09-02 Information Storage Devices Message management methods and apparatus
US5737725A (en) * 1996-01-09 1998-04-07 U S West Marketing Resources Group, Inc. Method and system for automatically generating new voice files corresponding to new text from a script
US5774857A (en) * 1996-11-15 1998-06-30 Motorola, Inc. Conversion of communicated speech to text for tranmission as RF modulated base band video

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5500919A (en) * 1992-11-18 1996-03-19 Canon Information Systems, Inc. Graphics user interface for controlling text-to-speech conversion
US5555343A (en) * 1992-11-18 1996-09-10 Canon Information Systems, Inc. Text parser for use with a text-to-speech converter
US5703655A (en) * 1995-03-24 1997-12-30 U S West Technologies, Inc. Video programming retrieval using extracted closed caption data which has been partitioned and stored to facilitate a search and retrieval process
US5809471A (en) * 1996-03-07 1998-09-15 Ibm Corporation Retrieval of additional information not found in interactive TV or telephony signal by application using dynamically extracted vocabulary
US5914719A (en) * 1996-12-03 1999-06-22 S3 Incorporated Index and storage system for data provided in the vertical blanking interval
US6263507B1 (en) * 1996-12-05 2001-07-17 Interval Research Corporation Browser for use in navigating a body of information, with particular application to browsing information represented by audiovisual data
US6489979B1 (en) * 1998-10-30 2002-12-03 International Business Machines Corporation Non-computer interface to a database and digital library

Cited By (98)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
USRE44954E1 (en) * 1996-01-24 2014-06-17 Sony Corporation Audio transmission, recording and reproducing system
US6762857B1 (en) * 1999-11-29 2004-07-13 Xerox Corporation Method and apparatus to enable processing multiple capabilities for a sub-job when using a set of commonly shared resources
US6717690B1 (en) * 1999-11-29 2004-04-06 Xerox Corporation Method and apparatus for managing job contention for system resources in an electrographic reproduction system where images are multibanded
US9628852B2 (en) 2000-10-23 2017-04-18 Clearplay Inc. Delivery of navigation data for playback of audio and video content
US20030018662A1 (en) * 2001-07-19 2003-01-23 Sheng Li Synchronizing multimedia data
US20040167932A1 (en) * 2001-08-31 2004-08-26 Openwave Systems Inc. Fault-tolerant queue with autonomous client operation
US6718327B1 (en) * 2001-08-31 2004-04-06 Openwave Systems Inc. Fault-tolerant queue with autonomous client operation
US7231406B2 (en) 2001-08-31 2007-06-12 Openwave Systems Inc. Fault-tolerant queue with autonomous client operation
US20030177171A1 (en) * 2002-01-22 2003-09-18 Brown Bruce Loring Electronic mail retrieval
US20040044532A1 (en) * 2002-09-03 2004-03-04 International Business Machines Corporation System and method for remote audio caption visualizations
US8055503B2 (en) 2002-10-18 2011-11-08 Siemens Enterprise Communications, Inc. Methods and apparatus for audio data analysis and data mining using speech recognition
US7953924B1 (en) 2003-03-03 2011-05-31 Netapp, Inc. System and method for coordinating cluster state information
US7685358B1 (en) * 2003-03-03 2010-03-23 Netapp, Inc. System and method for coordinating cluster state information
WO2004093059A1 (en) * 2003-04-18 2004-10-28 Unisay Sdn. Bhd. Phoneme extraction system
US8989785B2 (en) 2003-04-22 2015-03-24 Nuance Communications, Inc. Method of providing voicemails to a wireless information device
US20070117544A1 (en) * 2003-04-22 2007-05-24 Spinvox Limited Method of providing voicemails to a wireless information device
US20050027522A1 (en) * 2003-07-30 2005-02-03 Koichi Yamamoto Speech recognition method and apparatus therefor
US20050080631A1 (en) * 2003-08-15 2005-04-14 Kazuhiko Abe Information processing apparatus and method therefor
US20050114133A1 (en) * 2003-08-22 2005-05-26 Lawrence Mark System for and method of automated quality monitoring
US8050921B2 (en) 2003-08-22 2011-11-01 Siemens Enterprise Communications, Inc. System for and method of automated quality monitoring
US7584101B2 (en) * 2003-08-22 2009-09-01 Ser Solutions, Inc. System for and method of automated quality monitoring
US11432043B2 (en) 2004-10-20 2022-08-30 Clearplay, Inc. Media player configured to receive playback filters from alternative storage mediums
US11615818B2 (en) 2005-04-18 2023-03-28 Clearplay, Inc. Apparatus, system and method for associating one or more filter files with a particular multimedia presentation
US20070127688A1 (en) * 2006-02-10 2007-06-07 Spinvox Limited Mass-Scale, User-Independent, Device-Independent Voice Messaging System
US8976944B2 (en) * 2006-02-10 2015-03-10 Nuance Communications, Inc. Mass-scale, user-independent, device-independent voice messaging system
US20080052070A1 (en) * 2006-02-10 2008-02-28 Spinvox Limited Mass-Scale, User-Independent, Device-Independent Voice Messaging System
US20080049908A1 (en) * 2006-02-10 2008-02-28 Spinvox Limited Mass-Scale, User-Independent, Device-Independent Voice Messaging System
US20080049906A1 (en) * 2006-02-10 2008-02-28 Spinvox Limited Mass-Scale, User-Independent, Device-Independent Voice Messaging System
US8903053B2 (en) 2006-02-10 2014-12-02 Nuance Communications, Inc. Mass-scale, user-independent, device-independent voice messaging system
US8934611B2 (en) 2006-02-10 2015-01-13 Nuance Communications, Inc. Mass-scale, user-independent, device-independent voice messaging system
US8953753B2 (en) 2006-02-10 2015-02-10 Nuance Communications, Inc. Mass-scale, user-independent, device-independent voice messaging system
US20080162132A1 (en) * 2006-02-10 2008-07-03 Spinvox Limited Mass-Scale, User-Independent, Device-Independent Voice Messaging System
US9191515B2 (en) 2006-02-10 2015-11-17 Nuance Communications, Inc. Mass-scale, user-independent, device-independent voice messaging system
US20070188657A1 (en) * 2006-02-15 2007-08-16 Basson Sara H Synchronizing method and system
US7913155B2 (en) * 2006-02-15 2011-03-22 International Business Machines Corporation Synchronizing method and system
US8442197B1 (en) * 2006-03-30 2013-05-14 Avaya Inc. Telephone-based user interface for participating simultaneously in more than one teleconference
US20080005656A1 (en) * 2006-06-28 2008-01-03 Shu Fan Stephen Pang Apparatus, method, and file format for text with synchronized audio
US8989713B2 (en) 2007-01-09 2015-03-24 Nuance Communications, Inc. Selection of a link in a received message for speaking reply, which is converted into text form for delivery
US20080235384A1 (en) * 2007-03-20 2008-09-25 Microsoft Corporation Web service for coordinating actions of clients
US7984158B2 (en) * 2007-03-20 2011-07-19 Microsoft Corporation Web service for coordinating actions of clients
US20100073559A1 (en) * 2008-09-22 2010-03-25 Basson Sara H Verbal description method and system
US20120188446A1 (en) * 2008-09-22 2012-07-26 International Business Machines Corporation Verbal description
US8219899B2 (en) * 2008-09-22 2012-07-10 International Business Machines Corporation Verbal description method and system
US8839086B2 (en) * 2008-09-22 2014-09-16 International Business Machines Corporation Verbal description
US8621011B2 (en) 2009-05-12 2013-12-31 Avaya Inc. Treatment of web feeds as work assignment in a contact center
US8707381B2 (en) * 2009-09-22 2014-04-22 Caption Colorado L.L.C. Caption and/or metadata synchronization for replay of previously or simultaneously recorded live programs
US10034028B2 (en) 2009-09-22 2018-07-24 Vitac Corporation Caption and/or metadata synchronization for replay of previously or simultaneously recorded live programs
US20110069230A1 (en) * 2009-09-22 2011-03-24 Caption Colorado L.L.C. Caption and/or Metadata Synchronization for Replay of Previously or Simultaneously Recorded Live Programs
US20110125499A1 (en) * 2009-11-24 2011-05-26 Nexidia Inc. Speech recognition
US9275640B2 (en) * 2009-11-24 2016-03-01 Nexidia Inc. Augmented characterization for speech recognition
US9620118B2 (en) 2010-12-01 2017-04-11 Nuance Communications, Inc. Method and system for testing closed caption content of video assets
US8826354B2 (en) * 2010-12-01 2014-09-02 At&T Intellectual Property I, L.P. Method and system for testing closed caption content of video assets
US20120143606A1 (en) * 2010-12-01 2012-06-07 At&T Intellectual Property I, L.P. Method and system for testing closed caption content of video assets
US8942414B2 (en) * 2011-10-25 2015-01-27 Samsung Electronics Co., Ltd Method and apparatus for making personalized contents
US20130101217A1 (en) * 2011-10-25 2013-04-25 Samsung Electronics Co., Ltd. Method and apparatus for making personalized contents
WO2013152682A1 (en) * 2012-04-12 2013-10-17 天脉聚源(北京)传媒科技有限公司 Method for tagging news video subtitles
US9905221B2 (en) * 2013-04-02 2018-02-27 Igal NIR Automatic generation of a database for speech recognition from video captions
US20160293160A1 (en) * 2013-04-02 2016-10-06 Igal NIR Automatic Generation Of A Database For Speech Recognition From Video Captions
US9741343B1 (en) * 2013-12-19 2017-08-22 Amazon Technologies, Inc. Voice interaction application selection
US20150278872A1 (en) * 2014-03-29 2015-10-01 Google Technology Holdings LLC Method and Electronic Device for Distributing Advertisements
US11863593B2 (en) 2016-02-22 2024-01-02 Sonos, Inc. Networked microphone device control
US11832068B2 (en) 2016-02-22 2023-11-28 Sonos, Inc. Music service selection
US11900937B2 (en) 2017-08-07 2024-02-13 Sonos, Inc. Wake-word detection suppression
US11816393B2 (en) 2017-09-08 2023-11-14 Sonos, Inc. Dynamic computation of system response volume
US11817076B2 (en) 2017-09-28 2023-11-14 Sonos, Inc. Multi-channel acoustic echo cancellation
US11893308B2 (en) 2017-09-29 2024-02-06 Sonos, Inc. Media playback system with concurrent voice assistance
US11223878B2 (en) * 2017-10-31 2022-01-11 Samsung Electronics Co., Ltd. Electronic device, speech recognition method, and recording medium
US11432045B2 (en) * 2018-02-19 2022-08-30 Samsung Electronics Co., Ltd. Apparatus and system for providing content based on user utterance
US11706495B2 (en) * 2018-02-19 2023-07-18 Samsung Electronics Co., Ltd. Apparatus and system for providing content based on user utterance
US11797263B2 (en) 2018-05-10 2023-10-24 Sonos, Inc. Systems and methods for voice-assisted media content selection
US11792590B2 (en) 2018-05-25 2023-10-17 Sonos, Inc. Determining and adapting to changes in microphone performance of playback devices
US11445266B2 (en) * 2018-09-13 2022-09-13 Ichannel.Io Ltd. System and computerized method for subtitles synchronization of audiovisual content using the human voice detection for synchronization
US11790937B2 (en) 2018-09-21 2023-10-17 Sonos, Inc. Voice detection optimization using sound metadata
US20210295848A1 (en) * 2018-09-25 2021-09-23 Sonos, Inc. Voice detection optimization based on selected voice assistant service
US11727936B2 (en) * 2018-09-25 2023-08-15 Sonos, Inc. Voice detection optimization based on selected voice assistant service
US11790911B2 (en) 2018-09-28 2023-10-17 Sonos, Inc. Systems and methods for selective wake word detection using neural network models
US11899519B2 (en) 2018-10-23 2024-02-13 Sonos, Inc. Multiple stage network microphone device with reduced power consumption and processing load
US11881223B2 (en) 2018-12-07 2024-01-23 Sonos, Inc. Systems and methods of operating media playback systems having multiple voice assistant services
US11817083B2 (en) 2018-12-13 2023-11-14 Sonos, Inc. Networked microphone devices, systems, and methods of localized arbitration
US11798553B2 (en) 2019-05-03 2023-10-24 Sonos, Inc. Voice assistant persistence across multiple network microphone devices
US11270123B2 (en) * 2019-10-22 2022-03-08 Palo Alto Research Center Incorporated System and method for generating localized contextual video annotation
US11862161B2 (en) 2019-10-22 2024-01-02 Sonos, Inc. VAS toggle based on device orientation
US11869503B2 (en) 2019-12-20 2024-01-09 Sonos, Inc. Offline voice control
US11887598B2 (en) 2020-01-07 2024-01-30 Sonos, Inc. Voice verification for media playback
US11849193B2 (en) * 2020-02-14 2023-12-19 Dish Network Technologies India Private Limited Methods, systems, and apparatuses to respond to voice requests to play desired video clips in streamed media based on matched close caption and sub-title text
US11032620B1 (en) * 2020-02-14 2021-06-08 Sling Media Pvt Ltd Methods, systems, and apparatuses to respond to voice requests to play desired video clips in streamed media based on matched close caption and sub-title text
US11509969B2 (en) * 2020-02-14 2022-11-22 Dish Network Technologies India Private Limited Methods, systems, and apparatuses to respond to voice requests to play desired video clips in streamed media based on matched close caption and sub-title text
US20230037744A1 (en) * 2020-02-14 2023-02-09 Dish Network Technologies India Private Limited Methods, systems, and apparatuses to respond to voice requests to play desired video clips in streamed media based on matched close caption and sub-title text
US11881222B2 (en) 2020-05-20 2024-01-23 Sonos, Inc Command keywords with input detection windowing
US11683558B2 (en) * 2021-06-29 2023-06-20 The Nielsen Company (Us), Llc Methods and apparatus to determine the speed-up of media programs using speech recognition
US20220417588A1 (en) * 2021-06-29 2022-12-29 The Nielsen Company (Us), Llc Methods and apparatus to determine the speed-up of media programs using speech recognition
US20230124847A1 (en) * 2021-10-15 2023-04-20 Rovi Guides, Inc. Interactive pronunciation learning system
US11736773B2 (en) * 2021-10-15 2023-08-22 Rovi Guides, Inc. Interactive pronunciation learning system
US20230127120A1 (en) * 2021-10-27 2023-04-27 Microsoft Technology Licensing, Llc Machine learning driven teleprompter
US11902690B2 (en) * 2021-10-27 2024-02-13 Microsoft Technology Licensing, Llc Machine learning driven teleprompter
US11785278B1 (en) * 2022-03-18 2023-10-10 Comcast Cable Communications, Llc Methods and systems for synchronization of closed captions with content output
US20230300399A1 (en) * 2022-03-18 2023-09-21 Comcast Cable Communications, Llc Methods and systems for synchronization of closed captions with content output
US11934742B2 (en) 2022-12-14 2024-03-19 Sonos, Inc. Playback device supporting concurrent voice assistants

Also Published As

Publication number Publication date
US6243676B1 (en) 2001-06-05

Similar Documents

Publication Publication Date Title
US6243676B1 (en) Searching and retrieving multimedia information
US6412079B1 (en) Server pool for clustered system
US6230190B1 (en) Shared-everything file storage for clustered system
EP0830779B1 (en) Voice mail system
US6233318B1 (en) System for accessing multimedia mailboxes and messages over the internet and via telephone
US8762853B2 (en) Method and apparatus for annotating a document
US5822537A (en) Multimedia networked system detecting congestion by monitoring buffers' threshold and compensating by reducing video transmittal rate then reducing audio playback rate
US6397345B1 (en) Fault tolerant bus for clustered system
US9171545B2 (en) Browsing and retrieval of full broadcast-quality video
US7286990B1 (en) Universal interface for voice activated access to multiple information providers
US7877774B1 (en) Browsing and retrieval of full broadcast-quality video
US20030140121A1 (en) Method and apparatus for access to, and delivery of, multimedia information
US20020124100A1 (en) Method and apparatus for access to, and delivery of, multimedia information
JP2002366552A (en) Method and system for searching recorded speech and retrieving relevant segment
US8112449B2 (en) Systems and methods for implementing a content object access point
US11721344B2 (en) Automated audio-to-text transcription in multi-device teleconferences
US20050165606A1 (en) System and method for providing a printing capability for a transcription service or multimedia presentation
JP3474130B2 (en) Method for accessing messages stored in a voice mail system via the Internet World Wide Web
WO2001076212A1 (en) Universal interface for voice activated access to multiple information providers

Legal Events

Date Code Title Description
AS Assignment

Owner name: ATMOTION INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ARABESQUE COMMUNICATIONS, INC.;REEL/FRAME:015041/0075

Effective date: 19990709

Owner name: ARABESQUE COMMUNICATIONS, INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:WITTEMAN, BRADLEY J.;REEL/FRAME:015041/0065

Effective date: 19981222

Owner name: OPENWAVE SYSTEMS INC., CALIFORNIA

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:ATMOTION INC.;REEL/FRAME:015041/0052

Effective date: 20010319

STCB Information on status: application discontinuation

Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION