WO1997042761A1

WO1997042761A1 - Videophone system

Info

Publication number: WO1997042761A1
Application number: PCT/US1997/007653
Authority: WO
Inventors: David D. Mccurley; Nicholas E. Broome
Original assignee: The Camelot Corporation
Priority date: 1996-05-06
Filing date: 1997-05-06
Publication date: 1997-11-13
Also published as: AU2829697A

Abstract

A videophone system (10) including a computer (60) with a display screen (61). The system further includes an independent processing unit (84) for compressing data. The system even further includes a camera (21) in communication with the processing unit for taking pictures and outputting video signals to the processing unit (84), and a telephone unit (12, 15) in communication with the processing unit (84) for sending audio signals to and receiving audio signals from the processing unit (84). The processing unit executes compression software which compresses audio and video signals received by the processing unit (84) into compressed signals and outputs compressed signals to the computer (60).

Description

PATENT SPECIFICATION

TITLE: VIDEOPHONE SYSTEM

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to videophones and to methods of processing media data streams. In another aspect, the present invention relates to videophones which operate in association with a personal computer, and to methods of processing data utilizing such videophones and personal computers. In even another aspect, the present invention relates to videophones which operate in association with a personal computer, and which transmit over the internet or a phone line, and to methods of processing data relating thereto.

2. Description of the Related Art

U.S. Patent No. 4,645,872, issued February 24, 1987 to Pressman et al., discloses a videophone network system which includes a special microprocessor controlled control console disposed at each of a plurality of stations, with each station including video and audio inputs and outputs. The system further includes a central controller.

U.S. Patent No. 5,077,784, issued December 31, 1991 to Fujita et al., discloses a visible telephone, which includes a housing in which a video camera is accommodated in a manner that an aperture is directed to the front. The camera is fixed to a rotatable holder which is rotated by an operation of a knob outside the housing.

U.S. Patent No. 5,325,423, issued June 28, 1994 to Lewis, discloses an interactive multimedia communication system stated to enhance the interactivity of multimedia information across a communication network or in combination with other networks, including telecommunication and cable broadcasting. The system includes a plurality of multimedia call processing system, an interactive call processing system, and a plurality of interactive multimedia devices. The system further utilizes compression and/or transmission algorithms to maximize enhancement of the multimedia information. U.S. Patent No. 5,351,076, issued September 27, 1994 to Hata et al., discloses a terminal system having video telephone equipment cooperating with a computer system. The video telephone equipment includes a controller with a processor, a telephone unit and a TV camera unit. Data received in the video telephone equipment is transmitted from the controller to a remote station via a telecommunication line, such as an ISDN line.

U.S. Patent No. 5,374,952, issued December 20, 1994 t Flohr, discloses a videoconferencing system for digital computer workstations that operate on a local area network to exchange data. The network includes a signaling local area network (A-LAN) for tiansmitting and receiving data between workstations, and a broadband local area network (B-LAN) for transmitting and receiving television signals between workstations. The A-LAN generates data messages which initiate and control the transmission of the signals on the B-LAN.

U.S. Patent No. 5,412,417, issued May 2, 1995 to Tozuka, discloses a video telephone device with automatic video camera angle adjustment.

U.S. Patent No. 5,436,654, issued July 25, 1995 to Boyd et al., discloses video teleconferencing unit, which includes a lens assembly, speaker, and microphone in a single housing, and further includes a lens tilt mechanism wherein the lens is fixedly mounted to a bezel that is pivotally mounted on a front cover of the housing. Rotation of the lens can be accomplished by rotating the bezel without having to move the housing. U.S. Patent No. 5,438,357, issued August 1, 1995 to McNelly, discloses a teleconferencing system that allows for natural eye contact between conferees. Eye contact is provided, despite the mounting of the camera above eye level, by image manipulating the image of a conferee to remove any distortion caused by camera placement and to redirect the apparent direction of the conferee's gaze. U.S. Patent No. 5,473,679, issued December 5, 1995 to La Porta et al., discloses a signalling system for broadband communications networks, which allows a communications service subscriber to select a signalling provider independently of the transport carriers which control the local loops for particular services, and the providers of those services. U.S. Patent No. 5,485,504, issued January 16, 1996, discloses a hand-held radiotelephone with video transmission and display, which includes electroacoustic transducers mounted at sound apertures in the housing, a keypad mounted between the apertures, electronic circuitry, a video camera mounted in the housing and connected with the electronic circuitry, and a liquid crystal display mounted in the housing and also connected with the circuitry. U.S. Patent No. 5,502,727, issued March 26, 1996 to Catanzaro et al., discloses an image and audio communication system having graphical annotation capability. A first means encodes the acquired audio signal into a group of audio data packets which are periodically transmitted over the analog line, during a first time interval. The image signal, encoded as a group of image data packets, and the annotation signal, encoded as a group of annotation data packets, are interleaved and transmitted with the audio data packets. The audio packets are transmitted during a first time interval and the image and annotation data packets are transmitted during a second time interval following the first time interval. Control signals for terminal apparatus operation are encoded as control data packets and also transmitted during the second time interval.

U.S. Patent No. 5,506,964, issued April 9, 1996 to Arshi et al., discloses a PC- based conferencing system, which resides partially on a host processor and partially on a video board. The video and audio respectively subsystems generate and pass local compressed video and audio signals to the communications subsystem, which transmits the local compressed video and audio signals over a communications link to a remote computer system. The communications subsystem also receives remote compressed video and audio signals over the communications link from the remote computer system, and passes the remote compresses video and audio signals to the video and audio systems, respectively, for remote playback. However, in spite of these advancements in the prior art, none of these prior art references disclose or suggest a videophone system for use with a PC that will provide suitable performance.

This and other needs in the art will become apparent to those of skill in the art upon review of this specification, including its drawings and claims.

SUMMARY OF THE INVENTION

It is an object of the present invention to provide a videophone system and method of processing audio and video data.

It is another object of the present invention to provide for a videophone system for use with a PC that will provide suitable performance. Thess and other objects of the present invention will become apparent to those of skill in the art upon review of this specification, including its drawings and claims. According to one embodiment of the present invention, there is provided a videophone system. The system generally includes a computer with a display screen.

Standing alone from but in communication with the computer is a processing unit with software algorithms for compression of audio, video and other data. The system further includes a camera in communication with the processing unit for taking pictures and outputting video signals to the processing unit, and a telephone unit in communication with the processing unit for sending audio signals to and receiving audio signals from the processing unit. According to another embodiment of the present invention, there is provided a method for processing audio, video or other media signals. The method generaly includes capturing the audio, video, or other media signals. These captured are then forwarded to a compression processor for data compression. These compressed signals are then forwarded to a computer. BRIEF DESCRIPTION OF THE DRAWINGS FIG. 1 is an illustration of the videophone 10 of the present invention, and includes a base unit 15, handset 12 and camera assembly 21.

FIG. 2 is a schematic representation of base unit side 10B, showing power input 40, power cord 42, connection slot 51, expansion interface port 55, external microphone input 48, and external loudspeaker 45.

FIG. 3 is a schematic representation showing videophone 10 of the present invention mounted to the side of a typical computer system 60 of the present invention having computer terminal 61 and computer housing 62. FIG. 4 is an illustration of the back of computer housing 62 of FIG. 3, showing fan vent 75, power input 77, power plug 79, SCSI port interface 74, and slots 71 including parallel port interface 72.

FIG. 5 is a schematic representation showing an expansion peripheral 82 and a chained expansion peripheral 92, expansion peripheral 82 includes port 85 for connection with expansion interface port 55 of videophone 10, and port 87 for communication with additional chained expansion peripherals.

FIG. 6 is a schematic representation of the electronics of videophone 10, showing base unit 15, expansion interface 55, external microphone input 48, external loudspeaker 45, base unit microphone 18, base unit loudspeaker 13, handset 12, connector 51, digital signal processor 84, lens assembly 26, and power input 40.

FIG. 7 is a block schematic of the various functions of digital signal processor array 84, showing initial program load routines 190, audio routines 130, expansion peripheral routines 160, video routines 180, and computer communication routines 150. FIG. 8 is a schematic representation of data flow from videophone 10-1 to video phone 10-2, and provides an overview of the data flow through each main software module, with FIGs. 9-18 providing detail views of the flow through the primary modules listed on FIG. 8.

FIG. 9 is a schematic representation of the audio input data flow, showing the sequence that generates audio media stream 303 of FIG. 8. FIG. 10 is a schematic representation of the video data flow, showing the sequence that generates a video media stream 305 of FIG. 8.

FIG. 11 is a schematic representation of sequence 330 that generates media stream 308 for any peripherals.

FIG. 12, is a schematic representation of further detail of videophone to computer connection module 350, showing various methods 360 to connect videophone 10 to computer 60.

FIG. 13 is a schematic of processing sequence for transmission software module 400, and details the sequence in which the data is received from videophone 10-1 by the transmitting computer. FIG. 14 is a schematic showing one side of the connection scenarios of a videophone to videophone connection.

FIG. 15 is a schematic showing the sequence for reception software module 500, in which the data is received by the computer and forwarded to the videophone unit. FIG. 16 is a schematic representation of the expansion communications proxy module 630, which is the sequence that reproduces or interprets an incoming peripheral date stream.

FIG. 17 is a schematic showing the details of sequence 620 that physically reproduces video from an incoming video media stream. FIG. 18 is a schematic representation of sequence 610, which is the sequence that physically reproduces audio from an incoming audio media stream. DETA1LED DESCRIPTION OF THE INVENTION

Description of hardware

The present invention will first be described by reference to FIG. 1, which is an illustration of the videophone 10 of the present invention. As shown in FIG. 1, videophone 10 includes a base unit 15, handset 12 and camera assembly 21.

While base unit 15 as illustrated is designed to be mounted to the side of a personal computer (PC), it must be understood that base unit 15 may be designed to be wall mounted, to sit or be mounted on top of the computer screen, or even designed as a stand-along unit. For the embodiment shown in FIG. 1, most conveniently, brackets designed to interlock either directly with the base unit 15 or with mating brackets on base unit 15, are mounted to the side of the PC. Referring now to FIG. 3, there is illustrated on method of mounting, in which base unit 15 is secured to the side of the PC. Base unit 15 encloses electronics including the digital signal processor (DSP), and further includes cradle 30 for receiving and securing handset 12, internal hands free microphone 18, internal hands free loudspeaker 13, volume control 20, one or more mode switch/indicators 34, and camera 21.

Cradle 30 is designed to receive and secure handset 12, regardless of whether videophone 10 is mounted on the left or right side of a PC.

The one or more mode switch/indicators 34 are provided to control and indicate the operational mode of the videophone 10. In the embodiment as shown, these button/indicators control a specific mode by being depressed, and will illuminate to indicate that the particular mode is active. Non-lirniting examples of modes include "power" to indicate whether the unit is on or off, "conference" to indicate if a conference is in session, "hold" to indicate if the session is on hold, "mute" to indicate if the audio is muted, and "blank" is the video equivalent of the audio hold and indicates if the "video" is blanked. Of course, it must be understood that more modes Of course, it must be understood that videophone 10 may have more or less of these switch/buttons 34 than as illustrated in FIG. 1, depending upon the particular design.

Internal hands free loudspeaker 13 and internal hands free microphone 18 are designed to be active during a session when handset 12 rests in cradle 30, and will become inactive if handset 12 is lifted to allow for private conversation. Such a hands free mechanism for such operation is well known in the art, and any suitable mechanism may be utilized. Volume control 20 is provided to allow for control of the volume of internal hands free loudspeaker 13.

Handsets are well known in the telecommunications art, and it is expected that in the practice of the present invention that any suitable handset may be utilized as handset 12. As illustrated in FIG. 1, handset 12 is a standard telephone handset including ear section 11 housing a speaker, mouthpiece section 14 housing a microphone, handle grip section 16, and cord 17 connecting handset 12 to base unit 15. It is to be understood that the present invention is not limited to any particular type of handset, for example, handset 12 can comprise a cordless handset.

Camera 21 is mounted in base unit side 10A in tapered aperture 28, and includes camera lens assembly 26, lens directional mount 25, lens focus controller 32, and camera aim controller 23.

The taper of tapered aperture 28, provides a wider viewing range for lens assembly 26, and allows for camera lens assembly 26 to be mounted slightly recessed from the surface of base unit side 10A to provide some protection to the lens assembly 26. Lens directional mount 25 is a gimble mechanism that secures lens assembly 26, and may be movable permit the aiming of lens assembly 26 at a desired target. Camera aim controller 23 is provided to allow the user the point lens assembly at a desired target. While camera air controller 23 is illustrated as a common joystick, it is to be understood that any suitable controller may be utilized.

It must be understood that any suitable type or quality of lens assembly 26 may be utilized in the present invention, generally limited only by the desires of the user and economics.

For example, for an economy unit, lens assembly 26 may be a single focus lens.

As another example, two or more lens of various focal lengths may be provided a linear mount or a circular mount, which mount is then either linearly or angularly moved to exchange the lens. For example, camera lens assembly of FIG. 1 includes three bubble lens of various focal lengths, "close-up", "normal" and "wide angle", mounted on a polycarbonate strip in a linear fashion. Sliding lens focus controller 34 up and down, will move the mounting strip through directional mount 25, interchanging the three lens.

As even another example, lens assembly 26 can comprise a single adjustable lens assembly. Such lens adjustment can be made manually, or automatic focusing may be provided.

The camera 21 may be any camera suitable for capturing and generating video signals, and generally includes a solid-state image sensor which converts a light image gathered by the lens assembly 26 into a solid-state signal, most commonly for example, a Charged Coupled Device (CCD). Camera 21 may be selected to accommodate monochrome or color images.

Referring now to FIG. 2, there is shown a schematic of base unit side 10B, showing power input 40, power cord 42, connection slot 51 , expansion interface port

55, external microphone input 48, and external loudspeaker 45. Power input is generally a DC power input. While connection slot 51 may any suitable connector, it is conveniently an industry standard 25wayD connector.

Configuration of hardware Referring now to FIG. 3 there is shown videophone 10 of the present invention mounted to the side of a typical desktop type of computer system 60 of the present invention having computer terminal 61 and computer housing 62. While the present invention is illustrated as being used with a desktop type of computer, it is to be understood that it may also be utilized with a portable computer. In FIG. 3, video phone 10 is shown schematically, as parallel port 51, expansion interface port 55, and the digital signal processor (DSP) 84 is shown on the camera side. As shown in FIG. 1, connector 51 and expansion interface port 55 are both located on side 10B. The digital signal processor is internally located within electronics assembly housing 15. While connector 51 may be any type of connector, it is conveniently an industry standard 25wayD connector.

As shown in FIG. 3, computer 60 includes computer terminal 61 and computer housing 63. Computer terminal 61 include screen 61 and control panel 63. Computer housing 62 includes diskette drive 65, CD drive 64, fan vents 67, and power switch 68. Of course, it is to be understood that while desktop computer 60 is illustrated as having a horizontally oriented computer case 61, the present invention may also be utilized with a "tower" type of computer case.

Software resident in computer 60 will query and update the capabilities of videophone 10 and any attached expansion peripherals by communicating with and/or replacing microcode programs resident in the digital signal processor 84 located in videophone 10 and in any expansion peripherals. This resident software of computer

60 will then utilize the capabilities of the product to provide functionality and increased processing throughput that is not achievable using the central processing unit in standalone personal computer systems.

Referring additionally to FIG. 4, there is shown the back of computer housing 62, showing fan vent 75, power input 77, power plug 79, SCSI port interface 74, and slots 71 including parallel port interface 72. Video phone 10 communicates with computer 60 via connector 51 of video phone 10, and either SCSI port interface 74 or parallel port interface 72 of computer 60. It is to be understood that communications between video phone 10 and computer 60 may occur via cable, or wireless such as RF signal, infrared signal or the like, or any other suitable transmission medium.

Referring additionally to FIG. 5 there is shown an expansion peripheral 82 and a chained expansion peripheral 92. Expansion peripheral 82 includes port 85 for connection with expansion interface port 55 of videophone 10, and port 87 for communication with additional chained expansion peripherals. Likewise, chained expansion peripheral 92 includes port 95 for connection with expansion interface port

87 of expansion peripheral 82, and port 97 for communication with additional chained expansion peripherals.

Numerous expansion peripherals may be chained to videophone 10. Non- limiting examples of such peripherals include additional videophone units, scanners, mobile phone units (cordless phone), alarms, pagers, cameras, projectors, answering machines, and the like. It is to be understood that communications between video phone 10 and the peripherals may occur via cable, or wireless such as RF signal, infrared signal or the like, or any other suitable transmission medium. Electronic configuration

Referring now to FIG. 6, there is shown a schematic representation of the electronics of videophone 10, showing base unit 15, expansion interface 55, external microphone input 48, external loudspeaker 45, base unit microphone 18, base unit loudspeaker 13, handset 12, connector 51, digital signal processor 84, lens assembly

26, and power input 40.

As explained above, handset 12 is a standard telephone handset including ear section 1 1 housing a speaker, mouthpiece section 14 housing a microphone, handle grip section 16. Handset 12 further includes handset speaker 14A positioned in ear section 14, and handset microphone 1 IA positioned in mouthpiece section 11. Cord

17 connects handset 12 to base unit 15 via sockets 101 on handset 12 and socket 103 on base unit 103.

Handset mode detect switch 106 operates to detect if if handset 12 is on or off hook. If handset 12 is on hook, on hook handset switch 108 operates to activate base unit microphone 18 and base unit loudspeaker 13. Alternatively, if handset 12 is off hook, on hook handset switch 108 operates to activate handset 12 and to mute base unit microphone 18 and base unit loudspeaker 13.

Plugging an external microphone into external microphone input 48, and an external loudspeaker into external loudspeaker output 45, serves to deactivate base unit microphone 18 and base unit speaker 13.

Sound received by the handset microphone 11 A, the external microphone 48, or the base unit microphone 18, is directed to the digital signal processor array 84 via microphone amplifier 110 and analog-to-digital converter (A-D) 111. Microphone amplifier serves to amplify the signal, and converter 111 serves to convert the analog signal to a digital signal.

Sound to handset speaker 14A, external loudspeaker 45, or base unit loudspeaker 13, arrives from the digital signal processor array 84 via digital-to-analog converter (D-A) and loudspeaker amplifier 1 18. Loudspeaker amplifier 118 serves to amplify the signal, and converter 115 serves to convert the digital signal to an analog signal. Volume control 20 serves to control the output volume to handset speaker 14A, external loudspeaker 45, or base unit loudspeaker 13.

Multiplexer 122 serves to sequence images received by CCD array 26. On external static RAM 225 reside the initial program load microcode which is loaded upon start-up, and compression and decompression routines to process audio and video signals. Expansion peripherals communicate with digital signal processor 84 through expansion interface 55 and buffer 128. Communication occurs between digital signal processor 84 and computer 60 via connector 51 and buffer 131.

Digital signal processor

Referring now to FIG. 7, which is a block schematic of the various functions of digital signal processor array 84, there is shown initial program load routines 190, audio routines 130, expansion peripheral routines 160, video routines 180, and computer communication routines 150.

Expansion peripheral routines 160 include expansion driver manager 155, and expansion communication proxy 169. The expansion peripheral routines 160 query the capabilities of any attached peripherals and establish a communication and control channel between the peripherals and computer 60.

Audio routines 130 consist of audio device manager 125, echo cancellation 131, input audio bandpass filter 133, compression 135, encryption 138, decryption 148, decompression 145, and output audio bandpass filter 143. Video routines 180 consist of video device manager 126, image enhancer 181, decompression 185, compression 187, image normalizer 189. Static ram 225 contains initial program load microcode 250, and one or more compression routines 228, and one or more decompression routines 230. Upon start up, initial program load routines 250 is downloaded from static RAM 225 into the digital signal processor as initial program load microcode 190. Referring still to FIG. 7, the computer software queries the videophone communications port ( either parallel or SCSI ) to verify that the communications arbitration manager is in a ready state.

The computer software requests the communications arbitration manager to establish a data connection with the feature arbitration manager. The computer software requests a description of the active videophone configuration from the feature arbitration manager and is returned a list of the features set up in the Default

Configuration Load.

If the feature arbitration manager reports an attached peripheral, the computer software requests the communications arbitration manager to establish a data link with the attached peripheral. The computer software then iterates down the chain of attached peripherals to determine their active configuration.

The computer software compares the user's feature(s) request with the active videophone and attached peripherals configuration. If there is not a one-to-one match of request vs configuration, the negotiation stage continues. The computer software requests that the feature arbitration manager install the user requested feature. If the requested feature is represented by a microcode module stored in Static Ram, the feature arbitration manager replaces the appropriate active feature with the correct representation stored in Static RAM and reports success to the computer software and the stage is complete. If the requested feature is not represented by a microcode module in Static

Ram, the feature arbitration manager reports failure to the computer software. The computer software examines the list of stored feature microcode in its storage area in the computer, If the requested feature is not available, the computer software reports failure to the user and the stage is complete.

The computer software uploads the stored feature microcode module to the feature arbitration manager. The feature arbitration manager stores the downloaded feature in Static RAM. The feature arbitration manager replaces the currently active feature module with the newly downloaded feature and updates the configuration data in the Static RAM so that the new feature becomes the default.

The computer software establishes a multi-channel data link with one or more remote computers. The computer software arbitrates between the requested features of each user to establish a common denominator of video features for the connection.

The computersoftware enters the computer to videophone feature negotiation stage to adjust videophone to the settings required for the multi-channel data link.

If an audio media stream is required for the connection, the computer software establishes a control and data link through the communications arbitration manager to the analog-to-digital processing stack and to the digftal-to-analog processing stack,

If a video media stream is required for the connection, the computer software establishes a control and data link through the communications arbitration manager to the output media channel processing stack and to the input media channel processing stack.

If additional features are required and those features are provided by attached peripherals, the computer software establishes a link through the communications arbitration manager to the features arbitration manager of each respective peripheral until all features requests have been activated or denied. The computer software regulates the outgoing media streams from the videophone and the incoming media streams from the remote computers and routes the streams to the appropriate videophone processing stack for end point processing. The computer software continuously monitors the state of the multi-channel data connection and communicates with the feature arbitration manager to dynamically adjust the videophone processing abilities in order to provide the best perceived performance to the user.

Compression/Decompression Routines

In the practice of the present invention, it is generally desirable to provide a screen refresh rate of at least about 1 frame/second, preferably at least about 5 frames/second.

In the practice of the present invention, it is preferable to achieve a screen refresh rate that will provide video which appears to the naked eye to be continuous.

The nώiimum refresh rate to provide video which appears to the naked eye to be continuous is at least about 12 frames/second. For most applications, no additional advantage is achieved utilizing refresh rates above about 30 frames/second.

The amount of compression necessary to achieve the above desired refresh rates will depend upon the pixel size of the displayed image, and upon the communications bandwidth. For example, to achieve 2 to 3 frames/second for a 160x120 pixel image utilizing a 14.4kbps modem over a phone line, requires compression on the order of about 3 to about 5. As another example, to achieve 12 to 15 frames/sec, requires compression on the order of about 7 to about 12.

In the practice of the present invention, any suitable compression routine may be utilized that will provide the necessary compression for the desired refresh rate. Examples of commercially available compression routines include Wavelet

Compression available from RICOH California Research Center, Fractal Compression available from Iterated Systems, and MPEG available from Motion Picture Coding Experts Group.

The inventors have invented a novel compression routine which they prefer in the practice of the present invention, which is generally described as follows. The primary audio compression/decompression algorithm utilized in videophone 10 of the present invention is derived from the modification and unique combination of two well-known compression algorithms. These algorithms are known as Adaptive Delta Pulse Code Modulation (ADPGM) and Arithmetic Compression. The modifications made to each algorithm cause the algorithms to be complimentary and dependent. Both-algorithms have been enhanced to provide high compression ratios and efficient CPU utilization when used in conjunction with each other.

The ADPCM algorithm is a lossy compression algorithm. Therefore, the audio data coming out of the decompressor will not exactly match the audio data that was input to the compressor. Lossy audio compression algorithms take advantage of the fact that a person can still understand a sound after it has been modified or noise has been introduced.

Like most most personal computer sound cards ( using analog to digital converters), the present invention generates audio data in a form called Pulse Code Modulation. Pulse Code Modulation represents the value of the audio wave form in a discrete time interval (generally measure in milliseconds ) as a binary value known as a sample. As the amplitude of the audio wave increases, so does the absolute value of the sample. Delta Pulse Code Modulation (DPCM) records the numerical difference between two consecutive samples rather than the actual value of each sample. Due to the nature of audio waveforms and speech in particular, the difference between two consecutive samples is much smaller than the actual value of each sample. This difference can be represented by a binary value that is smaller than an actual sample and therefore offers a degree of compression.

ADPCM takes DPCM one step further by attempting to predict which direction the wave form is going to take and how large the difference will be between the next samples, ADPCM is typically used as the only compressor/decompressor (codec) stage where in videophone 10 of the present invention, the ADPCM codec has been coupled with an Arithmetic codec to achieve greater compression and lower processor utilization.

In the present invention, the ADPCM algorithm has been modified in two ways. The first way the ADPCM algorithm has been adjusted is that the DPCM portion encodes sample differences in 2 bits of data, thereby yielding a fixed 4-to-l compression ratio when operating on 8-bit samples. In addition to compressing the data, the 2-bit compressed data allows the Arithmetic Compression algorithm to operate with more efficient processor utilization. Arithmetic compression tends to exhibit heavy processor utilization and the 2-bit nature of the incoming data eases the processor load in order to allow real-time compression of the data.

The second way the ADPCM algorithm has been adjusted is that it has been modified to use a static predictor table that accommodates the differences arriving as

2-bit quantities. The static predictor table provides an efficient, indexed lookup to predictor coefficients that are used to modify the DPCM output to more closely model the original audio data.

Arithmetic compression is a non lossy compression algorithm. The data being generated by the decompressor stage exactly matches the original data sent to the compressor. Arithmetic compression is applicable to any type of data where as ADPCM is adapted strictly for audio data.

The Arithmetic codec of the videophone 10 of the present invention is coupled to the ADPCM codec. The audio data is first compressed by the ADPCM (lossy) stage and then further compressed by the Arithmetic (lossless) stage.

Arithmetic compression works by taking a stream of input data and converting that data to a single floating point number. This conversion is based on a probability table that is used to adjust the floating point number. The floating point number is adjusted to ever higher degrees of precision in order to accurately represent the input data.

Arithmetic compression is typically applied at the byte level in order to compress a stream of bytes into the floating point number representation. It examines each byte and indexes into the probability table to determine how to adjust the floating point number.

The videophone of the present invention modifies the arithmetic compression algorithm to accept an incoming data length of 2 bits. This is the 2 bits of output data that the modified ADPCM algorithm generates for each byte of incoming data. Because 2 bits of data can only represent 4 possible values and because the DPCM difference values can only be represented by these 4 values, the probability of each value occurring consecutively increases dramatically and therefore the arithmetic compressor can achieve higher compression ratios. Also, because the arithmetic compressor only has 4 input values to deal with and therefore 4 values to maintain in the probability range table, the compressor can produces results faster and with lower processor utilization than arithmetic compressors operating on a data size of 1 byte or larger.

The 2-bit modification to the ADPCM algorithm and the Arithmetic algorithm is also carried one step further in the videophone of the present invention to include adjustable input data width. This means that instead of the present invention's modified implementation of the ADPCM and Arithmetic algorithms being fixed to operate on 2 bits of input data, the software module implementing the algorithms can be dynamically adjusted to a different input data width such as 3 bits or 4 bits.

As the input data size increases, the processor load increases. The videophone of the present invention utilizes digital signal processor array 84 to carry the increased load. This enables the compression/decompression stages to accommodate larger input data sizes in real-time than would be possible with a stand alone personal computer executing the codec as well as other tasks necessary for audio and video communications.

Logical communications and control

The computer software of computer 60 provides the computer interaction with the user and communicates with videophone 10 through the parallel port 72 or the SCSI port 74 to the 25wayD connector 51 to utilize the additional functionality provided by the product and any attached peripherals. Of course, the present invention can also communicate with computer 60 through means other than the parallel port or serial port, for example, with radiofrequency or infrared.

The microcode executing in digital signal processor array 84 is queried by computer 60 to determine processing power, microcode functionality, and feature revision number. The communications protocol and command capabilities are then established based on the results returned from the microcode in digital signal processor 84.

The established communication protocol is then used to send commands to digital signal processor 84 that require it to report the capabilities of the product such as the resolution of the CCD array 26, the frequency response of the sound circuitry, the currently loaded microcode programs/algorithms, and the presence of attached peripherals. The microcode programs and algorithms will provide functionality such as video compression, audio compression, and peripheral signaling control.

Any attached peripherals are then queried through the expansion interface using the same procedural approach described above that was used to query digital signal processor 84. Computer 60 presents the user with feature options based on the capability responses provide by digital signal processor 84 and any attached peripherals. The user selects the features necessary to accomplish the desired task (such as audio compression/decompression only, or audio and video codec processing ) and computer 60 sends commands to videophone 10 to prepare it for operation. Additionally, the user may request a feature that is not currently loaded in digital signal processor 84 (such as a compression algorithm with different characteristics). Computer 60 will then upload the microcode for the requested feature to digital signal processor and activate the feature.

The computer 60 will then initiate a communications session another computer also having a videophone 10 of the present invention. The user is provided with a multi-channel data link to the other computer or a user operating the other computer. Computer 60 is sensitive to the transmission rate and quality of service of the multi¬ channel data link and signals the microcode in digital signal processor 84 array in order to compensate for an increase or decrease in data channel performance. Computer 60 utilizes the attached peripheral devices and digital signal processor 84 to achieve very high data compression ratios and data redundancy in real time that otherwise would not be possible using the software and single CPU's of modern personal computers alone. The very high data compression of the audio and video media streams being generated by videophone 10 allow the transmission of media streams in real time over very low speed, lower integrity communications devices such as 14.4 Kbps and 28,8 Kbps asynchronous modems. The low bandwidth requirements of the media streams generated by videophone 10 allow the computer software of computer 60 to introduce or remove data redundancy as necessary in the multi-channel data link when service quality changes. Computer 60 commands the codes executing in digital signal processor array 84 to increase compression ratios in the media stream, which lowers the bandwidth of the media stream and allows redundant data to be introduced into the multi-channel data link.

Computer 60 will command the microcode executing in the digital signal processor array to adjust compression ratios in response to changes in the available bandwidth of the multi-channel data link. The compression ratio, audio quality, and video quality settings are balanced by the software in cooperation with videophone

10 to provide the best perceived performance to the end user.

The digital signal processor array 84 will be computer 60 to provide media stream encryption and decryption services in the post compression/pre decompression microcode should the user desire secure multi-channel data link connections. One or more channels, such as audio or audio and video, can be selectively encrypted to provide the necessary security coverage.

Initial Program Load and Default Configuration Static RAM module 225 persistently stores previous configuration data and default startup code. The following sequence lists the significant microcode events that occur upon power-up.

Upon power up the digital signal processor 84 executes boot code loaded from memory module 5, which is generally a read only memory (ROM) or an electronically programmable read only memory (EPROM). The executing boot code loads the initial program load microcode 250 and transfers execution to the initial program load microcode 250.

Initial program load microcode 250 is downloaded into digital signal processor 84 as initial program load microcode 190 and begins executing.

Initial program load microcode 190 loads into digital signal processor 84, device manager 126 for CCD array 26, device manager 155 for expansion interface port 55, device manager 125 for analog-to-digital converter 111 and digital-to-analog converter 115, and device manager 151 for 25 D connector interface 51.

Initial program load microcode 190 also loads the feature arbitration manager 193, expansion communications proxy 169, and communications arbitration manager 157.

Initial program load module 190 then signals feature arbitration manager 193 to load the default feature configuration.

Once loaded into digital signal processor 84, feature arbitration manager 193 reads the configuration data stored in static RAM 225, and associates the device manager ID stored in the configuration data with the corresponding executing device manager.

The configuration data associated with the device manager for the analog-to- digital converter is interpreted and the resolved echo cancellation filter 131, input audio bandpass filter 133, compression module 135, and encryption module 138 are layered on device manager 125 to form the processing stack for the input audio media channel.

The configuration data associated with device manager 125 for digital-to- analog converter 115 is interpreted and the resolved echo cancellation filter 131, output audio bandpass filter 143, decompression module 145, and decryption module 148 are layered on device manager 125 to form the processing stack for the output audio media channel. The configuration data associated with device manager 126 for CCD array 26 is resolved and the resolved image quality filter 181 and decompression module 185 are layered on device manager 126 to form the processing stack for the output video media channel. The configuration data associated with device manager 126 for CCD array 26 is resolved and the resolved image normalizer 189 and compression module 187 are layered on device manager 126 to form the processing stack for the input video media channel.

Feature arbitration manager 193 then queries the expansion communications proxy 169 to determine if there is an attached peripheral at expansion port interface

155. The expansion communications proxy 169 is then set to an asynchronous notification state that instructs it to notify the feature arbitration manager 193 in the event of a connection state change.

Feature arbitration manager 193 signals the communications arbitration manager 157 to enter a ready-waiting for connection state.

In the event of static RAM failure or failure of the standby battery that maintains the static RAM, the boot code can inform the computer software of the failure and any and all microcode modules can be downloaded from the computer through the videophone to computer connection.

Data flow

The data flow will now be described by reference to FIGs. 8 to 18, which figures describe a single direction of data flow through videophone 10 hardware and software. The data flow is illustrated as flowing from an originating videophone unit to a receiving videophone unit, although the flow is full-duplex ( bidirectional, both direction simultaneously). It must be remembered that when the connection between two videophone units is established, each videophone unit acts as both originator and receiver to allow simultaneous transmission and reception.

Additionally, data flow is not intended to be limited to single point-to-point connections. Three or more videophone units can be connected to each other simultaneously, forming a multi-way transmission/reception conference connection.

FIG. 8 is an overview of the data flow through each main software module, with FIGs. 9-18 providing detail views of the flow through the primary modules listed on FIG. 8.

Referring now to FIG. 8, there is shown a schematic representation of data flow from videophone 10-1 to video phone 10-2. A media stream 301 is generated from the originating videophone unit 10-1 by one or more of the microcode processing stacks, audio stack 310, video stack 320 and peripheral stack 330, in the originating videophone unit 10-1. The media streams consist of sound data 303, video data 305, and peripheral-specific data 308. Peripheral-specific data is generated by a peripheral attached to expansion interface port 55 of videophone 10-1.

Media stream 301 is transmitted through videophone 10-1 computer connection 350 to computer 405 associated with videophone 10-1 as media stream 351.

Transmission software module 400 receives media stream 351 and forwards it through computer 405 as media stream 401 to computer connection 450. From computer connection module 450, media stream 451 is received by receiving computer 505.

Reception software module 500 receives incoming stream 451 and forwards it as stream 501 through connection module 550. Media stream 551 arrives in the receiving videophone unit 10-2 and is distributed to the appropriate microcode processing stack, either audio stream 603 to audio stack 610, video stack 605 into video stack 620, or expansion peripheral proxy stream 608 into peripheral stack 630, depending upon the type of data contained in stream 551.

Referring now to FIG. 9, there is shown a schematic of the audio input data flow, showing the sequence 310 that generates audio media stream 303. The generated stream 303 represents audio that is to be transmitted to one or more receiving videophones,

A sound is generated by audio source 311, which is generally a person speaking into one of the microphones 1 IA, 18 or 48 of videophone 10-1. The analog sound is converted in to digital data by 8 bit analog to digital converter 111.

Device manager 125 retrieves the data from 8-bit analog to digital converter 111 and forwards it to echo cancellation module 131. The echo cancellation module 131 compares the data to any sound that was just produced by videophone output speakers 14A, 13 or 45. Any matching sounds are removed from the audio stream to prevent feedback between the output speakers 14 A, 13, or 45 and the microphones

11 A, 18 or 48. Echo cancellation module 131 forwards the data to input audio bandpass filter 133.

Input audio bandpass filter 133 modifies the audio waveform data _to remove artifacts (such as noise spikes) and undesired frequencies (those not normally produced by the human voice). This modifies the audio waveform data so that the compression module 135 may operate at peak efficiency when compressing the human voice. The input audio bandpass filter 133 then forwards the data to the compression module 135.

Compression module 135 compresses the audio data and forwards it to the encrytion module 138. The encryption module 138 serves to encrypt the data and forward it to the communications arbitration manager 157. Note that the encryption key used to encrypt the data has been supplied by the computer software, according to the user's preference, and will be provided to the recipient in a secure transfer independent of the communication.

The communications arbitration manager 157 receives the incoming data as stream 303, orders it with other incoming data streams, and forwards it to device manager 151.

Device manager 151 transmits the data through the 25wayD connector 51 to videophone to phone connection module 350 as data stream 301.

Referring now to FIG. 10, there is shown a schematic representation of the sequence 320 that generates a video media stream 305. The generated stream 305 represents video that is to be transmitted to one or more receiving videophone units.

CCD 26 receives reflected light rays from image source 321, which image is represented in digital form internally in CCD 26. CCD device manager 126 requests each pixel of data from CCD 26 and assembles the digital raster lines that represent the image. Device manager 126 subsequently forwards the assembled image to image normalizer 189.

Image normalizer 189 is responsible for removing artifacts (e-g. random isolated pixels of a different color in-a field of pixels all of one color) and details that are generally unnoticeable (e-g. slight variations in color shading) to the user. This "normalizes" and de-randomizes the image so that the best possible compression can be achieved by compression module 187. The image normalizer 189 then forwards the image- data to the compression module 187. Compression module 187 compresses the data and forwards it to communications arbitration manager 157 as stream 305 of FIG. 8. This communications arbitration manager 157 receives the incoming data, orders it with other mcoming data streams 303 and 308, and forwards it to device manager 151. Device manager 151 then transmits the data as media stream 301 through 25way D connector 51 to the videophone to computer connection 350.

Referring now to FIG. 1 1, there is shown a schematic representation of sequence 330 that generates media stream 308 for any peripherals. The generated stream 308 represents data that is specific to the type of peripheral device. An example of the data would be account information generated by a credit card reader (an expansion peripheral) attached to expansion interface port 55. Another example would be image data generated from a fax device attached to expansion interface port 55. The expansion peripheral 82 and possibly one or more chained peripherals 92, send data through expansion interface port 55 to expansion peripheral device manager 155. Device manager 155 then forwards the data to the expansion communications proxy 169.

Expansion communications proxy 169 forwards the data as peripheral data stream 308 to communications arbitration manager 151 ,

The microcode processing stacks, such as the input audio media channel processing stack 310 or the output video media channel processing stack 320, or stack

330 for expansion peripherals, are not required to be stored or executing in the main videophone unit, but each device that generates a media stream is required to have at least one associated processing stack and device manager.

Expansion communications proxy 169 acts as a representative for processing stacks that are executing in peripherals attached to the expansion interface port 55. It intelhgently forwards commands and replies to and from expansion peripherals 92 chained to the expansion interface part 55. Communications arbitration manager 157 forwards the data to device manager

151, which in turn forwards the data through 25way D connector 3-1 to the videophone to computer connection 350.

Referring now to FIG. 12, there is shown a further illustration of videophone to computer connection module 350, showing a schematic representation of various methods 360 to connect videophone 10 to computer 60. As illustrated, the communication between videophone 10 and computer 60 may be by parallel cable

361, SCSI cable 362, RF signal 363, infrared beam 364, or any other suitable means 365, such as satellite link.

Referring now to FIG. 13, there is provided the processing sequence for transmission software module 400, and details the sequence in which the data is received from videophone 10-1 by the transmitting computer. The computer software modules responsible for manipulating the data in the computer are depicted as items 412, 414 and 415. These modules can be independently operating software programs that cooperate together or they can be algorithmic entities incorporated into a single software program. Module 400 operates generally as follows. Data arrives through videophone to computer connection 410. An event is signaled in the computer that informs the collect data module 412 that data is available from the connection. The collect data module 412 queues the data in a memory buffer, signals the format data for transmission module 415 that data is awaiting to be formatted, and waits for the next event.

The format data for transmission module 415 interprets the data headers to determine the type of data. The format data for transmission module 415 then deteπnines the intended recipient of the data (in the case of a multi-way connection with other videophone units) and the type of connection existing between this originating unit and the target videophone unit. Examples of the connection type would be: TCP/IP network connection, SPX/IPX connection, Frame Relay Connection, asynchronous modem connection, etc.

The format data for transmission module 415 then formats the data into that required by the connection type (packets, multiplexed streams, etc.). The format data for transmission module 415 adds the formatted data to the outgoing queue for the connection, signals the transmit data module 415 that data is waiting to be transmitted, and waits for the next event.

The transmit data module 415 retrieves the formatted data from the connection queue and transmits it through the appropriate remote system connection as shown in FIG. 14. Referring now to FIG. 14 there is shown a schematic representation of one side of the connection scenarios of a videophone to videophone connection. Many video connections can be conferenced together through the entity labeled as a WAN (Wide

Area Network). The Internet, as well as private Intranets connected via various network topologies can be classified as a WAN. The arrowed lines on the FIG. 14 represent bidirectional data flow paths that carry videophone data. Dotted lines such as 452 or 454 are optional flow paths the depict the versatility of the videophone connection medium.

The videophone of the present invention sends data through the dual mode bidirectional port (25wayD connector). Data then passes to the computer through either the SCSI (Small Computer Systems Interface) port or the parallel port. From the computer, the data may take one of the 3 typical paths depicted in order to travel through the WAN to the receiving computer. Instead, the data may take another, less typical oath such as a VSAT (Very Small Aperture Terminal) satellite link or ATM

(Asynchronous Transfer Mode) fiber link. The first typical path depicted is through a 28.8 kbps asynchronous modem connection over a PSTN (Public Switched Telephone Network) link. The second typical path depicted is through a 64 kbps or 128 kbps digital modem connection over an IDSN (Integrated Services Digital Network) link.

The third typical path depicted is through a low speed network class link. The data travels from the computer to the router, through the CSU/DSU (Channel Service Unit/Data Service Unit), through the digital modem, over a 1.544 mbps Tl link to the

WAN,

Referring now to FIG. 15 there is shown the sequence for reception software module 500, in which the data is received by the computer and forwarded to the videophone unit. Data arrives through the remote system interconnection 530. An event is signaled in the computer that informs the receive data module 531 that data is available from the connection 530. The receive data module 531 queues the data in a memory buffer, signals the interpret data format module 533 that data is waiting, and waits for the next event. The interpret data format module 533 understands how the data has been formatted for transmission by examining which connection queue (TCP/IP,SPX/IPX, etc) the data arrived in. The module 533 removes the transmission headers from the data and reassembles/demultiplexes the data if necessary.

The interpret data format module 533 adds the data to the distribution queue, signals the distribute data module 537 that data is waiting in the distribution queue, and waits for the next event.

The distribute data module 537 sends the data through the videophone to computer connection 539 to the appropriate target. Note that the appropriate target may be another software module executing in the computer and not a target across the videophone to computer connection 539.

In this case, the data is redirected to the target module incoming queue and the target module mcoming event is signaled.

An example of this redirection is the case of an incoming fax that the user wants to view on the screen instead of it being printed by an attached expansion peripheral. The distribute data module 537 would redistribute the data to a screen drawing module instead of through the videophone to computer connection 539 to the printing peripheral.

Referring now to FIG. 16, there is shown a schematic representation of the expansion communications proxy module 630, which is the sequence that reproduces or interprets an incoming peripheral date stream. The generated stream represents data that is specific to the type of peripheral device. In FIG. 16 and in this description, an example is presented that represents sending peripheral data from an originating peripheral device (such as a fax peripheral) to a receiving peripheral device (such as a fax device or printer). Other scenarios are possible. A one-to-one correspondence does not have to exist. An originating peripheral device does not require an associated receiving peripheral device. Another receiver of the data could be the computer software. An example of this scenario is a fax device scanning a document and transmitting it through the videophone unit to the receiving computer. The software in the receiving computer displays the fax on the screen, instead of passing the information to a printing device. Operation of module 630 is generally as follows. The peripheral data travels through the videophone to computer connection 550, through the 25 way D connector 51(1 0-2), to the device manager 151.

The device manager 151 forwards the data to the communications arbitration manager 157. The communications arbitration manager 157 then forwards the date to the expansion communications proxy 169 as expansion media stream 608.

The expansion communications proxy 169 examines the data and determines to which attached expansion peripheral 82 or 92 the data is destined.

The expansion communications proxy 169 forwards the data to device manager 155 along with instructions about which device to deliver the data. The device manager 155 forwards the data through expansion interface port 55 to the targeted expansion peripheral 82 or 92.

Referring now to FIG. 17, there is shown details of sequence 620 that physically reproduces video from an incoming video media stream. The computer software 61S of computer 60 retrieves the data stream and forwards it through the videophone to computer connection 550, through the 25wayD connector 51 to the device manager 151.

The device manager 151 receives the data stream 551 and forwards it to the communications arbitration manager 157. The communications arbitration manager 157 then identifies any image data present in the incoming data stream 551 and forwards it as stream 605 to the decompression module 185. The decompression module decompresses the data stream 551 and forwards it to the image quality filter 181. The image quality filter 181 removes any image artifacts generated when the image was compressed and forwards the image to the device manager 126.

The device manager 126 redirects the decompressed image back to the communications arbitration manager 157, which then forwards the data to device manager 151.

Device manager 151 forwards the data through the 25wayD connector, through the videophone to computer connection 550, to the computer software 6 IS.

The computer software 6 IS renders the image on the computer monitor screen as picture 6 IP for the user to view.

Referring now to FIG. 18, there is provided a schematic representation of sequence 610, which is the sequence that physically reproduces audio from an incoming audio media stream.

The data stream 551 arrives through videophone to computer connection, through the 25wayD connector 51, and is received by the device manager 151. Device manager 151 forwards the data to the communications arbitration manager 157, which identifies any sound data present in incoming data stream 551 and forwards it as audio stream 603 to the decryption module 148.

The decryption module 148 decrypts the data and forwards it to the decompression module 145. Note that the decryption key used to decrypt the data has been supplied by the computer software, according to the user's preference, and must match the key used to encrypt the data or the final sound output may not be recognizable.

The decompression module 145 decompresses the data and forwards it to the output audio bandpass filter 143, which removes any noise spikes and undesirable frequencies from the waveform data and forwards it to the echo cancellation module

131. Note that lossy compression techniques can result in undesirable artifacts when the data is decompressed and it is this issue that the output audio bandpass filter addresses.

Echo cancellation module 131 receives the data and updates the echo comparison buffer. The echo cancellation module 131 maintains a constantly updating copy of the sound data that has passed through it in the last time interval.

This is termed the echo comparison buffer. It uses the comparison buffer during videophone audio input (FIG. 9) to prevent audio feedback.

The echo cancellation module 131 forwards the data to the device manager 125, which forwards the data to the 8 bit digital to analog converter. The 8-bit DAC converts the data from digital format to analog format and the signal is transmitted to the audio reproduction device 13 (alternatively to handset speaker 14A or external speaker 45) for physical reproduction.

While the illustrative embodiments of the invention have been described with particularity, it will be understood that various other modifications will be apparent to and can be readily made by those skilled in the art without departing from the spirit and scope of the invention. Accordingly, it is not intended that the scope of the claims appended hereto be limited to the examples and descriptions set forth herein but rather that the claims be construed as encompassing all the features of patentable novelty which reside in the present invention, including all features which would be treated as equivalents thereof by those skilled in the art to which this invention pertains.

Claims

I CLAIM 1. A videophone system comprising: (a) a computer; (b) a display screen connected to the computer; (c) a processing unit in communication with the computer; (d) a camera in communication with the processing unit for taking pictures and outputting video signals to the processing unit; (e) a telephone unit in communication with the processing unit for sending audio signals to and receiving audio signals from the processing unit; (f) compression software executable by the processing unit for compressing audio and video signals received by the processing unit into compressed signals and outputting said compressed signals to the computer;

2. A method for processing audio and video signals, comprising (a) capturing audio and video signals as captured signals; (b) forwarding the captured signals of step (a) to a processor; ( ) compressing the captured signals to form compressed signals; (d) forwarding the compressed signals to a computer.