WO2001071687A2

WO2001071687A2 - Phased array surveillance system

Info

Publication number: WO2001071687A2
Application number: PCT/US2001/008484
Authority: WO
Inventors: Carleton S. Hayek; James W. Sari; Peter Stankovich; Bruce K. Newhall
Original assignee: The Johns Hopkins University
Priority date: 2000-03-17
Filing date: 2001-03-16
Publication date: 2001-09-27
Also published as: AU2001252913A1; WO2001071687A3

Abstract

The present invention uses a pair of phased array of microphone panels in conjunction with a digital signal processor, disk storage, and beam-steering and noise suppression algorithms to discreetly provide a phased array surveillance system (PASS) for audio data. The above elements are covertly housed in a briefcase, for instance. Also included is a wireless RF transceiver system that relays data picked up by the microphone array panels to a remote unit for real-time analysis. The remote unit is also capable of controlling the beam forming and noise suppression algorithms as well as perform beam-steering of the PASS unit allowing the remote operator to selectively target individual speakers in real-time. Data recorded by the PASS unit is made available for subsequent analysis using beam forming and noise suppression algorithms.

Description

TITLE OF THE INVENTION Phased Array Surveillance System

CROSS-REFERENCE TO RELATED APPLICATIONS [0001] This application claims the benefit of prior filed co-pending U.S. Provisional Patent Application No. 60/190,204, filed on March 17, 2000.

BACKGROUND OF THE INVENTION [0002] The present invention relates to an audio surveillance system using an electronically steerable array of microphone elements.

[0003] Success in countering terrorism is often dependent on the quality of intelligence gathering. Human intelligence (HUMINT) in the form of monitored conversations is often valuable in preventing terrorist acts. However, monitoring conversations among people can be a complex task depending upon the environment. Ambient noise, multiple speakers, machinery, and other forms of deliberate or naturally occurring noise can significantly affect the quality of audio data gathered.

[0004] Traditional listening devices are highly directive and cumbersome. These include parabolic dishes and shotgun microphones. These highly directive devices are unsuited for covert data collection due to their unwieldy physical characteristics and the fact that they must be aimed at the a target.

[0005] What is needed is a portable audio surveillance system that can pick specific conversations out of a noisy environment and, does not need to be aimed directly at a target audio source.

SUMMARY OF THE INVENTION [0006] The present invention is a phased array surveillance system (PASS) comprised of at least one planar array of microphones that can be electronically steered, data acquisition electronics, data processing capabilities including beamforming and beam-steering, data storage capabilities, remote control electronics, a remote control unit, and a power supply. The PASS unit (everything except the remote control unit) can be housed in a carrying case that resembles a standard briefcase, small suitcase, or other custom designed enclosure.

[0007] The present invention adapts phased array technology into a portable system that can perform covert audio surveillance in noisy, interference filled environments where traditional directional microphones or high gain parabolic reflective dishes are not useful. The arrays of microphones are concealed within the housing and remain motionless. The microphone arrays can be electronically beam-steered such that areas of interest can be focused upon using advanced signal processing on the raw data picked up by the microphone arrays. The advanced signal processing includes beamforming algorithms and noise suppression or cancellation algorithms that significantly reduce or eliminate unwanted background noise.

[0008] A beamformer is a spatial filter that operates on the output of an array of sensors in order to enhance the amplitude of a coherent wavefront relative to background noise and directional interference. The goal of beamforming is to sum multiple elements (microphone array elements in this case) to achieve a narrower response in a desired direction which is known as the Maximum Response Angle (MRA). Thus, when sound is present in a given beam, it can be determined which direction the sound came from. In addition, the MRA can be selectively chosen for the beams. The act of choosing the pointing direction or MRA of a beamformer is referred to as beam-steering. [0009] The PASS unit is advantageously placed to ensure the best possible audio pick-up of the targets. At normal speaking levels, targets can be tens of meters from the PASS unit and still be heard. The PASS unit range can be increased or decreased depending on the level of noise in the environment. Audio surveillance data is stored locally by the PASS unit on a hard drive or other similar memory device concealed within the housing. The stored data is recovered and subsequently processed on a separate computer. Beamforming algorithms allow an operator to filter data of interest emanating from a selected direction. The noise suppression algorithms can automatically suppress noise emanating from directions other than the direction of interest thereby clarifying the voice of a target. [0010] Real-time monitoring of the PASS unit is also available via a wireless RF connection with a remote unit. The remote unit can activate or de-activate the PASS unit and perform beamforming, beam-steering, and noise suppression functions in real-time. An operator can control PASS unit functions and monitor PASS unit output via a discreet RF transceiver system built into the PASS housing. Remote control does not affect or interfere with the PASS unit's primary objective of storing raw data picked up from the microphone arrays. Thus, real-time monitoring and beam-steering will not affect the integrity of the data collected by the PASS unit. Rather, the remote unit allows an operator to monitor and track a specific target in real-time while raw data from the microphone arrays is preserved for subsequent analysis of other possible targets within the overall range of the PASS unit.

BRIEF DESCRIPTION OF THE DRAWINGS [0011] FIGURE 1 illustrates a block diagram of the elements that comprise the phased array surveillance system (PASS) of the present invention. [0012] FIGURE 2 A illustrates one example of a microphone array orientation. [0013] FIGURE 2B illustrates another example of a microphone array orientation. [0014] FIGURE 2C illustrates yet another example of a microphone array orientation. [0015] FIGURE 3 illustrates one example of a PASS application where multiple targets are recorded. [0016] FIGURE 4 is a logic flow diagram illustrating a typical surveillance operation.

DESCRIPTION OF THE PREFERRED EMBODIMENTS [0017] FIGURE 1 is a block diagram of the phased array surveillance system (PASS). A PASS unit may be housed in, for instance, a briefcase 310 as shown in FIGURE 3. Referring again to FIGURE 1, microphone array panels 100 respond to sound in the vicinity of the PASS unit. The microphone array panels 100 include individual microphone elements 110. The microphone elements 110 can be either directional or omni-directional, depending on the PASS application and the baffling created with respect to the microphone array panel housing. Directional microphone elements (e.g. cardioids) that have main response axes directed normally along a microphone array panel may be interleaved with directional microphone elements that have response axes directed along the opposite normal. This configuration yields back-beam rejection so long as the microphone array panel is acoustically transparent, or does not interfere with cardioid microphone operation. Depending on the PASS application, the array housing is designed for utility or covert functionality. The present invention is not limited to a single housing configuration. Those of ordinary skill in the art can readily devise alternative housing configurations without departing from the spirit or scope of the present invention. [0018] Sound that is picked up by the microphone array panels 100 is subjected to a preprocessing stage 120. Pre-processing typically includes power amplification and bandwidth filtering. An analog-to-digital (A/D) signal converter 130 converts the analog output of the pre-processing stage 120, from each microphone element 110, into a digital bit stream. Digital bit streams from the A/D converter 130 are stored in a memory device 140. It is recommended that memory device 140 have sufficient capacity to store data for the anticipated surveillance time. Data storage can be accomplished as part of the A/D conversion, using a special purpose card connected to a computer data bus. One possible memory device is a hard disk that is has sufficient storage and speed characteristics to meet the anticipated requirements of a surveillance operation. Once audio data has been acquired, processing algorithms can be applied, via a digital signal processor (DSP) 150 to the audio data to perform spatial filtering, temporal filtering, beamforming, noise cancellation, and other signal enhancement filtering functions.

[0019] The present invention also includes remote control capability. An RF transceiver unit 160 is concealed within the housing of the PASS unit. The RF transceiver unit 160 is communicable with a remote unit 170. Communication between the RF transceiver unit 160 and the remote unit 170 is two-way. The RF transceiver unit 160 is operatively connected with the DSP 150 such that the remote unit 170 can control the DSP 150. [0020] The remote unit 170 enables the operator to control the PASS unit. The remote unit 170 can turn the PASS unit on and off, and can perform real-time processing on audio data as it is collected. Real-time processing permits the operator to beam-steer the PASS unit, adaptively noise cancel, adaptively spectral filter, and monitor the audio data as it is picked up by the pass unit. Real-time processing does not, however, interfere with audio data recording to memory device 140. Thus, all of the audio data picked up by microphone array panels 100 is available for alternative beamforming analysis and beam- steering during a post-recording analysis session. This is true for each individual microphone element 110 in a microphone array panel 100. [0021] The remote unit 170 can take the form of a personal digital assistant PDA such as a HandspringVisor™, or a web-enabled cell phone having a data screen. Those of ordinary skill in the art can readily devise alternate means for the remote without departing from the spirit or scope of the present invention.

[0022] FIGURE 2 A illustrates one example of a three dimensional microphone array configuration. A three dimensional configuration of microphone array panels provides gain in three dimensions amplifying the audio target signal and spatially separating and attenuating interference sources. One such three dimensional configuration is comprised of two parallel planar microphone array panels 210A, 210B that are offset. The second offset microphone array panel 210B provides modest back-beam rejection to the beams formed by the first microphone array panel 210 A.

[0023] FIGURE 2B illustrates another example of a three dimensional microphone array configuration that utilizes two planar microphone array panels 210C, 210D. The planes of microphone array panels 210C, 210D are different, however. In this case the microphone array panels 210C, 210D are oriented in perpendicular planes. Using perpendicular planar microphone array panels permits enhanced audio pick-up. For instance, the first planar microphone array panel experiences a decreased response as the angle of target audio source relative to the plane of the microphone array panel increases. If multiple targets are at 90 degrees relative to the plane of the microphone array panel it becomes extremely difficult to isolate among targets that are in close proximity. However, if a second planar microphone array panel that is perpendicular to the first planar microphone array panel is introduced, then the same targets are more easily distinguishable by the second planar microphone array panel. [0024] The present invention illustrates a second microphone array panel that is perpendicular to the first in FIGURE 2B. Other orientations besides perpendicular can be chosen for specific applications. Those of ordinary skill in the art can readily orient a second, third, fourth, or n^Λ microphone array panel with respect to one another to create a set of panels that maximizes response in several directions. Typically, the design choice and panel orientation is dependent on the type of covert housing being used to conceal the microphone array panels. A briefcase, for instance, lends itself to using two offset parallel microphone array panels where each one is placed proximal to the largest surface area on either side of the briefcase. If the housing possesses a cubical geometry, six panels could be implemented on the faces of each of the sides of the cube as shown in FIGURE 2C. This would provide a robust audio data collection unit.

[0025] A second planar microphone array panel is necessary to obtain a three dimensional configuration. For purposes of the present invention, a second planar microphone array panel is present when any microphone element is not in the same geometric plane as any other microphone element. Moreover, a planar microphone array panel can be comprised of a single microphone element or a plurality of microphone elements. In addition, the degree of difference between planes of microphone elements does not change the definitions of terms herein. Rather, the degree of difference between planes is a design choice that will affect the effectiveness of the microphone array panels' ability to distinguish targets at various locations.

[0026] Other microphone array configurations are possible and can be designed to accommodate space, cost, and/or convenience constraints. Thus, the present invention is not limited to a single microphone array configuration. Those of ordinary skill in the art can readily devise alternative microphone array configurations without departing from the spirit or scope of the present invention.

[0027] PASS also functions if microphone arrays are physically separated in separate housings, provided that the digital bit data streams are brought together and processed simultaneously. This can be accomplished by using a single remote unit that is in communication with each of the PASS units.

[0028] FIGURE 3 illustrates one example of a PASS application where multiple targets are recorded. A PASS unit concealed within briefcase 310 is advantageously situated so as to encompass as many targets as possible. In this case there are several tables 320A-F each having potential targets. These targets are engaged in conversation with one another. The PASS unit briefcase 310 remains motionless but is capable of picking up audio data from each of the targets at each of the tables. Some targets will be picked up better than others but data for all of them is likely to be picked up by the PASS unit. Once the raw audio data has been collected it can be subjected to noise suppression algorithms and beamforming algorithms in order to focus on a particular target. In order to focus on a different target, the data is subjected to the same algorithms but with different parameters. By changing the parameters of the algorithms, one is able to electronically beam-steer the microphone array to listen to sounds from a particular direction. This procedure can be repeated over and over on the raw data in order to hear what each target is saying at a given moment.

[0029] The algorithms may also be run in real-time via a remote unit that has access and control over the PASS unit. This allows the remote operator to focus on one target such as 320F while data from all other targets 320A-E is still being collected for subsequent analysis. The remote operator can electronically steer the beam to follow a moving target or to switch to another target.

[0030] FIGURE 4 is a logic flow diagram illustrating a typical surveillance operation. It can be partitioned into three sections: data collection, real-time processing, and post collection processing. For data collection, a PASS unit is placed in the vicinity of one or more target audio sources and activated. Activation can be a remote function. Once activated the microphone array panels concealed within the PASS unit housing begin to acquire audio data 410. All audio data within the range of the PASS unit is picked up. The audio data is then pre-processed 420 to amplify and bandwidth filter the audio picked up by the microphone array panels. Bandwidth filtering optimizes audio signals of interest (e.g. frequency range for human speech). Each of the microphone elements are pre- processed separately and in parallel.

[0031] The pre-processed analog audio signal is then converted 430 to a digital bit stream by an analog-to-digital (A/D) converter. The digital sample rate must satisfy Nyquist criteria across all channels (microphone elements) simultaneously. The time jitter between channels must be held to a small fraction (on the order of 1/10) of the cycle time of the highest frequency in order to allow beamforming. Following A/D conversion, the digital bit stream is stored 440 in a memory device such as a hard disk. [0032] Once data is acquired it can be processed to determine what a given target has said. Processing can occur in real-time or during a post collection session. Real-time processing requires instant access to the data as it is acquired. In order to preserve the covert nature of a surveillance operation, an RF transceiver system is employed. The RF transceiver system allows an operator to remotely communicate with the electronics within the PASS unit from a remote unit. [0033] There are two possible implementations with respect to real-time processing of data picked up by the microphone array panels. One implementation is to process the data within the PASS unit under remote control 450 and have the processed data sent to the remote unit. The other implementation is to have the PASS unit send raw data 460 to the remote unit and allow the remote unit to process the data 470 locally. Processing includes beamforming, beam-steering, and noise suppression analysis.

[0034] Post collection processing entails accessing or downloading the stored data 480 on the PASS unit to a separate computer. Once the data is obtained signal processing 490 such as beamforming, beam-steering, and noise suppression algorithms can interpret the raw data.

[0035] The signal to noise (S/N) ratio of the audio source of interest (e.g., target) can be increased by spatial filtering and temporal filtering. A beamformer is a spatial filter that operates on the output of an array of sensors in order to enhance the amplitude of a coherent wavefront relative to background noise and directional interference. [0036] Beamforming algorithms include conventional beamforming (coupled with adaptive noise cancellation to suppress sidelobe response to signal interferers), or adaptive beamforming , which maintains an optimum S/N ratio in the presence of moving interference sources.

[0037] The approach of the present invention improves the noise reduction performance against moving interferers relative to existing adaptive beamforming algorithms (e.g. Minimum Variance Distortionless Response (MVDR) algorithms). In practice, the location, direction of travel, speed, and mean noise level of each interferer are not known with sufficient precision to use a modeled ensemble mean as a basis for adaptive beamforming. This can be overcome by accurately estimating the ensemble mean based on data samples.

[0038] The ensemble of co variance samples is comprised of rapidly varying random terms associated with the emitted noise and more slowly oscillating deterministic terms associated with the source and receiver motion. The non-stationary ensemble covariance mean can be estimated by filtering out the rapidly varying noise while retaining the slower oscillatory terms. Performance of the filters can be visualized and assessed in the "epoch frequency domain" which is the Fourier transform of the covariance samples. In this domain, higher bearing rates show up at higher frequencies. Whereas, traditional sample mean estimators retain only the zero frequency bin corresponding to stationary interference. However, techniques that can identify and include the appropriate non-zero frequency contributions are better non-stationary estimators than the sample mean. [0039] . Temporal filtering suppresses interference that has temporal and spectral characteristics that are unrelated to the target audio source. Signal spectral filtering can also be applied to optimize system response to particular speech characteristics of the target audio signal. For real time surveillance, parameters for spectral filtering (e.g. poles and zeros) could be adaptively estimated with the human operator designating which speaker to adapt to. Or, the parameters could be downloaded from a stored file corresponding to speech characteristics of a particular target individual. [0040] The data stream relayed to the remote operator can be as simple as a single beam that the operator listens to, or multiple beams whose energy is plotted so that the operator is cued for which directions have the highest signal levels. PASS control, spatial and temporal filtering, and plotting of highest signal beams is accomplished using a graphical user interface and a data screen on the remote unit.

[0041] There are several advantages associated with the present invention. The PASS unit can record all conversations within its range whether targets are moving or stationary. The microphone array remains stationary and discreet. Signal processing rejects multiple interference and noise sources to a high degree of efficiency. The PASS unit can be left unattended to record and/or relay raw data that can be processed and analyzed later for specific conversations.

[0042] Inherent in the design is the ability to recover each of several conversations occurring at different locations without having to re-position the PASS unit. Thus, an operator can be discreetly positioned and remotely control the PASS unit to focus (beam- steer) on any of several sources in real-time. Moreover, the operator can electronically beam-steer the PASS unit to follow targets if they happen to move during the conversation.

[0043] It is to be understood that some or all aspects of the present invention illustrated herein are readily implementable by those of ordinary skill in the art as a computer program product having a medium with computer program(s) embodied thereon. The computer program product is capable of being loaded and executed on the appropriate computer processing device(s) in order to carry out the method or process steps described. Appropriate computer program code in combination with hardware implements some of the elements of the present invention. This computer code is typically stored on removable storage media. This removable storage media includes, but is not limited to, a diskette, standard CD, pocket CD, zip disk, or mini zip disk. Additionally, the computer program code can be transferred to the appropriate hardware over some type of data network.

[0044] The present invention has been described, in part, with reference to flowcharts or logic flow diagrams. It will be understood that each block of the flowchart diagrams or logic flow diagrams, and combinations of blocks in the flowchart diagrams or logic flow diagrams, can be implemented by computer program instructions. [0045] These computer program instructions may be loaded onto a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions which execute on the computer or other programmable data processing apparatus create means for implementing the functions specified in the flowchart block or blocks or logic flow diagrams. [0046] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart blocks or logic flow diagrams. The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart blocks or logic flow diagrams.

[0047] Accordingly, block(s) of flowchart diagrams and/or logic flow diagrams support combinations of means for performing the specified functions, combinations of steps for performing the specified functions and program instruction means for performing the specified functions. It will also be understood that each block of flowchart diagrams and/or logic flow diagrams, and combinations of blocks in flowchart diagrams and/or logic flow diagrams can be implemented by special purpose hardware-based computer systems that perform the specified functions or steps, or combinations of special purpose hardware and computer instructions.

[0048] In the following claims, any means-plus-function clauses are intended to cover the structures described herein as performing the recited function and not only structural equivalents but also equivalent structures. Therefore, it is to be understood that the foregoing is illustrative of the present invention and is not to be construed as limited to the specific embodiments disclosed, and that modifications to the disclosed embodiments, as well as other embodiments, are intended to be included within the scope of the appended claims. The invention is defined by the following claims, with equivalents of the claims to be included therein.

Claims

CLAIMS:

1. A computer system for acquiring audio data, comprising: an array of microphone elements that acquires audio data, wherein said array of microphone elements comprise: a first planar microphone array panel having at least one microphone element; and a second planar microphone array panel having at least one microphone element, wherein said first planar microphone array panel and said second planar microphone array panel are in different geographical planes; a memory device that stores acquired audio data; and a housing that conceals said array of microphone elements and said memory device.

2. The computer system of claim 1 , wherein said first and second planar microphone array panels include at least three microphone elements.

3. The computer system of claim 1, wherein said first and second planar microphone array panels are acoustically transparent.

4. The computer system of claim 1, further comprising: an RF transceiver that transmits said acquired audio data, said RF transceiver being concealed within said housing; a remote unit that receives audio data transmitted from said RF transceiver, said remote unit separate from said housing.

5. The computer system of claim 4, wherein said remote unit further comprises a digital signal processor for processing said acquired audio data by applying beamforming and noise suppression algorithms to said received audio data.

6. The computer system of claim 5, wherein said digital signal processor includes temporal filtering that suppresses interference which has temporal and spectral characteristics that are unrelated to a target audio source.

7. The computer system of claim 5, wherein said digital signal processor includes estimating an ensemble covariance mean by filtering out rapidly varying random terms associated with emitted noise while retaining slower oscillating deterministic terms associated with source and receiver motion.

8. The computer system of claim 5, wherein said digital signal processor includes using spectral filtering to optimize computer system response with respect to particular speech characteristics of a target audio source by estimating spectral filtering parameters for said target audio source.

9. The computer system of claim 5, wherein said digital signal processor includes using spectral filtering to optimize computer system response with respect to particular speech characteristics of a target audio source by downloading speech characteristics of said target audio source from a stored file.

10. The computer system of claim 5, wherein said remote unit includes a display that provides a graphical user interface that controls said digital signal processor.

11. The computer system of claim 1, further comprising a digital signal processor for processing said acquired audio data by applying beamforming and noise suppression algorithms to said acquired audio data.

12. The computer system of claim 11, wherein said digital signal processor includes temporal filtering that suppresses interference which has temporal and spectral characteristics that are unrelated to a target audio source.

13. The computer system of claim 11, wherein said digital signal processor includes estimating an ensemble covariance mean by filtering out rapidly varying random terms associated with emitted noise while retaining slower oscillating deterministic terms associated with source and receiver motion.

14. The computer system of claim 11, wherein said digital signal processor includes using spectral filtering to optimize computer system response with respect to particular speech characteristics of a target audio source by estimating spectral filtering parameters for said target audio source.

15. The computer system of claim 11, wherein said digital signal processor includes using spectral filtering to optimize computer system response with respect to particular speech characteristics of a target audio source by downloading speech characteristics of said target audio source from a stored file.

16. The computer system of claim 11, further comprising: an RF transceiver concealed within said housing and communicable with said digital signal processor such that said RF transceiver can control said digital signal processor; and a remote unit communicable with said RF transceiver, said remote unit separate from said housing, said remote unit being able to control said digital signal processor via said RF transceiver, such that said remote unit can cause said digital signal processor to process said acquired audio data and said RF transceiver can relay processed audio data to said remote unit.

17. The computer system of claim 16, wherein said remote unit includes a display that provides a graphical user interface that can control said digital signal processor.

18. A computer system for acquiring audio data, comprising: a microphone array component that acquires audio data, wherein said microphone array component comprises: a first planar microphone array panel having at least three microphone elements; and a second planar microphone array panel having at least three microphone elements, said first planar microphone array panel and said second planar microphone array panel being in different geographical planes; a pre-processing component, operatively connected to said microphone array component, that bandwidth filters and amplifies audio data acquired by said microphone array component; an analog-to-digital converter component, operatively connected to said pre- processing component, that converts analog output from said pre-processing component to a digital bit stream; and a storage component, operatively connected to said analog-to-digital component, that stores said digital bit stream.

19. The computer system of claim 18, wherein said first and second planar microphone array panels are acoustically transparent.

20. A computer system for acquiring audio data, comprising: an array of microphone elements that acquires audio data, said array of microphone elements comprises: a first planar microphone array panel having at least three microphone elements; and a second planar microphone array panel having at least three microphone elements, said first planar microphone array panel and said second planar microphone array panel being in different geographical planes; a memory device, operatively connected to said array of microphone elements, that stores acquired audio data; and a housing, operatively connected to said array of microphone elements and said memory device, that conceals said array of microphone elements and said memory device.

21. The computer system of claim 20, wherein said first and second planar microphone array panels are acoustically transparent.

22. The computer system of claim 20, wherein said first and second planar microphone array panels are concealed in separate housings.

23. A method for providing a phased array surveillance system, said method comprising the steps of: acquiring audio data through an array of microphone elements in different geographical planes; estimating an ensemble covariance mean by filtering out rapidly varying random terms associated with emitted noise while retaining slower oscillating deterministic terms associated with source and receiver motion. storing the acquired audio data; and concealing the array of microphone elements in a housing.

4. A method for acquiring audio data, said method comprising the steps of: preparing audio data; converting the audio data from analog to digital; receiving control signals; transmitting the audio data to a remote unit; performing signal processing on the audio data in real-time; storing the audio data; and performing signal processing on the audio data.