WO2002029570A1

WO2002029570A1 - System, apparatus, method and article of manufacture for testing information systems

Info

Publication number: WO2002029570A1
Application number: PCT/US2000/028286
Authority: WO
Inventors: Michael Schechter; Brett Maurer; Curtis Elsasser
Original assignee: Metratron Technologies, Inc.
Priority date: 2000-10-03
Filing date: 2000-10-03
Publication date: 2002-04-11
Also published as: AU2001213325A1

Abstract

There is provided a framework that interfaces with several measurement devices, and records, manipulates, analyzes, and displayes measurement data. Application-specific functionality can be added, upgraded or removed to address a wide variety of test requirements.

Description

System, Apparatus, Method and Article of Manufacture for Testing Information Systems

CROSS-REFERENCE TO RELATED APPLICATIONS

This patent application relates to United States Non- Provisional Application entitled Apparatus, Method and Article of Manufacture for Testing Information Systems, filed October 2, 2000 and to United States Non-Provisional Application entitled System, Apparatus , Method and Article of Manufacture for Evaluating the Quali ty of a Transmission Channel Using Voice Recogni tion Technology, filed September 1, 2000.

Each of the foregoing patent applications is incorporated in its entirety herein by reference.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

Not Applicable.

GLOSSARY

This glossary contains terms used in this specification . The reader is urged to look to this glossary if any term used in the specification is unclear . COM - The Component Object Model (COM) is a way for software components to communicate with each other. It's a binary and network standard that allows any two components to communicate regardless of what machine they're running on (as long as the machines are connected) , what operating systems the machines are running (as long as it supports COM) , and what language the components are written in. COM further provides location transparency: it doesn't matter to you when you write your components whether the other components are in-process DLLs, local EXEs, or components located on some other machine, (this definition from "Dr. GUI's Gentle Guide to COM")

COM Interface - A method by which COM components can communicate with each other.

GP-IB - The GPIB (general purpose interface bus) was specifically designed to connect computers, peripherals and laboratory instruments so that data and control information could pass between them. It is also known as IEEE-488 or HPIB, and is electrically equivalent to IEC- 625 bus. It is defined completely in the IEEE standard 488.1-1987 Standard Digital Interface for Programmable Instrumentation . Information systems - a system containing both hardware and software that was created using many different technology elements and product vendors. Such systems often have problems with interoperability, performance, and stability. Examples are wireless networks, VoIP networks, and distributed financial systems.

Measurement Device - a hardware or software test tool that measures the state of a complex system and generates measurement data. One measurement device might generate several disparate types of data. Examples of measurement devices are mobile phones, scanning receivers, protocol analyzers, and voice quality measurement tools.

Framework - software with little application-specific functionality. It is designed as a foundation to support the implementation of functional add-in components.

Protocol Analyzer - a test tool that monitors the operation of a device to see if it complies with a communications protocol standard. For example, a protocol analyzer for TCP/IP would monitor the transfer of packets and check addressing, timing, data rates, and error rates.

TCP/IP - two layers of a communications protocol that manage the transmission of data packets. These protocol layers are used in communications over the Internet . SDK - Software Development Kit. This is a document detailing the interfaces of a software application, framework, or library. It allows 3^rd parties to write components, integrate libraries, etc. that work with the given software.

Stream - a container that hold measurement data and behavior data. Streams are very flexible because they can contain almost any type of data. Streams can be directly accessed using interfaces.

User - technical personnel who use test equipment to test, monitor, and optimize information systems.

VoIP - Voice Over Internet Protocol

BACKGROUND

1. Field of Invention

The present invention, also referred to herein as

DimensionZ™, relates to systems analysis and more

particularly, to a system, apparatus, method and article of manufacture for recording, manipulating, analyzing, and displaying measurement data.

2. Description of the Related Art

Information systems have become very complex and require significant effort to build, optimize, and maintain. The Internet expanded the capabilities of information systems connecting millions of people throughout the world. Another information revolution will soon occur as 3^rd generation wireless technologies add mobility to the Internet .

As information systems become more complex, so do tools that test, monitor, or optimize the systems. Most test tools today do not capture all the necessary system operation data because they are designed to measure only one part of the system. There are two major types of test systems existing today that would be used on these information systems. They are as follows: Protocol Analyzer - test systems that specifically test a product's compliance to a communications protocol. These devices are often used in research and development but not in build out, optimization, or lifecycle support; and Resource Management - systems used to manage network resources . These systems are primarily used during lifecycle support to manage resources while the system becomes loaded.

Moreover, a protocol analyzer might analyze a TCP/IP network but it will not analyze VoIP performance. The opposite is also true. Most VoIP test equipment will not analyze the performance of the TCP/IP network. The first solutions that strived to test, monitor, and optimize information systems had two parts, a data recording part and a post-processing part. The data recording part was often a unit that interfaced with different measurement devices and recorded data into a single log file. A log file is a type of data file where disparate types of measurements are interleaved. Log files were used because early computer systems were not powerful enough to pre-process data. The post-processing part of the solution was then used to parse through the log file looking for information required for analysis.

The first test solutions had many shortcomings as follows:

• Real-time display and analysis of data was limited due to the little pre-processing during data recording.

• The architecture of data recording software was not modular and required significant development and system-wide regression testing to integrate new measurement devices .

• The format used to store data was not efficient thereby slowing down the speed of post-processing.

• Recorded data was not stored in a way that preserved all information about the measurement device. As computer technology advanced, new test solutions were implemented that utilized database technology instead of log files. The database technology improved some of the performance of post-processing but was still limited by inflexible software architecture.

A solution is needed that will interface with many types of measurement devices, correlate disparate types of data, analyze the data, and has the ability to take action to improve or optimize system performance.

SUMMARY OF THE INVENTION

The present invention addresses the forgoing needs and provides a solution for testing, monitoring, and optimizing today's information systems by offering the following:

• Support for quick integration of 3^rd party measurement devices . Technology changes very fast and measurement devices change equally fast.

• Process, analyze, and display data in real-time.

• Data from different devices should be correlated with an accuracy of no less than 0.5 seconds. Some testing applications require even more precise correlation.

• Support the implementation of intelligent, user- defined algorithms that can automatically analyze data. • Support the implementation of one centralized mechanism for configuring and controlling measurement devices .

• Support the implementation of one centralized mechanism for displaying information to the user.

• Record data in such a way as to preserve all information from the device that generated the data in order to be able to replay data exactly as it was recorded. This allows for the user to closely examine system events.

• Manage and store data in such a way that it can be quickly accessed and manipulated.

The present invention is a framework that interfaces with several measurement devices and records, manipulates, analyzes, and displays measurement data. Application specific functionality can be added, upgraded or removed to address a wide variety of test requirements .

The framework is a core application specially suited to fast data manipulation using "streams" of data (defined below) . The framework is built using a method of software development that utilizes strict and well-defined sets of functions called "interfaces" . Components that plug into this framework are used to implement application specific functionality and can be added and removed as needed. The core application manages all aspects of component lifetime, data manipulation, and data correlation.

The components communicate with the core application using a set of interfaces uniquely designed to support a wide variety of test requirements. These interfaces allow components to access the current system or replay time, create or connect to data streams, push data onto their data streams, iterate through data streams, receive notification when data has changed, send status/error information to the core application, and many other useful functions .

Data is stored in a unique container that supports a limitless variety of data types. These containers, called streams, contain both measurement data and behavior data. Measurement data might be a sequence of discrete points or a sequence of groups of points . Behavior data is used to record changes in the state of the measurement device that generated the Stream.

These and other aspects, features, and advantages of the present invention will become better understood with regard to the following description, appended claims, and accompanying drawings . BRIEF DESCRIPTION OF THE DRAWINGS

Turning now to the drawings,

Exemplary embodiments of the present invention are now briefly described with reference to the following drawings:

Figure 1 is a physical diagram of one embodiment of the present invention constructed in accordance with the teachings described herein.

Figure 2 is a functional of one embodiment of the present invention constructed in accordance with the teachings described herein.

Figure 3 shows how one embodiment of the architecture of the present invention constructed in accordance with the teachings described herein implements external functional modules .

Figure 4 is a diagram of another embodiment of the present invention.

Figures 5 and 6 are flow diagrams of a simple example illustrating an application of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description of the embodiments, reference is made to the accompanying drawings which form a part hereof, and in which is shown by way of illustration specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized and that structural, logical and electrical changes may be made without departing from the spirit and scope of the present inventions. The following detailed description is, therefore, not to be taken in a limiting sense, and the scope of the present inventions is defined only by the appended claims. The leading digit (s) of the reference numbers in the Figures usually correspond to the figure number, with the exception that identical components which appear in multiple figures are identified by the same reference numbers .

Diagrams for the framework' s physical and functional descriptions are shown in Figures 1 and 2. The framework can easily be adapted to different test applications by connecting measurement devices and adding functional add-in components .

FIGURE 1: System Diagram

Referring to Figure 1, each of the items in the diagram are described as follows : • Devices - a test tool used to measure a specific characteristic of an information system. Each device will often generate one or more data streams. A device may also accept input data to control its operation. An example might be a wireless phone that produces test data from the bottom of the unit and accepts control commands such as dialing specific phone numbers. Another example is a GPS that outputs latitude and longitude measurements.

• Device Interface Unit - device interface units are sometimes required to connect devices to the computer. Sometimes a device will provide a communications port that is not supported by the computer and must be adapted. Examples might include a USB Ethernet adapter, a set of USB RS-232 ports, or a GP-IB card.

• Computer - the computer executes the Dimensionz software.

•• Dimensionz Core - the Dimensionz core coordinates all components, streams, connections, timing, and persistence of data (storing and retrieving) .

• Data Storage - the files, databases or other storage that permanently store recorded data. • Display - the display that communicates information to the user. Common displays are CRT monitors, LCD screens, printers, etc. This can also be an HTML page, a report, or other output.

• User Input Device - the device that provides the user with a way to configure and control measurement devices software. Common devices include keyboards, mice, etc.

FIGURE 2 : Functional Diagram

Referring to Figure 2, each of the items in the diagram are described as follows :

• Physical Device - a test tool used to measure a specific characteristic of an information system. Each device will often generate one or more data streams. A device may also accept input data to control its operation. As above, examples might include a wireless phone or GPS .

• Device Interface Unit - device interface units are sometimes required to connect devices to the computer. Sometimes a device will provide a communications port that is not supported by the computer and must be adapted. • Component - a software application object that exposes the necessary software interfaces required by the core application ("core") framework. The core manages the life of this object.

• Display Window - a graphical user interface window that is provided by the core application. The component uses this window to display operational information. An example might be a graphical picture of a scanning receiver, a text listing, etc.

• Control Algorithm - an algorithm or set of algorithms in the component that control a measurement device, process data, generate data, or perform any other application specific function.

• Configuration - the current configuration of the component . This is information that the component can give to the present application that specifies the component's current state, such as which channels to scan, what phone number to dial, etc. The present application gives this information back to the component when the component is started in subsequent sessions (i.e.: when reading in a saved data file) .

• Interfaces - a set of strict and well-defined software interfaces that allow components to communicate with the core application. They are uniquely designed to support a wide variety of test requirements. Components must expose certain interfaces in order to be used as part of the present system, expose others as the method of transferring data between itself and the core, expose others to request a graphical window, to report operational status, etc. We have chosen Microsoft COM for our interface model in our preferred embodiment .

• Dimensionz Core/Cre - the core application coordinates all components, streams, connections, timing, and persistence of data.

• Server Manager - manages the lifetime of components.

• Stream Manager - manages the lifetime of streams

• Stream Engine - manages all connections to streams

• Playback Engine - controls data replay

• System Timer - provides a consistent source for timing information to all components.

• Data Storage - permanent storage of recorded data and configuration information. FIGURE 3 : Application Architecture Diagram

The framework is built using a strict methodology of well-defined software interfaces. Add-in components that communicate with the core framework using these interfaces are used to implement application specific functionality and can be added to and removed from the system as needed. The core application manages all aspects of component lifetime, data manipulation, and data correlation.

Field measurement products have historically divided data recording and post-processing into two separate software applications. The separation led to limited realtime analysis capabilities and timely post-processing procedures. The present invention is fundamentally different combining data recording functionality and postprocessing functionality into one application framework.

The architecture of the present invention implements all testing-specific functionality using add-in components. Functionality can easily be added, updated or removed depending on the test requirements.

Figure 3 shows how the architecture of the present invention implements functional modules external to the core application. Even though every component is a separate module, all components run seamlessly within the framework giving a consistent application environment. Most components fall into one of three general classifications. The classifications are as follows:

• Device Object (DO) -software modules that control test devices and write measurements to data streams (i.e. phones, scanning receivers, navigation, oscilloscopes, temperature sensors, etc.)

• Active Filter Object (AFO) - software algorithms that process stream inputs and generate new stream outputs

(i.e. floating average, event detection, data reduction)

• User Interface Object (UIO) - software user interface that displays stream data (i.e. text display, line graph, map)

All data that flows between components is contained in data streams. The term "stream" refers to a sequence of discrete measurements recorded over a period of time. Our preferred embodiment of a discrete measurement is referred to as a "datapoint", and consists of an 8-byte (double) timestamp that handles resolutions to the millisecond, a VARIANT (a flexible Microsoft data type) that holds a value, and a VARIANT that holds a behavior. The structure of the datapoint must be the same for all components, and thus must be flexible enough to hold almost any type of data. For more complex data types, Dimensionz exposes an interface to handle a tagged data format to facilitate runtime discovery of how a datapoint is made up. For example, you have a GPS component, and the value of your datapoint is a structure that holds latitude, longitude, speed, direction, and altitude. The structure can be picked-apart by any other component so it can access a specific part of your data, such as speed in this example.

The core of the present invention manages all component behavior and stream data. It also insures that all components are synchronized when performing data recording and replay.

The core manages data in ways that deliver optimum performance. Many common post-processing tasks take only a fraction of the time they used to with older software products .

Application specific functionality is implemented using add-in components. Components can be activated before, during, or after data recording depending on the task being performed. Add-in components must expose the software interfaces the present invention expects to see in order to be used within the framework. Our current system includes an SDK (software development kit) , so 3^rd parties can create their own add-ins. Figure 4 is a more detailed diagram of another embodiment of the present invention.

Figures 5 and 6 are flow diagrams of a simple example illustrating an application of the present invention wherein a GPS device is the only device producing data in the system.

What follows is a detailed description of another example illustrating another application of the present invention.

In this example, a wireless phone service provider has gotten many complaints about data and voice call quality in a particular part of town and wants to test their network. They offer two different models of wireless phones to their customers. They have one computer set up in the- lab (the "landline side"), and another in a van that will drive the route in question (the "mobile side") . Some of the components set forth below are described in detail in United States Non-Provisional Application entitled System, Apparatus, Method and Article of Manufacture for Evaluating the Quality of a Transmission Channel Using Voice Recognition Technology, filed September 1, 2000.

The landline side has: • Software : • two PCS Opera™ components that test voice quality of audio

• a PCS VQSync component that synchronizes multiple incoming Opera scores

• two data quality components that tests data quality over a TCP/IP link

• various User Interface Objects (UIOs) to show text, graphs, status, errors, maps, etc.

• Hardware :

• four PSTN interfaces with audio-out ports

• a soundcard to plug the audio-out from the PSTN to for use with the PCS Opera components

The mobile side has :

• Software :

• two PCS Opera™ components

• a PCS VQSync component

• two data quality components

• various UIOs to show text, graphs, status, errors, maps, etc.

• Hardware :

• four wireless phones: two of each kind they offer • a Trimble Placer dead-reckoning GPS integrated into the van

• a scanning receiver

• a USB-to-serial hub that supports four RS-232 ports

• a soundcard for use with PCS Opera

• two sets of custom audio cables to connect the audio from the phones (earphone jack) to the soundcard. The engineer has set up the landline side to receive calls using the PSTN interface on four phone numbers: 0001, 0002, 0003, and 0004. He decides the odd-numbered lines will be for voice, and the even for data.

Outside in the van, he sets up the mobile computer as such: a gray Model A and gray Model B phone each hooked to separate audio channels, and a black Model A and a black Model B phone hooked to two of the RS-232 ports for data testing. The GPS is connected to a third RS-232 port, and the scanning receiver to the fourth.

The engineer starts Dimensionz and sets up his workspace (starts the appropriate Device Objects, Active Filter Objects, and User Interface Objects) . He configures the Opera components to dial 0001 and 0003. He configures the data quality components to dial 0002 and 0004, and to run a range of tests from "increasing-payload ping" to iMode simulation. He sets the scanning receiver to listen to every fourth channel in their frequency range. He presses "Record" in the Dimensionz Transport bar and starts driving his route .

Once the record button is pressed, all of the components are notified that a new data collection session has begun, and they are to start collecting, manipulating, analyzing data as appropriate. All at the same time, the following data is being collected, logged, and in some cases analyzed and filtered in real-time:

• the GPS is sending information such as latitude, longitude, speed, heading, and elevation

• the scanning receiver is sending 200 different streams of information

• the PCS Opera components are testing the voice quality of the line on the two gray phones

• the PCS data quality components are running through their data quality tests on the black phones

The downlink audio from the gray phones is being collected on the mobile side for later analysis back at the lab (or in real-time, if the engineer has a partner doing the driving) . The uplink audio is likewise being collected back at the lab for later analysis. Opera scores are being shown on the screen so the engineer has immediate feedback on the audio tests. Results from the data tests are being shown on the screen in columnar displays, graphs, etc. The position of the van is being plotted on the mapping component so the engineer sees exactly where he's been, where he hasn't, and can start to get an idea of where the coverage problem might lie.

The present invention differs from prior solutions in that:

• It implements all application specific functionality using add-in components. In this way, product capabilities can change by adding or removing functional components.

• It supports all data recording and post-processing functionality using one application framework. This is different from other products that separate data recording and post -processing into separate applications. This saves time by not having to perform a second step to analyze the data.

• It implements a standard set of interfaces so that functional components can communicate with the core application and application data. These interfaces enhance product stability and flexibility. The interfaces also enable the creation of a Software Development Kit (SDK) that lets 3^rd parties develop add-in components without needing access to core product code. In addition, the core product does not need to know about the internals of the add-in in order to allow it to function in the system.

• It stores information in data containers (streams) that are flexible enough for almost any kind of data.

• It can store state information from any object that can plug into the system.

• Streams can store state information from the device that generates the stream data.

• Connections to streams can store state information about the connection.

• It allows for run-time data discovery of user-defined types. This way a component can group relevant data together for a given timestamp (i.e.: longitude, latitude, and elevation) , and a component written by a different developer can still use any part of the encapsulated data

(i.e.: the elevation value).

• If there is not enough bandwidth to pass the data to the core application, components may save the large amounts of data local to the component and send only results of analysis to the core. In trials conducted, the framework described herein fulfills all the design goals for performance and flexibility. Several components have already been integrated into the system including measurement devices, data processing algorithms, and user displays. Real examples include : components to gather data from CDMA wireless phones, voice quality components, test data generating components, and user interface display components such as XY graphs, geographical maps, and text output .

The framework' s ability to correlate many types of measurement data in real-time, during data replay and during post-processing, makes it a great tool for engineers. For the first time, engineers can have a single, homogeneous view into the data they wish to analyze.

There are several advantages to the present solution, including:

• Allows software developers to adapt quickly to changing requirements by writing new components

• Intermix any variety of measurement devices

• Excellent performance

• Strict interface-based architecture and designing of effective interfaces enables the offering of an SDK • Giving Dimensionz-controlled GUI windows to components gives a consistent look and feel to the architecture.

• Good representation of measurement data using containers (streams) that store both measurement data and behavior data

• Capable of data recording, replay, and post-processing in one application

• Individual software components can be upgraded independently without sacrificing the stability of the present framework.

• The system allows engineers to develop data processing algorithms and share such algorithms with other engineers in the industry. These algorithms may take data from measurement devices and produce reports or output data to user displays.

• The present system does not interleave data in log files. Instead, it creates data streams for each type of information. These data streams are always managed independently from each other so that data access is much improved over prior solutions.

• Displays data in real-time and during replay

• Strict interface-based architecture allows for quick integration of new measurement devices. • Supports many device interface standards such as USB and RS-232.

• Easier to support with upgrades and bug fixes.

Having now described one or more preferred embodiments of the invention, it should be apparent to those skilled in the art that the foregoing is illustrative only and not limiting, having been presented by way of example only. All the features disclosed in this specification (including any accompanying claims, abstract, and drawings) may be replaced by alternative features serving the same purpose, equivalents or similar purpose, unless expressly stated otherwise. Therefore, numerous other embodiments of the modifications thereof are contemplated as falling within the scope of the present invention as defined by the appended claims and equivalents thereto.

Claims

CLAIMSWhat is claimed is:

1. In accordance with the teachings described herein, a method for testing information systems comprising the step of processing, analyzing, displaying, manipulating and recording data in real time using a framework that interfaces with one or more devices.