WO1992020059A2

WO1992020059A2 - Demonstration control system (dcs)

Info

Publication number: WO1992020059A2
Application number: PCT/US1992/003673
Authority: WO
Inventors: John P. Dawson; Jeffrey W. MCCARRELL
Original assignee: Matsushita Electric Industrial Co., Ltd.; Gain Technology, Inc.
Priority date: 1991-05-03
Filing date: 1992-05-04
Publication date: 1992-11-12
Also published as: WO1992020059A3

Abstract

The Demonstration Control System (115) provides the ability to record, annotate, and playback the graphic display generated by an application (111) in a preemptive multitasking, client-server environment. The record and playback modes of the DCS (115) are designed to permit the user to take advantage of the benefits of a client-server environment -- networking capabilities, different playback environments and multitasking at record time and at playback. The playback mode operates without the presence of the application (111). However, when implemented in conjunction with an application (111), the DCS (115) can be used as a debugging aid by performing automated regression testing on the application (111).

Description

DEMONSTRATION CONTROL SYSTEM (DCS.

Background of the Invention

Introduction

This invention relates to the recording and playback of software screen displays. Such systems allow a user to capture and play back demonstrations of software, with or without the presence of that software during playback. Other desired features include allowing a user to annotate the demonstration with visual information (text, charts and images) in separate windows, and with audio information to be played during the demonstration through an audio channel.

A major commercial use of such systems is to record and then later demonstrate features of a software product to potential customers. They can also be used as presentation devices. Charts, graphs, text, etc. can be recorded and then played back later, e.g. in a meeting.

Although various existing systems perform basic record and playback functions, the following simple example will be used to illustrate the shortcomings of these systems, particularly those designed for use with preemptive multitasking systems in client/server-based environments. In the example, illustrated in Figure 1, the user of a computer running a drawing application 1) creates a window 11 on the screen 10; 2) draws a circle 12 in that window; 3) creates a second window 13; 4) draws a circle 14 in that second window; and 5) moves the second window and circle toward the bottom right- hand corner 15 of the screen. The functions of recording and playing back this sequence of events involve capturing and causing to be reproduced some or all of the interaction among the drawing application, the operating system software and the video controller which ultimately causes information to be displayed on the screen.

Single-Taskinσ and Cooperative Multitasking Environments In certain computing environments, the operating system acts like a slave to the application by allowing it to take control and merely responding to its commands to perform certain actions, such as creating windows on the screen and drawing into these windows. Such environments are often single-tasking systems in which only one application is running at any one time, or systems which allow for a limited degree of multitasking (cooperative multitasking) in which applications cooperate with one another by giving up control periodically.

For example, the Macintosh operating system, illustrated in Figure 2, is an interrupt-driven, cooperative multitasking system. Multiple applications 21, 22, and 23 can be visible on the screen simultaneously. The user is able to switch between the applications on the screen, for example, by clicking the mouse in the desired application's window. In such situations, only one application is really in control — in the "foreground." It will either suspend its operation and relinquish control, or it will permit another application to run simultaneously in the "background," in essence giving up control temporarily. The key factor is that the foreground application is in control of the system (or, temporarily, the background application) . The operating system 24 is not in control. When certain events 26 occur, such as the user inputting information via the keyboard 27 or the mouse 28, the operating system is interrupted, though it merely queues the events and waits for the foreground application to ask for and respond to such events, or to pass them on to the background application. Only if applications choose to ignore such events will the operating system take control, and even then only temporarily. Whenever the application requires assistance from the operating system (e.g. to respond to an event) , the application issues a request by calling a particular operating system routine.

Due to this "master-slave" relationship between applications and the operating system, screen recording and playback in the Macintosh environment is a relatively straightforward process. Mere capture and reissuance upon playback of requests from the application to the operating system is generally sufficient to reproduce the visible activity on the screen. Products such as Farallon's "MediaTracks" provide examples of this functionality.

Returning to the example illustrated in Figure 1, screen recording and playback in single-tasking or cooperative multitasking environments, such as the

Macintosh environment, involves primarily recording and reissuing the requests from the application to the operating system. Figures 3-1 and 3-2 provide a simplified illustration of the requests 30 from the application, the responses 31 to those requests from the operating system and the resulting graphic displays 32 which occur when an application, in response to user input, creates the sequence of screen displays illustrated in Figure 1 in a single-tasking or cooperative multitasking environment. In essence, the application examines the event requested from the operating system resulting from the user's input, and then determines which actions should be taken. The application requests 33 the next event from the operating system, examines the information 34 regarding that event returned by the operating system and initiates an action 35 based upon the particular functions of the application in response to particular user input and/or actions. The operating system then performs that action, resulting in the modified graphic display, the creation of a window 36 on the screen 37. After creating window 36, the drawing application requests the next user event 38 from the operating system, which indicates that the user is drawing in window 36. The application, upon determining from the relevant context that the user is attempting to draw a circle, issues such a command 39 to the operating system, which causes that circle 301 to be displayed on the screen. This process continues as the application requests and receives the next event 302 and issues a command 303 calling for the creation of window 304 on the screen. The user again draws a circle in that window. Upon receiving that event 305, the application issues the command 306 to cause the operating system to display that circle 307 in window 304 on the screen.

Finally, the application detects certain events indicating that the user is attempting to drag the window 304 to the corner of the screen. Upon detecting the user's mouse click 308 on the title bar of the window 304, and the user's subsequent dragging 309 of the mouse across the screen, the application issues a command 310 to direct the operating system to move window 304 across the screen. The application will continue to issue such drag commands while the user drags the mouse across the screen until the application receives the event 311 which indicates that the user let go of the mouse button, at which point the window 304 will remain displayed at its final destination 312 in the bottom, right-hand corner of the screen.

At record time, all requests 30 from the application are recorded. On playback, the relevant recorded requests are simply resent to the operating system. During playback, the operating system will perform the same steps in the same sequence every time.

Preemptive Multitasking Environments

Other environments, such as the X/UNIX ("X") windowing environment, make screen recording and playback significantly more difficult. This difficulty is due primarily to the shift in control, from applications to the operating system, found in preemptive multitasking systems. The X environment supports networked window servers. The X server 41 in Figure 4 can interact with an application 42 and user at a remote computer, via a network link 43, in a manner which is transparent to the application software. This is a useful feature of the X environment which users of a screen recording and playback system desire to see supported.

There are several features of the X environment which do not permit an approach to screen recording and playback similar to that employed by single-tasking and cooperative multitasking systems such as the Macintosh. Merely capturing and reissuing screen-drawing requests from the application to the operating system in an X environment will not reproduce the visible activity on the screen. Figure 4 illustrates the division of labor among various portions of the operating system and applications. The Window Manager 44 is separate and autonomous from the X server 41 and is responsible for managing communication between independent applications and for maintaining a consistent user interface policy at the window level. In effect, the window manager is on the same level as other applications 43 and 42. Communication between the X server 41 and the window manager 44 involve a sequence of window graphics requests and responses 45, initiated by the X server as a result of a higher-level request from an application (e.g. create a window) or input 48 from the user via the keyboard 46 or mouse 47. Responses from the window manager go only to the X server. However, sometimes the window manager tells the X server to inform the application that a particular event has occurred.

There are at least three distinct and significant impediments to capturing and replaying a screen-drawing sequence in a preemptive multitasking environment which make the approach used in single-tasking and cooperative multitasking environments inadequate: (l) the playback environment (i.e. the screen, the X server, the window manager and/or the user's preferences) may be different from the recording environment; (2) the X server, not the application, is in control of the system; and (3) the asynchronous nature of the communication between the application and the X server and between the X server and the window manager, in environments such as the X environment, creates various timing and synchronization problems.

A different playback environment can result from something as simple as a different screen or different user preferences. In addition, the X environment is able to support different window managers and X servers. The user of a terminal or workstation is able to select a particular window manager. Similarly, there are a number of different X servers which can be used in the X environment. If the screen, the user preference, the X server and/or the window manager are different on replay, then different graphic results may occur. Windows and figures may be put in the wrong place, or they may not be drawn at all if the X server and window manager present in the playback environment do not "speak" the same "language" as did the X server and the window manager of the recording environment.

If the screen recording and playback system does not support these differences between recording and playback environments, the user of such a system is severely restricted. This is particularly true with respect to the distribution of recordings to third parties, whose playback environments may well differ from the recording environment.

The X server's control over the system is most evident with respect to interrupts resulting from events such as a user's input via the keyboard or mouse. Rather than permitting applications to respond to such events, the X server may handle them itself, via requests to the window manager, without the application's permission and/or knowledge. In response to such events, the window manager may, for example, drag a window, re-size a window, or rearrange the stacking order of windows on the screen. Although the window manager will then ask the X server to notify the affected applications of changes in the window positions (via "ConfigureNotify" events) , it may, or may not, ask the applications to become involved in the event by updating their windows as appropriate.

Record and playback systems, such as those used in single-tasking and cooperative multitasking environments, which merely record requests from the application to the X server would not be able to capture and replay those instances where changes in the visual state of the windows occurred due to requests which did not originate with the application. Looking at Figure 4, a system which recorded communication between the application and the X server would record only requests 40 from the application to the X server. If the user, via the mouse 47, moves a window across the screen, the events 48 generated by that move would be sent to the X server via the UNIX operating system 49. The X server would then initiate a dialogue 45 with the window manager, which would eventually direct the X server to move the window. The window manager would also tell the X server to notify the application (via a configure event) that its window had moved; but the screen recording system would not capture, much less understand, this configure event notification when it eventually arrived. Thus, because the application never sent a request which resulted in the window being moved, no such request would be recorded and, on playback, no request would occur and the window would not be moved.

However, at some point, the application usually receives an "expose event" (indicating that some portion or an application's window has recently been exposed or made visible on the screen) and will then correctly update the graphic display by issuing a request to the X server. This update would incorporate the fact that the window had moved. Unfortunately, the timing sequence would not be portrayed accurately on playback. The window would not appear to move when it was moved during the recording session. It would move when the window's next expose event occurred.

The chief problem in recording and playback in the X environment is the asynchronous nature of the dialogue between the X server and each application, and between the X server and the window manager. Put simply, the timing is such that requests from an application to the X server or from the X server to the window manager cannot easily be synchronized and correlated with the responses to those requests.

The asynchronous nature of the X environment results from its preemptive multitasking capabilities. The "client-server" approach places the X server in control and accommodates multiple application. The X server is a shared resource, available equally to various applications running on the same processor or on remote processors. Normally, no application is in control of the X server; instead, any application may issue a request to the X server, which may honor or reject the request. In turn, the X server may require any application to perform graphics operations on all or a portion of its windows, at any time. The X environment is not simply a passive display interface. It can interpret and reject application requests and can ask the application to perform certain tasks. There is a constant dialogue between the application and the X server as they "negotiate" what needs to be done.

The X server will ask for requests from all the applications running. However, the X server does not necessarily perform these requests in the order received. The X server may receive a request from application #1, application #2 and application #3, in that order. However, the X server may decide to respond to application #2's request first, then application #3's request, and finally application #l's request. In fact, the X server may "store" a number of requests from an application before responding to any of them and, when responding, may not respond to the requests in the order they were received. The interaction between the X server and the window manager complicates things further. The X server decides when requests need to be sent to the window manager. Once these requests are sent, the X server will continue operating without waiting for the window manager to respond. Eventually, the window manager will tell the X server how to respond to particular requests. For example, the window manager may tell the X server how to create a window. When a window is created, destroyed, or changed, the X server will notify the application of this fact by sending an expose event, requiring the application to send graphic information back to the X server regarding the contents of the relevant portion of its window. These expose event responses are critical in the X environment. Applications are not allowed to draw into a window prior to being notified that the window has been created. If an application tries to do so, its draw requests will simply be discarded by the X server. In addition, if an application requests that a window be destroyed prior to such notification, the window manager could produce undesired and unpredictable results.

Thus, until the application receives the relevant expose event from the X server, at the request of the window manager, it cannot be certain that its window has been created, and, therefore, cannot yet ask the X server to draw into that window. The application cannot rely on information generated by the X server to determine whether a window has been created. When the X server receives a request, e.g. _r to create a window, it forwards that request to the window manager. Since it no longer has the request, the X server, if queried by the application, will assume that the request has been completed, even though the window manager might not have even begun processing the request. As a result. applications must wait until the window manager directs the X server to create or change a window, which in turn may cause expose events to be sent to the application. Moreover, due to the asynchronous nature of the communication between the X server and the application and the X server and the window manager, the relevant expose event may arrive during playback at a later time than it did during recording. If so, the X server would receive recorded requests to draw into or destroy windows before it sends the relevant expose event to the application, and would either ignore such requests or produce unpredictable results. Thus the timing of the expose events will differ with each session.

More importantly, the expose events will not always occur. For example, if during recording, a corner of window A is covered by window B and window A is then brought to the top, this action will generate an expose event, indicating that the previously covered portion of Window A should now be exposed. On playback, it may take the window manager a shorter or longer time to generate the expose event. In fact, the expose event might never occur. If, for example, a slightly different screen is present during playback, window B might not cover the corner of window A. When the request to bring Window A to the top is processed, the window manager will not need to redraw window A and, therefore, no expose event will occur.

Another reason why an expose event might not occur on playback results from the nature of windows in the X environment. Each window is actually made up of sub- windows. As each sub-window is created, an expose event will be sent to the application for that sub-window. This lets the application know that it can draw into that sub-window. Once the entire window is created, an expose event for the overall window is sent. Assume, for example, that a recorded window is made up of two sub-windows, and expose events were received during the recording session for both sub-windows and the overall window. On playback, if the overall window is completed simultaneously with the sub-windows, then only the expose event for the overall window will be sent. In short, the asynchronous nature of the X environment, and of the timing of expose events, pose significant impediments to the proper implementation of recording and playback systems.

A simplified example of the recording and playback of an application session, which creates the sequence of graphic images shown in Figure 1, will illustrate the three problems, described above, which result from the use of single-tasking or cooperative multitasking recording and playback methods in a preemptive multitasking environment.

Figure 5 illustrates a simplified version of the dialogue that occurs between the application and the X server in order to generate graphics on the screen in response to the actions of the user. Assuming the user has previously started the application, and now requests (e.g. , by selecting a menu item) that a new window be created, the X server passes this event 50 to the application. The application notifies 51 the X server that a window should be created. It then waits to receive an expose event 52 which indicates that the X server and the window manager have created a window 53 on the screen 54. This same process is repeated for the second window, including the create window event 58, the create window request 59 to the X server, the expose event 501 indicating that the window 502 has been created, the draw event 503 in that window and the request 504 to draw a circle 505 in that window. At this point, the user drags window B to the bottom right-hand corner of the screen. This action does not generate dialogue between the application and the X server. The X server, being in control of the system, handles the drag event 506 (generated by the user) itself, with the assistance of the window manager. After responding to the X server's request for assistance, the window manager directs the X server to send a ConfigureNotify event 507 to the application, letting it know the new location 508 of the window 502 at the bottom right-hand corner of the screen.

Assuming a record and playback system where one merely records requests from the application, and then resends these requests on playback, a variety of things may go wrong on playback. The first, due to the asynchronous nature of the multitasking environment, is illustrated in Figure 6.

During playback, the recording is sent, request by request, to the X server. In response to the create window request 61, the first window 62 is created and an expose event 63 is sent to the playback application. In response to the draw circle in window A request 64, a circle 65 is drawn into the window 62. The playback application then sends he request to create window B 66. But, due to the asynchronous nature of the X server the expose event 67 is not received (and window 68 is not created) until after the request to draw the circle 69 is sent. Because the X environment does not permit drawing into a window which has not yet been created, the request is discarded. Therefore, when the window 68 is finally created, no circle is ever drawn into that window.

The asynchronous nature of the X environment requires that playback applications wait for expose events before drawing into windows. However, as explained above, the timing of expose events will differ on playback and, in fact, the expose events may never arrive. As demonstrated in Figure 6, a system which merely records and replay requests from the application will not operate successfully in an asynchronous environment, such as the X environment.

Figure 7 illustrates the playback problem associated with the X server being in control, rather than applications. The recorded requests 71 are sent to the X server and the circles are accurately displayed in each window 72 — window A 73 is created and a circle 74 is drawn into that window; and window B 75 is created and a circle 76 is drawn into that window.

But then the recorded transcript ends. The request to move window B at record time was not initiated by the application. The user's request to move the window was never recorded because the move was handled by the X server and the window manager. Consequently, on playback, window B 75 is never moved. Finally, Figure 8 illustrates the problem associated with different playback environments. Due to either a different window manager, a different screen or different user preferences, when window B 81 is created on playback it partially covers window A 82. The circle 83 is then drawn into window B and that window is moved to its new location 84. Then the transcript ends. A portion 85 of window A and its circle had been obscured by window B. When window B moved, this portion 85 was re-exposed. However, because this portion 85 of window A was never obscured during recording, there is no request in the transcript to redraw that portion, and thus it is essentially erased (obscured and then exposed but never redrawn) during playback.

It is evident from the above examples that a record and playback approach of the type used in a single- tasking or cooperative multitasking environment cannot be successfully implemented in a multitasking environment, such as the X environment. Moreover, other methods of accomplishing basic recording and playback functions in an environment such as the X environment also have a variety of drawbacks.

Server Emulation

One system, which takes advantage of the networking capabilities of the X environment, is XMX, an X protocol multiplexor. However, XMX is not a record and playback system. It is a utility for displaying, in real time (as opposed to recording for future playback) an X Windows session on multiple displays. XMX taps the communications link between an X client and an X server. XMX can connect to any number of X servers. One of the servers is the "master." The other servers are "slaves." The necessary protocol requests are sent to the slaves to enable them to maintain a visual look- alike of the master server's display.

Bit Map Capture

It is possible to record and replay a screen drawing session by capturing periodically the actual bit maps generated by the video controller and stored in the video RAM. However, there are serious problems associated with this method. It is simply not feasible to store a sufficient number of bit maps to reproduce the recorded visual displays. They occupy far too much space in memory.

Capturing bit maps does not prevent the use of the networking feature of the X environment at record time, but it does eliminate its usefulness. The application can still be a remote client. But the recording must occur at the X server and display terminal site. Therefore, the recording user must be able to access the remote computer and the local server and might as well run the application on the local computer.

Even if it were feasible to capture all the bit maps, the X server and the window manager present on the playback computer would have to be disabled to allow the playback application to send the low level requests directly to the video RAM, bypassing the X server entirely. This approach has at least two undesirable results.

The playback user would not be able to take advantage of the networking capability of the X environment, because these low level requests must be sent directly to the video controller. They cannot be sent over a network link. Also, no other X window applications can run while the X server and window manager are disabled. Most applications require both to be present.

Capturing Input Events Systems which capture (record) input events (mouse clicks, key strokes, etc.) such as the Sun Microsystems "journaling" product, have numerous problems. The major drawback of this type of record and playback system is that the application must be present at playback. Such a system does not attempt to "record" application requests to the operating system. It only records the user's keyboard and mouse inputs. On replay, the application must be present in order to process these inputs and generate the appropriate screen displays. However, even if the application is present, the system will not necessarily function properly if the playback environment is different from the recording environment. Even when using the same X server, window manager and display screen on playback, a recorded request to create a certain window will not always create exactly the same window on playback.

For example, during playback, the window may be shifted by a few pixels. At record time, an input event (such as a mouse click on a window to drag it) is recorded. However, on playback, the window may be shifted slightly and the mouse click location may no longer be within the window. When the recorded input event is replayed, it will have no effect and the window will not be dragged. Obviously, this problem is magnified when the playback environment is different from the recording environment.

Modifying the X Server

The primary method used to record and playback screen-drawing sessions in the X environment has been to modify the X server. Systems such as Bell Laboratories' "$Home: ovie," MIT's "X Trap" and University of Edinburgh's screen recording and playback program, are designed to avoid the three problems discussed above. However, these problems are intricately tied to the advantageous features of the X environment. In bypassing the problems, these systems also eliminate many of these features.

In general, as illustrated in Figure 9, these systems modify the X server 91 to allow the playback application 92 to intercept 93 the low level display commands 94 from the X server to the display hardware 95 via UNIX 95. As was the case with the capturing of bit maps, this method does not interfere with the networking capabilities of the X environment at record time, but it does prevent the use of these features during playback. Requiring a user to employ a modified X server is an unattractive aspect of a commercial screen recording and playback system. Customers are extremely reluctant to purchase a product which requires them to modify their environment solely for that particular product. Moreover, during playback, the user is unable to take advantage of the client-server aspects of the X environment. As illustrated in Figure 10, the X server 101 has to be disabled to allow the playback application to send the low level display commands 103 directly (via UNIX 104) to the display hardware 105. As with the bit map method, disabling the X server prevents the playback user from accessing a host computer on a network. The low level display commands 103 must be sent directly to the display controller 105. They cannot be sent over a network link.

Finally, no other X window applications can run while the X server is disabled, because such applications require the presence of both an X server and a window manager.

In essence, modifying the X server eliminates most of the distinctions between preemptive multitasking environments (such as the X environment) and single- tasking or cooperative multitasking environments. Yet, the trade-off for avoiding the problems associated with preemptive multitasking environments (such as asynchronous communication, operating system control and differing playback environments) is the elimination of the very advantages of such environments, such as recording one of multiple applications running in a user's normal environment and replaying the visual results of that recording in another user's normal (but slightly different) playback environment.

The present invention achieves the best of both worlds, by eliminating the problems associated with preemptive multitasking environments without eliminating the advantages of those environments and inconveniencing the user, either during recording or playback. Summary of the Invention The Demonstration Control System (DCS) is designed to record and playback screen drawing sessions in a client-server environment, such as the X environment, without sacrificing the benefits of the environment — networking capabilities, different playback environments, multitasking at record time and at playback time, and so forth. In addition, the DCS does not require the recorded application to be present during playback.

The DCS allows the user to specify window preferences during playback different from those specified at record time (and have those preferences honored) . In addition, no modification of the operating system environment (e.g.. the X server) is required.

Therefore, the DCS does not need to disable the X server on playback.

At record time, the DCS inserts itself between the application and the operating system/window server, without interfering with the multitasking and networking capabilities of the recording user's environment. During playback, the DCS, in effect, replaces the application, again without interfering with the multitasking and networking capabilities of the playback user's environment.

Both at record time and during playback, the DCS uses a variety of techniques to handle the problems associated with preemptive multitasking environments (the X environment in the preferred embodiment) . For example, during recording, the "bounding boxes" (visual boundaries) of expose events are enlarged to account for differences in screen size, window managers and user preferences during playback; a special connection is employed to record the cursor position; and window manager responses which change the visual state of a window, independent of the application, are synthesized and recorded so that they can be simulated on playback.

During playback, mapping tables are used to accommodate different window servers and window managers. The mapping tables are also used to assist in synchronizing the playback within the asynchronous nature of the windowing environment.

Although the preferred embodiment of this invention resolves the particular problems associated with the X environment, it should be noted that the techniques employed in this invention are equally applicable to many other environments, particularly preemptive multitasking environments, in which the mere capture and replay of an application's screen drawing requests is insufficient to reproduce the visual screen displays created by that application.

In addition, these techniques, as discussed below, can be used to debug an application during the software development process. When implemented in conjunction with an application, the DCS techniques allow the user to perform automated regression testing.

Brief Description of the Drawings Figure 1 illustrates screen snapshots generated during the execution of a drawing application. Figure 2 illustrates the Macintosh environment, including an application, the operating system, the mouse and the keyboard.

Figures 3-1 and 3-2 illustrate a simplified listing of application requests, operating system responses and the resulting graphic displays generated in the Macintosh environment.

Figure 4 illustrates the X environment, including the X server, the window manager, local and networked clients, the mouse and the keyboard. Figure 5 illustrates the dialogue among an application, the X server, and the window manager as well as the resulting graphic displays generated in the X environment (emphasizing the recorded requests from the application to the X server) .

Figure 6 illustrates the playback problem associated with the asynchronous nature of the X environment.

Figure 7 illustrates the playback problem associated with the X server's exertion of control over its client applications.

Figure 8 illustrates the playback problem associated with different recording and playback environments. Figure 9 illustrates recording in the X environment using a modified X server.

Figure 10 illustrates playback in the X environment using a modified X server.

Figure 11 illustrates the logical connections employed by the DCS to record an application in the X environment.

Figures 12 illustrate a simplified transcript representing significant elements recorded by the DCS, and the corresponding graphic displays produced by the recorded application.

Figures 13-1, 13-2, 13-3 and 13-4 illustrate a flow chart of the playback methodology.

Figure 14 illustrates the playback of a transcript recorded by the DCS, including the corresponding graphic displays generated thereby. Detailed Description of the Preferred Embodiment (s1

Record Mode Connections

At record time, the Demonstration Control System is inserted into the link between the application to be recorded and the X server. As illustrated in Figure 11, the application 111 is usually connected 112 to the X server 113 via "socket" 114. However, in order to record the visual displays generated by that application, the DCS 115 connects 116 to the X server 113 through socket 114. The application 111 is connected 117 to the DCS 115 through another socket 118. An additional link 119 is set up between DCS 115 and the X server 113, via socket 114, to record the cursor movements captured by the X server.

Preliminary Steps

The "save-under" feature of the X server is disabled because, during recording, a window might be obscured and then exposed. If the X server handled the updating of that window in accordance with the "save- under" feature, it will have stored the update information itself and would not need the application to redraw the window. The update information would not be communicated between the application and the X server and, therefore, would not be recorded. During playback, when the window is obscured and then exposed, if the playback X server did not use the "save-under" feature, DCS would get an expose event and would be asked to redraw a region for which it did not record the contents. Disabling the "save-under" feature forces the application to redraw the windows at record time and allows DCS to record the necessary data to reproduce the appropriate visual displays during playback. All special screens and non-default visuals are disabled, as they are unique to the particular X server used. The application might otherwise take advantage of certain screen capabilities at record time that could not be reproduced at playback time due to the presence of a different X server.

A special connection 119 is set up to record the cursor position periodically. This connection is used because the application might issue requests which make it impossible or impractical for the DCS to record the cursor position on a sufficiently frequent basis. Using the special connection, the DCS pretends to be an application which constantly requests the position of the cursor.

Recording

For the most part, the DCS, during recording, captures the actual dialogue between the application and the X server and stores that dialogue in a transcript. There are, however, a few exceptions, as illustrated by the simplified transcript shown in Figure 12.

First, a time stamp 121 is attached to each line of the dialogue. These time stamps facilitate the timing of requests during playback.

Second, when an expose event 122 is sent from the X server to the application, the DCS intercepts the expose event, enlarges the actual bounding box 123, and sends the expanded version 124 of the response to the application. In other words, the DCS "lies" to the application by providing it with a larger domain in which to draw. The application relies on the larger coordinates to generate its drawing request to the X server. The X server, however, will clip off any portion of the requested drawing which exceeds the actual bounding box, thus producing no visual differences, or "side effects," on the screen during recording. On playback, if the window is in a slightly different position (due to differences in the playback user's environment, such as a different screen size, window manager or user preferences) , the larger recorded coordinates will enable the playback application to compensate for the change.

Finally, requests which are generated by the window manager, but not in response to a request from the application (e.g. requests in response to keyboard or mouse events 125) , will not appear explicitly in the transcript, but may generate visual changes on the screen (e.g. , dragging a window) . The DCS detects that such visual changes occurred via the "configure" event 126 sent by the X server to the application. The DCS determines how to simulate the actions which generated that configure event by creating, and inserting into the transcript, a synthesized command 127 to the X server which, during playback, will produce the same visual display as was generated during record time by the user, the X server and the window manager without the application•s participation.

Playback Mode As illustrated in the flow chart shown in Figures 13-1 to 13-4, the DCS initially interrogates 131 the X server to determine the playback environment, including various characteristics of the particular X server and window manager present at playback time. The type of window manager cannot be determined directly by the DCS because an application (in this case the DCS) cannot directly communicate with the window manager. So the type of window manager is determined by examining certain properties set by the window manager which are found in the X server. If the playback X environment is incompatible 132 with the recording (e.g. the recording is color but the playback X server only handles monochrome) , playback is terminated 133. Otherwise 134, a connection for the window manager "flushing window" is set up 135 and the flushing window is created 136. This window is barely visible on the screen, and is created for use in the "window manager synchronize" procedure 1310, discussed below. The cursor connection is then set up 137 for frequent replay of the recorded cursor movement. Mapping tables are set up for the window identifiers, color identifiers, resource identifiers and expose events. Part of the mapping includes a process for determining how to identify window coordinates.

Different window managers require different coordinates (e.g. some window managers include their banner when discussing coordinates while others do not) . The mapping tables are used as "translators." They map the recorded information to the information obtained during playback in order to reference the proper windows, resources and other elements of the X environment. For example, a window might be identified as "window 33" at record time and "window 92" at playback time. Information is stored in the mapping tables and enables the DCS to translate between the recorded information and the playback information. Recorded information is placed into the mapping tables as it is read from the transcript. The DCS sets up a connection to "listen" to the X server and fills in the playback information as it is received.

The DCS then begins "replaying" the transcript file 138. Different recorded events cause different actions to be taken. When a "connection" request 139 is read, the DCS sets up a new connection and the appropriate mapping tables 1301 are generated.

When a graphics request 1302 (as distinguished from a window request) is read, the DCS translates (using the mapping tables) and forwards that request 1303 to the X server.

When a window request 1304 is read, the DCS processes the request 1305 to determine the proper format required by the playback window manager, and then forwards the translated request to the X server.

When an expose event is read 1306, the DCS checks the expose event mapping table 1307 to see if the equivalent expose event response has been received on playback. If it has been received 1308, then the DCS continues to read the transcript. If it has not been received 1309, then the DCS uses the "window manager synchronize" process 1310. This process uses the X server synchronizing request and a specially created window manager synchronizing request to determine when it is permissible to draw into, or destroy, a window.

The "X synch" request is a request in the X environment which will cause the X server to return a response to the application (in this case the DCS) when the X server's request queue is "empty" (i.e. , devoid of unprocessed requests) . However, this does not cover the situation where the request has been delegated to the window manager. In other words, the X server's indication that its request queue is empty does not exclude the possibility that the X server delegated a request to the window manager, which is still, or has yet to begin, processing that request.

Thus, some form of window manager synchronizing request must also be used. This request is one which, when received by the X server from the DCS, will be sent to the window manager. Window managers process requests in sequence, i.e.. on a "first in first out" (FIFO) basis. Thus, the synchronizing request will not be completed until the window manager has completed all pending activity.

The DCS sends the window synchronizing request, and then waits for one of two events: either (l) receipt of the expose event; or (2) an indication that the synchronizing request has been processed. In either case, the DCS will know that it may now draw into, or destroy, the window.

The DCS requests the procedure "SetWindowSize" as its synchronizing mechanism. This procedure restacks the flushing window on the bottom of the "stack" of windows, after which the X server issues a configure event to the application (the DCS) , indicating that the request has been processed.

The window manager synchronize process has at least two advantages. First, the DCS will not wait longer than necessary for the expose event to occur. Second, if the expose event will never occur, the DCS will recognize this fact relatively quickly, i.e.. upon receiving an indication that the synchronizing request is completed. As the time stamps are read, the DCS checks to see whether the playback is on time, is late or is early. If it is on time or late 1312, the DCS continues to read the transcript. If the playback is early 1313, the DCS waits for the proper time, and then continues reading. When a MapWindow request is read, the DCS makes entries in the window identifier table 1315, listing the old and new window identifiers.

When an AllocateColor request is read 1316, the DCS makes entries in the pixel identifier table 1317, listing the old and new pixel identifiers. When a DestroyWindow request is read 1318, the DCS follows the same "window manager synchronize" procedure 1319 used for expose events. Otherwise, if an application tries to destroy a window before it was created, the window manager would produce unpredictable results.

Figure 14 illustrates some of the aspects of playing back the recording illustrated in Figure 12. After the initial connections are made and the mapping tables are set up, the DCS will read the transcript line "Create Window A" 140. Processing is done on the request to make it compatible with the local window manager, at which point the revised request 141 is sent to the X server. Next the DCS reads expose event 142. The expose event mapping table is checked to see if the corresponding expose event 143 has been received on playback. If it has been received, then the DCS knows window A 144 has been created, and will continue reading the transcript. If it has not been received, the DCS will initiate the window manager synchronize procedure. Then, when either the expose event 143 or the flushing window configure event 145 is received, the DCS will know that the window 144 has been created and will continue reading the transcript.

The DCS continues reading and processing the transcript. As the playback proceeds, a circle 146 is drawn in window A, window B 147 is created and a circle 148 is drawn in window B. The graphic display of these events is comparable to that which occurred at record time.

At the point where, in Figure 12, the user clicked on the mouse 125 during record time and caused window B to be dragged across the screen, the playback system will encounter the synthesized command in the transcript "Update window B in new location" 149. This will result in a movement of window B 147 to the bottom right-hand corner 150 of the screen.

The end result is that, at all times, the evolving graphic display 151, 152, 153, 154 and 155, virtually mirrors that which occurred at record time.

Annotate Mode The annotate mode allows the user to modify the playback speed and to add visual or audio annotations anywhere in the transcript (recording) .

The user may use the annotation mode to adjust the timing of the playback. This allows the user to lengthen and shorten the playback time of the transcrip . The recording is lengthened by inserting a command which will delay the replay of the rest of the transcript from the point at which the command is inserted. The recording is shortened by inserting a command which will subtract time from the time indication on every event in the transcript from the point at which the command is inserted.

Finally, the user is able to terminate the transcript at any point in the recording. This will effectively discard the remainder of the transcript. Visual annotations may be added to the transcript. Commands may be inserted into the transcript which: (1) add a pointer to the graphic display; (2) highlight a region of an application window; or (3) insert a separately prepared graphics slide or text slide. Audio annotation may also be added to the transcript. A command is inserted into the transcript which will cause a named audio file to play at the point of insertion. The audio tool is a means for allowing the creation and simple editing of audio recordings on the workstation.

Regression Testing The DCS also includes means for performing automated regression testing of applications. For example, by capturing a user's input events, as well as the applications' responses to those events, during the software development process, the DCS can resend those input events to a subsequently modified version of the application. It can then automatically detect whether the modifications introduced variations in the application's responses, such as errors, corrections or other intentional or unintentional changes in behavior. To perform automated regression testing, the DCS captures communications, including input events, between the application and the X server, as it does during the recording process described above. During testing of the modified application, the DCS is inserted into the link between the modified application and the X server, as it was during the capture/recording process, and as previously illustrated in Figure 11.

The DCS simulates the user, in part by resending the captured input events to the application at the appropriate time, as described below. It then compares the captured display requests sent by the original application to those sent by the modified application (in response to the simulated input events) to detect any variations in the application's behavior; such variations may or may not be reflected on the screen. The application tester can therefore view the screen displays generated by the modified application while receiving messages as variations in the application's behavior are detected. In one embodiment, the X server need not even be present. The DCS resends to the modified application the recorded user inputs and communications from the X server to the original application. It then merely compares the modified application's responses to those captured during the recording process.

In another embodiment, the X server is present and the DCS allows the modified application to generate screen displays by passing application requests through to the X server and vice versa (as is done during the capture/recording process) . However, the DCS must still simulate the user by sending the captured input events to the modified application at the appropriate time, and by sending to the X server the commands synthesized during the recording process, as described above. These synthesized commands cause the X server to simulate actions performed during the capture/recording process (in response to user input) which did not require the application•s participation. As was the case during the playback process described above, the DCS must map various graphic IDs and resources in order to translate the information generated during the capture/recording process into the "language" used during testing. In addition, mapping tables are used to compare the modified application's requests with those captured during the capture/recording process.

In yet another embodiment, the DCS functions as a playback mechanism, as described above, in order to allow for automated regression testing in different environments. In addition to communicating with the X server as part of the playback process, the DCS passes information received from the X server to the modified application. The modified application uses this information, along with the recorded inputs, to generate display requests in the "language" of the testing/playback environment. These requests are then compared to the playback requests generated by the DCS and sent to the X server. In all three embodiments, it is critical that the DCS send the captured input events to the modified application at the appropriate time. For example, if the DCS sends the user's response to a question prior to the time at which the modified application displays that question, the modified application may well ignore the user's response, as it would if the user had actually typed its response before the question was displayed.

The DCS uses the recorded transcript, including the captured requests generated by the original application, to determine when to send the captured input events to the modified application. In essence, the DCS sends a captured input event to the modified application once it determines that the dialogue between the modified application and the X server has reached the stage where the input event was sent to the original application, as reflected in the transcript.

Thus, if, prior to the user selecting a menu item, the X server sent an expose event to the original application, in response to which that application sent two display requests to the X server, the DCS will not send the input event representing the user's selection of the menu item to the modified application until the corresponding dialogue occurs during the testing process.

Claims

We claim:

1. An apparatus for reproducing the visual screen displays generated by an application executing in a preemptive multitasking environment, said environment comprising a computer, a preemptive multitasking operating system, an application and a screen for displaying information, including: means coupled to said computer for capturing at least one request from said application to said operating system, said request causing a first visual display on said screen, and means coupled to said computer for sending requests to said operating system while said application is not present in said environment, said request causing a second visual display on said screen, wherein said first and second visual displays are substantially identical.