WO1993008522A1

WO1993008522A1 - Computer-based drawing apparatus

Info

Publication number: WO1993008522A1
Application number: PCT/IE1992/000019
Authority: WO
Inventors: Patrick Rainsford; Peter Conlon; Karl Pawley; Richard Conway
Original assignee: Timesavers Limited
Priority date: 1991-10-21
Filing date: 1992-10-20
Publication date: 1993-04-29
Also published as: AU2788692A

Abstract

A drawing apparatus which, when connected to a like apparatus over a communication link, permits real-time interactive communication of image information includes a processor (10) having a display device (12) and a pointing device (13) operatively connected thereto such that an image may be drawn on the screen of the display device by moving the pointing device. The apparatus further includes means (17, 18, 21, 22) for transmitting movements of the pointing device to a like apparatus at a remote location and for receiving movements of the pointing device associated with the remote apparatus. The apparatus is adapted to display on the display device (12) the said received movements of the remote pointing device such that the movements of both pointing devices are displayed on a common area of the screen whereby the image may contain components derived from both pointing devices.

Description

Description Computer-based Drawing Apparatus

Technical Field

This invention relates to a drawing apparatus which, when connected to a like apparatus over a communication link, permits real-time interactive communication of image information and which, in a preferred embodiment, also permits oral (voice)

communication. Disclosure of Invention

According to the present invention there is provided a drawing apparatus including a processor having a display device and a pointing device

operatively connected thereto such that an image may be drawn on the screen of the display device by moving the pointing device, the apparatus further including means for transmitting movements of the pointing device to a like apparatus at a remote location and for receiving movements of the pointing device associated with the remote apparatus, the apparatus being adapted to display on the display device the said received movements of the remote pointing device such that the movements of both pointing devices are displayed on a common area of the screen whereby the image may contain components derived from both pointing devices. Preferably the means for transmitting and receiving includes a modem for communication with the remote apparatus over a telephone line. The apparatus may further include-audio circuits for the transmission of speech signals to and the reception of speech signals from the remote apparatus, and means for switching between a voice mode for the transmission and reception of speech signals and a data mode for the transmission and reception of pointing device movements.

In such case it is preferred that the switching means is switched to the voice mode automatically on detection of speech by the user of the apparatus.

The apparatus preferably includes a transmit buffer for storing, at least during the data mode, data defining movements of the pointing device intended for transmission- the pointing device movements being transmitted as relative and/or absolute screen

addresses.

The data defining the pointing device movements may be stored in the transmit buffer also during voice mode and transmitted when the switching means next switches the apparatus to data mode.

Alternatively, when the apparatus is in voice mode each pointing device movement can be transmitted by briefly and temporarily entering the data mode long enough to transmit the related screen address. However, since this tends to break up the speech it is preferable to limit it to sending move information without an accompanying draw, enabling a speaker to point occasionally to features of the image under discussion without modifying it. In such case drawing movements continue to be sent in the data mode proper. However, it is possible, if one were to accept reduced quality speech to have permanent speech mode and send all pointing device movements (both move

only and move and draw) this way, i.e. by briefly

entering data mode long enough to send the related address and an attribute defining move or draw.

Brief Description of the Drawings

Figure 1 is a block diagram of the apparatus of a first embodiment of the invention,

Figure 2, comprising transmit section figure 2a and receive section 2b, is a flow diagram of the program controlling the operation of the apparatus of figure 1,

Figure 3 is a modification of figure 2a

providing a modified form of the first embodiment,

Figure 4 is a block diagram of the apparatus of a second embodiment of the invention,

Figure 5 illustrates the structure of the software for the processor 10 of figure 4, Figure 6 shows how the software in the processor

10 of figure 4 breaks down individual lines or pen strokes into multiple line components, and

Figure 7 shows how such line components are stored in memory. Description of the Preferred Embodiment

Embodiments of the invention will now be described, by way of example, with reference to the accompanying drawings.

The first embodiment will now be described in terms of a communication system in which two mutually remote users, each having an apparatus as shown in figure 1, communicate with each other by image and speech interactively in real-time over a telephone line.

In the first embodiment, each user's apparatus includes a microprocessor 10 with memory 11, to which are operatively connected a display device 12 and a pointing device 13. The display device 13 may be an LCD display, a conventional TV or a monitor. However, in this embodiment it is assumed that the display device 12, together with the microprocessor 10 and memory 11, are components of a standard commercially available personal computer, for example a DOS- or Windows-based IBM PC or PS/2. However, other arrangements are

possible, as will be described later. The pointing device 13 is preferably a pen/tablet device because it provides the most intuitive and interactive interface for sketching and writing text, but other devices such as a mouse or trackball may be used. The memory 11 has transmit and receive registers 15 and 16 respectively. A modem 17 is provided to transmit and receive digital data across a telephone line 18, which is

connected to the like modem 17 of the other user's apparatus. The modem 17 is used when the apparatus is in data transmit/receive mode, to send data to and receive data from the remote apparatus, including data defining pointing device movements. Each apparatus also includes a microphone/ earphone headset or telephone handset 19. Audio

circuits 20 are provided to amplify and condition incoming and outgoing speech signals during voice

communication when the apparatus is in voice

transmit/receive mode. They also detect when the local user is speaking and notify the processor 10 which sets a voice detect timer (counter) as will be described. A data/audio switch 21 is controlled by the processor 10 according to the state of the voice detect timer to select between activating the modem 17 circuits onto the telephone line 18 or the audio circuits 20 onto the telephone line 18, i.e. to switch between data and voice modes. Switching between data and voice modes can also be initiated from the remote apparatus. A line

interface 22 provides the isolation and control that the various telephone companies require.

In operation, the processor 10 initially clears the display screen of the display device 12 and waits for the user to enter a telephone number or select a prestored number to dial. The apparatus then pulse dials or tone dials. When the remote user answers, the apparatus is initially placed in the voice mode, so that both users talk to each other. It is possible to have the apparatus initially placed in data mode, whereby the apparatus will attempt to communicate with the remote apparatus by looking for a) carrier detect, b) modem speed settings (using CRs) and c) ID code.

On the screen of each display device 12 each user is presented with a drawing (image) area and a smaller menu/command area. Each display screen has two different cursors, one indicating the position of the local user's pointing device 13 and the other indicating the position of the remote user's pointing device. When one user draws a line, arc, etc on his display screen, it is immediately, subject to inherent

transmission delays and providing the apparatus is then in the data mode, appears on the other user's display screen and vice versa. This is accomplished by transmitting cursor movement data via the modem 17 across the telephone line 18. Typically, the drawing comes in short bursts and hence most of the time the users

communicate in voice mode. Switching between the data and voice modes is effected by electronically switching out the modem 17 and switching in the audio circuits 20, as described above.

Since a user speaking will force the system into voice mode, in this mode the system stores each user's drawing movements and these will be exchanged the next time the data mode is entered.

The processor 10 services all the components in the system. When idle, the system is in data mode. The event of a user talking is signalled to the processor and it switches the audio circuits onto the line and instructs the remote user's processor to do likewise. It will also store the user's drawing activity for transmission.

The software protocol, described later, dealing with transmission and reception in the data mode, is controlled by the processor 10. It monitors the speed of the pen or other pointing device and decides which of three address methods to use.

To summarise, therefore, in a system comprising two sets of apparatus shown in figure 1, basically there are two pen cursors displayed on the display device 12 of each apparatus, one pen cursor associated with the local user's pen (pointing device) 13 and the other with the remote user's pen. Each apparatus draws its own pen movements on its own display screen and also sends these movements in the protocol form to the remote apparatus. When the pen movements arrive at the other apparatus that apparatus moves or draws using the second pen cursor. Thus two different pen cursors are maintained on the display screen of each apparatus and each

apparatus controls only its own pen cursor. The

important thing is that the two cursors draw on a common area of the screen at each apparatus, so that a common image is present at each apparatus which contains or can contain elements or components derived from both the local and remote pointing devices.

The drawing area is restricted to a certain area of each display screen to enable reserving an area of the screen for menus which are used to control the apparatus . These menu driven controls are associated with dialling (pulse or tone), answering, hanging up and clearing the display. The menu may also be used to initiate non-drawing data communication between the local and remote apparatus, for example sending text or sending a previously drawn and stored image for further modification by the users. The menu driven approach allows for future expansion capabilities of the

apparatus. The menu options are selected by pressing the pen 13 down on the selected item. The menu is located on the right hand side of the display screen and can not be written upon. When data movement is received, the second pen cursor is moved to the new location and if the sending pen is down (i.e. the pointing device at the sending apparatus is in a draw mode) a line is drawn to this new point, but if the sending pen is up (i.e. in a move only mode) no such line is drawn. The apparatus breaks the movements into the smallest possible lines depending on how fast the pen is moving (1 byte, 2 byte or absolute address).

The modem is assumed to operate at 300 baud. A higher baud rate would reproduce curves more accurately. However 300 baud works quiet well, as a two byte

relative address represents a relatively fast movement and still sends nearly 15 samples per second.

The protocol details will now be described.

The data is transmitted serially in 8 bit bytes with 1 start and 1 stop bit and no parity. The system will recognise normal ASCII in the pen up condition in the range 20H to 7FH.

The scheme was designed to recognise the following: a) Text information in the form of ASCII. b) Pen up/down information. c) Absolute address information. d) Relative address information, (delta) e) Clear page information.

The following codes were arbitrarily chosen for commands as they are out of the ASCII information band. 10H---- -Pen down

15H---- -Pen up

1FH---- -Clear display Three forms of address information are used namely: single byte delta

double byte delta

three byte absolute address

The three different ways are used to speed up the information transfer. The two most significant bits of the byte determine the type of information being sent. The break down of their significance is as follows.

Msb 2Msb 0 0 This is their state for ASCII and command bytes.

0 1 Two cases: a) When pen up then this byte is an

ASCII character, b) When pen down then this byte is a single-byte delta.

1 1 This signifies that a two byte relative address is being sent. This byte and the following byte.

1 0 This signifies that a three byte absolute address is being sent. This byte and the next two bytes.

Single Byte Delta (the pen is down)

The three least significant bits relate to the magnitude of the delta movement in the x direction, the most significant of the three being the direction indicator.

Similarly the next three significant bits apply to the magnitude of the delta movement in the y

direction, again the most significant of the three being the direction indicator. In both cases 0 is positive and 1 is negative. Double Byte Delta (pen is up or down)

The bytes will take the following form: 11xxxxxx Ddyyyyyy. The two most significant bits of the first byte indicate that a two byte delta is occurring. xxxxxx is the six bit x magnitude. yyyyyy is the six bit y magnitude.

D is the direction of the x magnitude. d is the direction of the y magnitude.

Three Byte Absolute Address (pen is up or down)

The bytes will take the following form: 10xxxyyy 00XXXXXX 01 YYYYYY where XXXXXX are the six least significant bits of the absolute address of the x coordinate of the pen, YYYYYY are the six least significant bits of the absolute address of the y coordinate, xxx are the three most significant bits of the coordinate and yyy are the three most significant bits of the y coordinate.

As can be seen from the three byte absolute address there is room for extra resolution in the two most significant bits of the following two bytes. Using this limited repertoire of addressing and commands the penphone can be implemented. Alternate Resolutions.

The nine bits of absolute address data ensure a 512 by 512 drawing capability, ignoring the reserved menu area. The two most significant bits of the third absolute address byte could be used to signal other resolutions eg:

0 0 256x256

0 1 512x512

11 00 1024x1024 and use the two most significant bits of the second absolute address byte. One bit for x and one bit for y (least significant bits).

1 1 2048X512 and use the two most significant bits of the second absolute address byte (for x or for y) Or perhaps for some other scheme. (625/512 etc).

Figure 2 is a flow diagram of the main loop of the program controlling the operation of the processor 10. It is assumed that another user at a remote

location has already been called up and has answered, and that both sets of apparatus have been placed initially in voice mode.

Figure 2a shows the transmit section of the flow diagram. The program enters the loop at the decision step 100 and because it is assumed the apparatus is initially in voice mode the program will move to step 101. However, it is more convenient to describe the operation first in data mode and then deal with

switching to and from voice mode. Thus it will be assumed that this is not the first time round the loop and that the apparatus is in the data mode i.e. the "NO" branch is taken at step 100. However, if the pen condition and/or the cursor position has changed, control passes to step 102 where the pen (pointing device 13) is interrogated for being up or down. Assuming it is down step 103 determines if its cursor is in the drawing area of the screen and if so step 104 draws the line on the screen (i.e. the local user's screen) from the previous cursor position to the new cursor position, and updates the cursor position on the screen. Now the apparatus has to update the cursor of the remote apparatus, and it does this by first putting a pen down command in the buffer at step 105, to tell the remote apparatus that this is a draw and not just a move, and then sending either a delta address (i.e.

offset from the last cursor position) or absolute address to the other apparatus, depending on how full the transmit buffer 15 is. In particular, step 106 determines if the transmit buffer 15 is nearly full, say 90% or more full, and if so puts a three-byte absolute address into the transmit buffer 15, step 107.

Otherwise, at step 108 it determines whether the buffer 15 is more or less than half full, and if the former it puts a two byte delta address into the buffer 15 (step 109) and if the latter it puts a one byte delta address into the buffer 15 (step 110). The nature of the pen down command and the addresses is as stated earlier.

Returning now to step 102, if the pen is up the cursor position is updated on the local user's screen (step 111) but a line is not drawn. This too has to be transmitted to the remote apparatus, so first a pen up command is put in the transmit buffer 15 at step 112 followed by either a two byte delta address at step 114 or a three byte absolute address at step 115, depending upon how full the buffer is as determined at step 113. When the pen is up the single byte delta address is not used - it will be recalled from the earlier description that when the pen is up the single byte is reserved for ASCII.

This deals with the movements of the pen when the pen cursor is in the drawing area of the screen. If, however, it is in the menu area when the pen is down, this is detected at step 103, whereupon control passes to step 116, where the command pointed to is executed.

This may be a purely local command, i.e. applicable only to the local user's apparatus, or it may require a command to be sent to the remote apparatus, for example to condition the latter to accept a new image or to receive ASCII. In the latter

case the appropriate command is put in the transmit buffer 15 at step 117.

The transmit and receive buffers 15 and 16 are driven by timed interrupt service routines which operate below the level of the flow diagram shown. These

routines periodically send data from the buffer 15 on a FIFO basis to the modem 17, where it is sent over the telephone line 18 to the modem 17 of the remote

apparatus, and thence into the receive buffer 16 of the remote apparatus. The rate of sending data over the modems 17 can be slower than the rate of filling the transmit buffer 15, particularly when a lot of drawing activity is taking place. Figure 2b shows the receive section of the flow chart, and in order to follow the logical flow it is easier to describe the operation of the receive section in relation to the remote apparatus which is receiving the transmitted data, considered now as the local apparatus. Of course, the same thing applies to the sending apparatus when it is receiving, since both sets of apparatus are assumed to be similarly programmed and cycle around equivalent program loops.

Referring now to figure 2b, at step 120 the program interrogates the receive buffer 16 to see if new data has arrived, and if not control passes to step 121 which will be described later. Step 122 determines if there is a command in the receive buffer 16, and if the answer is yes the program moves to step 123. Steps 123 and 124 are concerned with switching to data mode from voice mode, and will be described later. However, it is sufficient to say for the moment that because the apparatus is assumed to be already in data mode the answer to both these questions will be NO; thus control passes to step 128. Likewise, steps 128 and 129 are concerned with switching from data mode to voice mode, and these also will be describe later. For the moment it is assumed that the apparatus remains in data mode and the answer to both these questions will also be NO, and thus control passes to step 132, where the appropriate

command (not being concerned with switching modes) is executed. This command may, for example, be to condition the apparatus to receive and print ASCII characters on the screen.

If there is not a command in the receive buffer 16 at step 122, step 133 determines if there is an ASCII character in the receive buffer 16, and assuming for the moment that there is not, then the only other data that can be in the receive buffer 16 is address data relating to the movement of the remote pointing device. Depending whether the remote pen is up or down, step 134, either a line is drawn on the local screen to the new cursor position of the remote pen, step 135, followed by an update to the cursor position of the remote pen on the local screen, step 136, or the cursor position of the remote pen is simply moved to the new position without drawing a line by omitting step 135.

In the case where step 133 determines that there is ASCII in the receive buffer (pen is up, single byte) , step 137 prints the corresponding character on the appropriate area of the screen reserved for text, or on the drawing itself at the current cursor position of the sending apparatus, depending on the nature of the prior ASCII command. Thus, depending upon whether movement data (an address) was received, or ASCII, or command, the program arrives at step 121. Since the apparatus is in data mode the program passes to step 140. It will be recalled that when the user speaks, the detection of this speech by his own local audio circuits 20 is signalled to the processor 10 which sets a voice detect timer (effectively a counter) to a

predetermined maximum count. This is accomplished by an interrupt service routine, which also decrements the count at predetermined intervals. Thus, all the time the user is speaking, and providing there are not too long pauses between words or sentences, the timer count will be maintained at or near the maximum. However, should the user cease to speak for any length of time the voice detect timer will decrement to zero. Thus the voice detect timer is the means by which the apparatus determines whether or not the local user is speaking, and as will be described is the means which initiates a switch to voice from data mode and vice versa.

Thus step 140 determines whether the voice detect timer is zero, in other words, has the local user spoken ? Assuming that he has not, the program loops back to step 100 (figure 2a) and the whole cycle repeats while the apparatus remains in data mode, except that steps T05 and 112 can be omitted if the pen condition (up or down) has not changed since last time. This applies to both sets of apparatus, assuming neither of the users is speaking.

However, let us now assume that one of the users speaks into the headset 19 while his apparatus is in data mode, and we will now consider that to be the local apparatus. When the program reaches step 140, this time the voice detect timer will not be zero, so the program branches to step 141. The protocol is that when one user wants to switch to voice mode, by speaking into his headset 19, his apparatus puts a Voice Mode Command in the transmit buffer 15 which is eventually transmitted to the remote apparatus depending on how much data there is ahead of it in the buffer 15. The remote apparatus switches to voice mode in response to this command, by activating the audio circuits 20 onto the telephone line 18, and sends back a voice mode command acknowledgement (Voice Mode Command ACK) to the first user. When this is received by the first apparatus it too switches to voice mode. Clearly, once the apparatus has put a Voice Mode Command in its transmit buffer for transmission to the remote apparatus it will not immediately receive back the Voice Mode Command ACK, and several cycles of the program loop can occur while it is waiting. However, during this wait it does not need to send another Voice Mode Command; thus, returning now to step 141, this simply determines whether the apparatus has already sent a Voice Mode Command to the remote apparatus and is waiting for the Voice Mode Command ACK. If it has the program loops back to step 100 (figure 2a) whereas if it has not it places the Voice Mode Command in the transmit buffer 15, step 142 and then loops back to step 100. When the remote user receives the Voice Mode

Command in his receive buffer 16, this will be detected at step 129 (the answers at steps 123 and 124, which are concerned with switching from voice to data mode, and at step 120 will be NO). The program then puts a Voice Mode Command ACK in its own transmit buffer at step 130 and switches to voice mode at step 131. However,

because the switch to voice mode at step 131 will

inhibit servicing of the transmit buffer 15, either step131 is delayed sufficiently to clear the buffer or the Voice Mode Command ACK is put ahead of other data in the buffer 15 so that it is immediately transmitted. The local user receives the Voice Mode Command ACK in its receive buffer 16, detects this at step 128, and

switches to voice mode at step 131.

At this point both sets of apparatus are in voice mode, and both users are able to speak to each other over the telephone line. During voice mode, the program in each apparatus will continue to cycle round the loop as described above, except it will branch to step 101 after step 100. However, while each user continues to speak without too long pauses the

respective voice detect timer will not be zero at step 101 so the program will go back to step 102. During voice mode the transmit buffers 15 are not serviced by the timed interrupt service routine, so that any data intended to be transmitted by one user or the other is added to the contents of the respective transmit buffer 15 until it is full.

If one of the users now stops talking for long enough for the voice detect timer to go to zero, this will be detected at step 101, meaning that at least that user has finished speaking and the apparatus can

potentially revert to data mode, depending on the other user. The protocol, in this case is similar to the switch from data mode to voice mode, in that when a user's voice detect timer is zero the apparatus

transmits a Data Mode Command to the remote apparatus and waits for a data mode command acknowledgement (Data Mode Command ACK) to be returned before actually

switching to data mode. In this case, however, if the voice detect timer at the remote apparatus is not zero, no Data Mode Command ACK is returned so both sets of apparatus stay in voice mode.

Thus step 101 determines when the user has stopped speaking long enough for his voice detect timer to become zero. Step 150 determines if the apparatus is waiting for a Data Mode Command ACK to be received from the remote apparatus, meaning that a Data Mode Command has previously been transmitted to the remote apparatus. Assuming that this is the first time the voice detect timer has reached zero, the answer at step 150 will be NO, so a Data Mode Command is transmitted at step 151. Since in voice mode the apparatus cannot transmit through the transmit buffer 15, the modem 17 is placed briefly and temporarily on-line for only as long as it takes to transmit the Data Mode Command, which by-passes the transmit buffer 15. When the modem 17 at the remote apparatus detects the Data Mode Command it puts the command into its receive buffer 16. In this connection it is to be understood that the modems 17 are at all times able to receive data from the other apparatus even though in voice mode they do not send data from the transmit buffer 15.

The Data Mode Command is detected at step 124 at the remote apparatus, but before switching to data mode it first checks to see whether its own voice detect timer is zero (step 125). If it is zero, it transmits a Data Mode Command ACK to the first apparatus at step 126, again putting its modem 17 temporarily on-line and bypassing the transmit buffer 15. Then it switches to data mode proper at step 127. The first apparatus detects the Data Mode Command ACK at step 123, and switches to data mode at step 127. Both sets of

apparatus are now in data mode again. However, if the voice detect timer of the remote apparatus is not zero at step 125, then the user is still speaking and not ready to enter data mode. In this case nothing is sent to the first apparatus on that cycle and both sets of apparatus remain in voice mode.

In a case where the first apparatus has sent a Data Mode Command and is waiting for the Data Mode

Command ACK from the remote apparatus, ie a YES at step 150, it does not wait indefinitely. When the Data Mode Command is transmitted a timer is set to a predetermined value defining the maximum time that the first apparatus will wait for a reply to its Data Mode Command. This timer is interrogated at step 152. If the wait period is timed out, then the local voice detect timer is set, step 153, and the local user cannot request a switch to data mode until his voice detect timer goes to zero again.

Figure 3 shows how the transmit section of the program show in figure 2a can be modified to provide a modified operation of the apparatus. The receive section of the program remains the same.

Most of figure 3 is the same as figure 2a, and the same reference numerals have been used. The

difference is that in figure 3 steps 101, 150, 151 and 152 all lead to additional steps 160 and 161, rather than back to step 102. Thus when the apparatus is in voice mode there is no buffering of data in the transmit buffer 15. Rather, when the pen is down as determined at step 160, the three byte absolute address of the cursor position is transmitted to the remote apparatus at step 161. This is achieved, like the transmission of the Data Mode command, by briefly and temporarily putting the modem 17 on-line. The remote apparatus receives this address in its receive buffer 16, and processes it like movement data sent in data mode proper. However, in voice mode, even though the sending pen is down, the remote apparatus interprets this as a move only. This facility permits a user to point at the image while he is speaking. It should only be used sparingly, however, because temporarily putting the modem 17 on-line breaks up the speech for the short time the data is being transmitted. Many refinements of the above described

apparatus are possible. For example, users could zoom in to a small area of the image and just transmit it. The smaller area means the user has to wait less time for transmission to complete.

A scanning device may also be plugged into the apparatus allowing the user to transmit images of paper documents or photographs for discussion. This leads to the important concept of a user not being required to leave his/her desk or office. He/she does not have to go to a fax machine to transmit his/her paper document. He/she can complete all his/her work while sitting at his/her own desk.

In both above modes, RL, coding would be useful to reduce the amount of information that needs to be transmitted. Another use for the system would be akin to a telephone conference. One master and many listeners could use the system with the master outlining various concepts on his screen and immediately appearing on all other users' screens. The master can give control to any of the various users. The users may be in another building or country.

Another use may be TV/film/video production. The penphone would allow remote communications to enable storyboarding, camera angles/shots, lighting setup, etc., without requiring the staff or crew to be physically in one place.

With a pen input device, signature analysis is another use for the system, perhaps allowing access to a sensitive database remotely. In the foregoing it was assumed that the

components 10, 11 and 12 constituted the parts of a standard commercially available personal computer. Such an embodiment has the advantage that it can transmit and display images of Autocad or Oread files or any other drafting or painting packages, and the ability to store screen images. However, the basic apparatus shown in figure 1 is capable of implementation in various ways. For example, the invention can be implemented by a computer connectable via a network (ethernet, FDDI, etc.) to a second computer. To enable voice

communication, audio compression hardware would be provided to reduce the data rate and allow the network to handle real-time voice communications. The visual data would already be in a compatible format, and a plug -in board would handle the audio compression/

decompression. The user's headset would plug into this board. The other peripherals would plug into the

standard ports of the computer.

Alternatively the apparatus could be constructed as a device which has a built-in LCD display along with a modem, headset and pointing device. This would be small and portable and only a telephone line would be needed to use it.

The apparatus could also be designed as an accessory which plugs into a television set or a monitor and a telephone line. A headset would be connected to the accessory along with a pointing device.

If one were prepared to dispense with voice communication one could use two standard laptop machines or standard personal computers (e.g. 286, EGA, DOS) with internal or external modems, suitably programmed to allow interactive real-time visual drawings to take place. The users would communicate by typing text into the computer.

Alternatively, and again assuming no voice communication, one could use two computers connected via a network (ethernet, FDDI, etc.), suitably programmed to allow real-time drawing and ASCII-text interaction.

The second detailed embodiment of the invention will now be described with reference to figures 4 to 7, in which the same reference numerals have been used as in figure 1 for the same or equivalent parts.

Like figure 1, the block diagram of figure 4 shows the apparatus at one end of a communication system comprising two mutually remote users each having an apparatus as shown in figure 4 communicating with each other by image and speech in real-time over a telephone line 18.

This second embodiment is based upon the use of so-called "pen computers" such as the NCR 3125. Pen computers are shaped like an A4 tablet with an LCD screen on top, and have a separate pen which interacts with the computer when brought into contact with the screen. The standard configuration does not have a keyboard, and if simple typed commands are required a keyboard is displayed on the screen and keystrokes are made by pressing the pen against the desired character. Further, the standard pen computer does not display a cursor on the screen, so that the previously described function of just moving the cursor across the screen to indicate a point of interest is not available. All pen strokes, when the pen is in contact with the screen, are draw movements.

The pen computer runs under a multi-tasking object orientated operating system such as Penpoint O/S (by GO Corporation) or Windows for Pen (by Microsoft). These operating systems may, also be run on standard 386 family PC's. In figure 4 the standard pen computer

constitutes the components in the box 200, comprising the processor 10, memory 11, LCD display 12 and pen 13. The pen computer also has a plug-in modem card 17 which is commercially available for these computers.

The box 201 shows the additional hardware components provided to permit the use of the pen

computer to communicate image and voice data to the remote pen computer. These comprise the audio circuits 20 into which the external head set 19 is connected, the data/audio switch 21 and the line interface 22. In this embodiment data/audio switching is under the control of a dedicated processor, and this is shown at 10'.

Although the additional components 10', 20, 21 and 22 are shown separately in figure 4, they are preferably provided on a further plug-in card in the pen computer 200.

Since the pen stroke handling and data/audio switching are under the control of separate processors 10 and 10' it is convenient first to describe the

apparatus in the data mode, and then consider switching to and from the audio mode. The structure for the software controlling the processor 10 is shown in figure 5. Basically it comprises an observer task 202 which is driven by a penstroke task 203 and itself drives a screen update task 204. The observer task also interacts with a communications task 205, by which local pen strokes are sent to the remote apparatus via the modem 17 and remote pen strokes are received from the remote apparatus via the modem 17. The penstroke task 203 samples the pen movement to break each pen movement into a series of straight line components which approximate to the actual curve of the pen stroke, the number and length of the components depending upon the speed of drawing. A number of typical strokes S1 to S4 is shown schematically in figure 6, the small cross dashes indicating the sample points.

Each pen stroke is stored by the penstroke task in a portion of the memory 11 designated as global memory (i.e. accessible by all the various tasks shown in figure 5 ) as a data object comprising a base address followed by a series of relative addresses. The base address defines the starting point of the stroke, and the relative addresses define the offset of each

successive sample point from the previous sample point or, in the case of the first sample point following the start of the stroke, the offset from the start point. Thus, in the case of stroke S4, the relative addresses RA1, RA2, RA3 define successive sample points along the line from the base address which defines the start of the line at the bottom right.

A portion of global memory is shown

schematically at 206 in figure 7, and contains data objects defining n successive strokes #1 to #n. The structure of the data object for each stroke is shown on the right, and comprises the base address and a

plurality of relative addresses #1 to #m, the number of relative addresses m, and hence the accuracy with which the stored data approximates the actual line, being determined by the number of sample points as previously mentioned. Each pen stroke is reported to the observer task by passing a pointer Sj to the observer task 202, the pointer defining the position in global memory of the data object defining the stroke. Thus the pointers relating to the strokes #1 to #n shown in figure 7 would point to the positions S1 to Sn indicated. The actual data structure is opaque to the observer task, which merely handles the pointer, and this allows strokes to be displayed in various styles, such as thick or dashed, and also allows them to be deleted by re-writing them in the background colour, for example.

The observer task passes each pointer to the screen update task 204, which accesses the corresponding data object in global memory at the defined position and draws the line or stroke on the LCD screen. The

observer task also passes the pointer to the

communications task 205, which accesses the global memory at the position pointed to and sends the data object down the phone line 18 to the remote apparatus. The data objects are transmitted in buffered fashion as before, but in this case the transmit buffer contains only the pointers to the data objects, which remain in global memory until required for transmission. The observer task also receives data objects defining pen strokes from the remote apparatus, and passes these to the screen update task for drawing on the LCD screen. Thus, as in the first embodiment, the local LCD screen 12 of each apparatus displays a common image which contains components derived from both the local and remote apparatus, and is updatable in real time by each apparatus. The data objects are sent down the phone line as packets with a header indicating the length which is stripped at the receiving end.

Turning now to the switching to and from audio mode, it is first explained that, as is conventional, the two modems 17 communicate in duplex manner by sending information over carriers of two different frequencies. When a modem loses its carrier it normally assumes the line has been broken. However, in the absence of these carriers each local processor 10' can communicate with its respective local modem 17 by generating a pseudo carrier on which it sends data.

Also, in the absence of a carrier, the processors 10' can communicate across the line 18 with one another using a DTMF tone to signal one another.

This type of arrangement is known in the art, and in the present embodiment it is used as follows to switch from data to voice mode.

1. The local processor 10' sees that the voice detect timer has been set. As before, the voice timer is set when the local user speaks, and counts down to zero if there is a sufficiently long pause after

speaking.

2. The local processor then breaks the data link by removing the carrier. 3. It generates a pseudo carrier to tell its local modem that there has not been a break in the line and to go into hold mode, and also switches the data/audio switch 21 to voice mode.

4. It then issues a DTMF tone which the remote processor 10' will detect.

5. The remote processor 10' generates a pseudo carrier to tells its local modem to go into hold mode, and switches its data/audio switch 21 to voice mode.

Correspondingly, the switch from voice mode to data mode is as follows:

1. The local processor 10' sees that its voice detect timer has gone to zero.

2. The local processor 10' then puts a DTMF tone on the line.

3. The remote processor 10' detects this and if it too is prepared to go back into data mode it responds with another DTMF tone.

4. The remote processor 10' generates a pseudo carrier on which it tells its local modem to

re-establish a data link and also switches the data/ audio switch 21 to data mode.

5. When the local processor 10' detects the

responding DTMF tone it too generates a pseudo carrier and tells its modem to re-establish a data link and also switches the data/audio switch 21 to data mode. - 29 -

While the apparatus is in voice mode, any drawings movements made by the pen will continue to be stored as data objects in global memory, and their pointers stored in the transmit buffer, at least until the transmit buffer and/or the portion of the memory allocated to the storage of data objects defining pen strokes is full. Then, the data objects corresponding to the pointers stored in the transmit buffer will be transmitted when the data mode is restored. The communications task of each apparatus will also cause any packets of data objects corrupted by the switch from data mode to voice mode to be resent when data mode is once again re-established. It will be realised that the pen stroke may be stored in memory 11 in a number of different formats. The data object may comprise relative addresses which, instead of defining locations on the pen stroke itself, represent the locations of control points on a B-Spline or Bezier curve approximation of the pen stroke, as well as information relating to the continuity and other features of the curve.

It will also be realised that the configuration of plug-in card 201 will depend on the type of computer being used. A generic solution could be provided by using a separate module (not shown) . The pen computer 200 could then be connected to this module by connecting the phone line 18 from the modem 17 to the input of the module and connecting the output of the module to the phone line to be used.

Claims

CLAIMS :

1. A drawing apparatus including a processor having a display device and a pointing device operatively connected thereto such that an image may be drawn on the screen of the display device by moving the pointing device, the apparatus further including means for transmitting movements of the pointing device to a like apparatus at a remote location and for receiving

movements of the pointing device associated with the remote apparatus, the apparatus being adapted to display on the display device the said received movements of the remote pointing device such that the movements of both pointing devices are displayed on a common area of the screen whereby the image may contain components derived from both pointing devices.

2. A drawing apparatus as claimed in claim 1, wherein the means for transmitting and receiving

includes a modem for communication with the remote apparatus over a telephone line.

3. An apparatus as claimed in claim 2, further including audio circuits for the transmission of speech signals to and the reception of speech signals from the remote apparatus, and means for switching between a voice mode for the transmission and reception of speech signals and a data mode for the transmission and

reception of pointing device movements.

4. An apparatus as claimed in claim 3, wherein the switching means is switched to the voice mode

automatically on detection of speech by the user of the apparatus.

5. An apparatus as claimed in claim 4, including a transmit buffer for storing, at least during the data mode, data defining movements of the pointing device intended for transmission, the pointing device movements being transmitted as relative and/or absolute screen addresses.

6. An apparatus as claimed in claim 5, wherein data defining the pointing device movement are stored in the transmit buffer also during voice mode and transmitted when the switching means next switches the apparatus to data mode.

7. A drawing apparatus substantially as described herein with reference to figures 1 to 3 or to figures 4 to 7 of the accompanying drawings.