WO2001031416A2

WO2001031416A2 - System and method for simultaneous display of multiple information sources

Info

Publication number: WO2001031416A2
Application number: PCT/US2000/029850
Authority: WO
Inventors: Ovid Santoro; Klaus Lagermann
Original assignee: Surfcast, Inc.
Priority date: 1999-10-29
Filing date: 2000-10-30
Publication date: 2001-05-03
Also published as: CA2387933A1; IL149090A0; AU2573301A; EP1242856A2; JP2003513350A; WO2001031416A3; EP1242856A4

Abstract

A computerized method of presenting information from a variety of sources (10-90) on a display device (110). Specifically the present invention (Fig. 2) describes a graphical user interface (126) for organizing the simultaneous display of information from a multitude of information sources (10-90). In particular, the present invention comprises a graphical user interface (126) which organizes content from a variety of information sources into a grid (1) of tiles (10-90), each of which can refresh its content independently of the others. The grid (1) functionality manages the refresh rates of the multiple information sources. The present invention is intended to operate in a plateform independent manner.

Description

SYSTEM AND METHOD FOR SIMULTANEOUS DISPLAY OF MULTIPLE INFORMATION SOURCES

FIELD OF THE INVENTION

The present invention relates to methods of presenting information from a variety of sources on a display device. Specifically the present invention describes a graphical user interface for organizing simultaneous display of information from a multitude of sources.

BACKGROUND OF THE INVENTION

The scope of the global communications capacity, comprising fixed link and wireless networks, continues to expand rapidly. The variety and complexity of 5 communication devices proliferates and the number of users escalates. As a result, users are faced with increasingly complex systems and interfaces with which to manage multiple sources of information. At the same lime, society has increased its demands on time and productivity so that users no longer have the luxury of focusing their attention on a single 0 source of information or means of communication. Instead, the norm today is for people to carry out many tasks simultaneously.

As might be expected, these demands have exposed substantial problems in current communications technology. In particular, users are faced with insufficient resources to 5 manage and access the volume and variety of infonnation available to them in an efficient and productive manner. While a variety of tools designed to assist in accessing and managing these resources have been created, these tools remain unsatisfactory. Q Consequently, users are impeded by the myriad of infoπnation sources, each with its own method of use and often with its own login and password requirements, as well as by slow retrieval times to access the information. The result is an unacceptable delay for many operations. 5 Under the present art, for example, it is usually the case that a user lacks the bandwidth resources to receive multiple video signals simultaneously. If an individual were receiving one video signal, it is usually impractical to receive a second at the same time due to bandwidth constraints. Thus, the user could not, for example, monitor multiple video data streams of sporting or news events; instead, the user could monitor only one video data stream at a time.

To address such bandwidth resource limitations, the current art only accesses information when the user requests it. As a result, there is an inevitable delay between the user's request far information and the communications device's presentation of it. For example, if a user wants to monitor sources of news information on the Internet using current browser technology, the user must continuously mid manually request the news data from its source to determine whether tire data has been updated. Prior to requesting and subsequently receiving the data, the user has no way of knowing whether the data has been updated, hi any case, the user is unlikely to want to refresh the status of each application by manual intervention himself at the frequency necessary to ensure that the in ormation is up to date. Additionally, if a user wishes to view two or more webpages simultaneously, he must run two or more copies of the web-browser program. The act of manually refreshing the content of alternate programs in order to ascertain which have any new material to offer is fundamentally inefficient.

Similarly, the user's access to such data is not in real-time or even near real-time because each time the user wants to view the information, he must request it from its source and wait for the source to transmit it to him. Thereafter, he must wait until his communications device has received and processed the infonnation before it is presented.

For complex infonnation such as a video signal, this can take longer than a minute to occur; and, even for simple information, this process can take many seconds. Thus, the user is denied real-time or near real-time access to the information. Present technology that locally stores or "caches" previously accessed information to make it available to the user more rapidly does not solve this problem, because the cached infomvation is necessarily old. The user's communications device must still verify the accuracy of the information with the source before the system displays the cached information. As a result, the user is denied real-time or near real-time access to updated information

Similarly, if a user wishes to make two or more simultaneous downloads there is no control over the relative rates at which the respective downloads would occur So-called

"push technologies" attempt to addiess this problem by organizing infoimation from a number of related sources and sending it periodically to a usei While this arrangement fiees a user from actively participating in the download, the price is that the usei has little control over the organization of the information and can only practically handle a small number of such transmissions at any one time Each transmission is subject to the bandwidth available Of course, not all tasks require the same allocation of resouices and. correspondingly, not all tasks have equal priorities for a given user. In paiticulai, a user may wish to customize the information environment m such a way that many processes aie occumng synchronously, yet each is communicating with the user at a rate that is acceptable. For example, a television viewer may wish to know what is being broadcast on several channels at the same time but only care to watch one of them closely An Internet user may wish to be continually in touch with somces of data from audio, video, chat -room, video-conferencing and e-mail checker utilities, but not wish all of them to update at the same frequency; the user would be satisfied merely to see at a glance a recent status of each

Some of these processes, such as chat-room activities entail very little data transmission and can, indeed, be effectively updated on a continuous basis, whereas others require a great deal of bandwidth but could usefully be sampled at a lower rate. The current art lacks any technology for controlling the respective refresh rates of several simultaneous information sources.

At the same time that users are limited by system resources, they are also finding that they have no effective way of managing the multiplicity of available data types and information sources It is difficult both to conduct two or more different types of computmg activities at the same time or to monitoi two or more different information sources simultaneously because the tools available are confusing, inflexible, and/or otherwise difficult to implement Users require immediate access to a wide variety of up to date content presented in a flexible, easily customized interface

In addition to restrictions m the capacity of today's networks, there is very little conformity amongst the information content A typical communication device, such as a 5 peisonal computer, television oi mobile telephone, comprises a display unit connected to a processing unit that can accept information from many diffeient sources As descnbed above, the signals, data and/or datastreams that are available to such a device are diverse, _Q including, for example, HTML content, e-mail, or streaming audio and video

Correspondingly, the softwaie tools that interpret and process the different information sources present each in a different way to the user From a user's petspective, distinctions between the different types of infounation could usefully be removed so that each is viewed 5 a similar way and such that the current presentation associated with any information souice gives an immediate indication of its current content The present reality is different, however The user must contend with a wide range of icons and piogram windows that may occupy space on a user's display screen Another lack of conformity is the different mode 0 of behavior for programs that address different types of information An effort to standardize the ways in which different types of information are presented to the user would be advantageous. Equally, unification of the way m which those types of infonnation aie managed would save time and increase user productivity, for productivity is reduced when 5 users must cope with different attributes of different programs and learn distinct paradigms for different types of information.

The nature of the application program windows and their respective icons predominantly found on today's computer displays is restrictive. The application window typically displays the current content or output of only a single program and program icons convey nothing of the program's cuπ^*ent state or content. Often, an icon is a static image which is merely characteristic of the program or data represented thereby rather than the program's current state or its infonτiation content. In the present art, there is no intermediate between a window or an icon.

Thus, while a window may be resized as appropriate, it will frequently occupy the full display area, effectively limiting the user to a view of a single program. It may have 15 active areas around its borders such as menu bars, scroll bars, or tool bars designed to allow the user to control aspects of the window's appearance or to set parameters specific to the operation of the program controlling it. Icons, in contrast, offer ease of display when

-_jQ multiple programs are active, but they do not pennit viewing or control of the underlying program or data represented thereby. Instead, icons require user intervention, typically in the fonn of a mouse-click on an icon of interest, to view or control the program ov information. Consequently, the user's viewing options are limited to a choice between one

25 presenting very limited information about a multitude of programs and infonnation and one presenting full information, but of only a single program or data source.

The fact that the GUI's of the present ait are largely restricted to icons and windows diminishes the capacity to organize, manage, and access available infonnation. With the

30

Internet representing an ever expanding view of currently accessible global information, the need for flexible information management tools has become crucial. Similarly, with the current expansion of television programming available, for example, through cable television and satellite broadcasting, the need to manage this audiovisual content becomes acute. The convergence of television programming and computers increases these management needs all the more.

Current computer operating system software utilizes bookmarking schemes for managing Internet locations and on complex database technologies for managing specialist infonnation. Neither provides visual immediacy or ease of layout. Bookmark hierarchies are presented as cascading textual menus and database technologies arrange information into rigidly defined structures. The missing capability is a visual categorization in which an area of the display unit itself becomes the bookmark and the arrangement on the display becomes the categorization, independent of the type of content.

While the most common way of accessing information sources is via a personal computer, present day technology exists to communicate via a television, handheld 5 computing device, or even mobile telephones, in which case Internet content and other data can be displayed as some portion of the screen. There is a growing convergence of technologies: televisions are beginniug to find application as viewers of non-television Q data, (for example through use of "'Vertical Blanking Interval" technology in which a signal is inserted into the main video signal or through set-top boxes providing limited computer and communications functionality); computers are already finding application for the display of movies, real-time data streams, and the playing of audio data; handheld 5 computing devices and mobile telephones are also being enabled to access the Internet and other information sources.

To summarize the current state of the art, display technologies currently lack an interface which is capable of organizing any type of information, presenting such 0 information to the user in a consistent manner and in such a way that all currently open channels are able to indicate their activity on a continual basis and which could run on any device.

5 SUMMARY OF THE INVENTION

Accordingly, the present invention provides an easy to use graphical interface that facilitates the organization and management of multiple data sources corresponding to a user's needs and interests. The present invention comprises a grid of tiles that resides on the user's computer desktop. The grid of tiles provides a unifonn, graphical environment in which a user can access, operate, and/or control multiple data sources on electronic devices.

The graphical environment is unifoπn with respect to the source of accessed information and can manage multiple streams of content, entirely of the user's choice. For example, the invention presents video clips, e-mail messages, television shows, Internet sites, application programs, data files and folders, live video streams, music, radio shows, and any other form of analog signal, digital data or electronically stored infonnation, to the user uniformly and simultaneously, regardless of whether the information is stored locally or available via modem, TI line, infrared, or any other form of communication. The user's impression of the interface is also independent of the type of electronic device upon which it is implemented.

The present invention comprises a method executed by a computer under the control of a program stored in computer memory, said method comprising the steps of: partitioning a visual display of a computer into an array of tiles in a non-overlapping configuration; assigning a first refresh rate to a first tile of said array of tiles and a second refresh rate to a second tile of said array of tiles; updating information presented to said first tile in accordance with said first refresh rate; and updating information presented to said second tile in accordance with said second refresh rate.

The present invention additionally includes an electronic readable memory to direct an electronic device to function in a specified manner, comprising: a first set of instructions to control simultaneous communication with a plurality of datastreams: a second set of instructions to partition a display into an an^*ay of tiles; a third set of instmctions to associate a first datastream of said plurality of datastreams to a fust tile of said array of tiles and a second datastream of said plurality of datastreams to a second tile of said array of tiles; a fourth set of instructions to retrieve data from said first datastream in accordance with a first retrieval rate and retrieve data from said second datastream in accordance with a second retrieval rate; and a fifth set of instructions to present data to said first tile in accordance with said first retrieval rate and present data to said second tile m accordance with said second retrieval rate.

The application program of the present invention runs on many different devices, including, but not limited to set-top box, personal computer and hand-held device. Ihe gnd and tiles retain the same charactenstics, regardless of operating device. For example, the tiles remain individually configurable and can offer near real-time views of their data content The application therefoie permits the user's interaction with a range of electronic devices to be unified

BRIEF DESCRIPTION OF THE DRAWINGS Additional objects and features of the invention will be more readily apparent from the following detailed description and appended claims when taken in conjunction with the drawings, in which^*

Fig 1 shows a representative embodiment of the user tei face of the present invention comprising a grid of tiles as might be depicted on a display screen.

Fig 2 depicts a system that, in accordance with the present invention, accepts data m at least one form through at least one port and which additionally displays data to a user.

Fig. 3 shows stylised examples of an icon and an application window as are commonly found in computer display systems of the background art.

Fig. 4 shows several tiles as might be found in a typical embodiment of the present mvention.

Fig. 5 shows an exemplary data structure of the tile object within lire graphical user interface of the current invention. Fig 6 shows one embodiment of a tile in maikup language

Fig 7 shows the hieiarchy of software objects underlying the current invention, comprising a gnd object, tile objects and files or application software Fig 8 shows an exemplary layout of the display produced by the current invention

Fig 9 shows an alternative exemplary layout of the display produced by the current invention

Fig 10 shows an exemplary layout of the display of the current invention wherein a tile contains anolhei instantiation ot a grid

Fig 11 shows an exemplary layout of the display of the cuiτent invention including specific examples of tile contents

Fig 12 show ε the data stnictuie o the gnd object which forms part ot the graphical user 5 lntetface of the current venUon

Fig 1 shows one embodiment of a grid in markup language

Fig 14 shows a sequence of windows that demonstrate how a gnd might be set up for initial Q use by a '"wizard" tool in one embodiment o the present invention

Fig 15 shows an example of the architecture of the computer program m a piefeπed embodiment of the present invention

Fig 16 shows the architecture of the application program and its components m a preferred 5 embodiment of the present invention

Fig 17 shows the aiclntecture of components ot the computer program m a prefened embodiment ot the present invention

Fig 18 shows an outline of the widget-set used m a preferred embodiment of the piesent 0 invention

Fig 19 shows an outline of the metabase according to a preferred embodiment of the present invention

Fig 20 shows an outline of the XP Core and its interaction with the operating system library 5 m a preferred embodiment of the piesent invention Fig 21 shows an overview of the event system utilized in a preferred embodiment of the present invention

Fig. 22 shows an overview of the connection layers that are responsible foi controlling the download of multiple web-pages from the world wide web

Fig 23 shows a number of functions used by the bandwidth controller

Fig 24 is a schematic representation of the relationship between a server and a client device

Fig 25 shows a series of interactions between a client device and a server

Fig 26 shows how a user, a server and third party content providers communicate in accoidance with an embodiment of the invention

Fie 27 shows an embodiment which the application program communicates with one oi moie wireless devices

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Figme 1 shows an lllusnative configuration of the giaph cal user interface of the present invention A grid 1 consisting of a 3 by 3 matrix of nine tiles demonstrates some of the different contents that tiles can display. Tile 10 points to a database of stock quotes Tile 20 displays the active folders an electronic mail utihty. Tile 30 displays a portion of an alphabetical list of quoted companies Tiles 40, 50, 60, 70 and 80 point to websites displaying, respectively, high technology news, electronic goods for sale, categories of business news, items available by auction and the Wall Street Jounial Tile 90 points to the file- viewer of a windows-based operating system and displays the cunently accessible disc dnves

Within the scope of the present invention, an information source may compnse any analog signal, source of digital data or a datastream, including, but not limited to, video, audio, text and graphics The information may be m any format, including but not limited to ASCII, bitmap, MP3, JPEG, GIF, TIFF, a mark-up language such as HTML, XML,

VRML, HDML, formatted text such as rich text format, or binary.

Figure 2 is a general representation of a data display system 100 within which the present invention may be implemented System 100 compnses a central processing unit

104, an input device 106, data connection ports 108-1 through 08-N, a display 110 and a ma memory 112, all connected via bus 116 Residing in the memory 112 is an operating system 120, a file system 122, a cache 124 for temporary storage of information, application programs 128-1 through 128-N and a graphical user interface (GUI) program 126 that is responsible for presenting information on to display 110 Infonnation may enter the system through any one of the ports 108-1 through 108-N which may themselves be connected to a tuner 1 14, or via a network interface 134 to a communications network If a tunei 114 is employed, it may channel input iiom a wireless signal 130 oi a cable network 132

In one embodiment of the present invention, system 100 is a personal computer such as a desktop workstation or a portable notebook computei. hi that case the input device 106 may be a keyboaid, mouse, trackpad, trackball, or any combination thereof and the display 110 may be a conventional cathode ray tube (CRT) or active matrix flat-screen display The netwoik interface 134 may then be a connection to the internet or to a local area network via a cable and modem or a digital subscnber line. in another embodiment of the present invention, system 100 is a mobile phone oi personal digital assistant and the input device 106 may consist of seveial buttons on a keypad, a touch-sensitive screen with a touching device oi a microphone and voice- recognition software. In this embodiment, the display 110 is preferably an LCD scieen oi an electroluminescent display and ports 108 receive data from radio signals or a portable modem.

In yet another embodiment of the present invention, system 100 comprises a set-top box wherein display 110 is a TV screen or monitor and tuner 1 14 accepts input in the form of a wireless signal 130 from broadcast transmissions or cable signals from the cable network 132. Input device 106 may be a hand-held remote control apparatus or buttons located on the set -top box or touch-sensitive areas of display 110.

As is known to those skilled in the art, a graphical user interface is a computer program that resides in memory 112 of some data piocessing system and that provides means for presenting information, input to and output from application programs 128 or content of datastreams from ports 108 on an associated display. In the background art each datastream is associated with a window. The graphical user interface allows a user to control the aπangement and display format of the dala content in each window. Usually a graphical user interface permits a user to specify and alter operating parameters of application programs running on the system though, at any given time, a particular application program will take priority, meaning that a particular window will be displaying continually updating content Typical operating parameters that may be changed depend upon the application progiam but include the number of buttons on a tool bar and numbei of visible toolbars, the size of the text displayed and the color of the background By contrast, the gi plucal user interface of the current invention not only permits a user to control the layout of the data content but to prioritize each application program running on the system and each datastream of interest A novel feature of the present invention is that the data content of any number of the programs can vary in real time and the rale at which the display of each is updated can be controlled by the user.

Tile Objects

In the ensuing discussion, tile objects are introduced and descnbed and contrasted with existing elements of graphical user interfaces. A tile presents content from any infonnation source.

Conventional graphical user interfaces of the background art provide two distinct representations of programs, files and datastreams, as shown in figure 3. One representation is an icon 320, the other is a window 340. An icon typically occupies only a relatively small proportion of the available display area and is an easily recognizable depiction of the program or file, either through its logo 322 or some characteristic picture with the name 324 of the program visible. An icon can be selected by, for example, a touch screen pointer, a cursor controlled by a mouse with a button or by a keyboard stroke or any combination of the foregoing. In response to a further selection operation on an icon, for example a doubleclick of a mouse button, the graphical user interface will provide a window that can be used to communicate further infoπnation to the program or review the associated datastream.

The window may occupy a substantial percentage of available screen space, usually 90- 100%. The window 340 usually comprises a title bar 348 and a display area 354. The window 340 can commonly be resized by the user for example by using buttons 352 or a draggable area 356 and has a format which contains many active areas around its borders.

Examples of active areas include a menu bar 342, a vertical scroll bar 344, a horizontal scroll bar 350 and one or more tool bars 346. Each active area may be used to control aspects of the window's appearance or to set parameters specific to the operation of the program associated with it such as text formatting options in a word-processing package or redirection of a web-browser to its stored home location.

In the present invention, a third graphical representation of programs and files, herein called a tile, is introduced. Tiles permit "dynamic bookmarking" of information in that each tile is a viewer of a single information source - including streaming data sources - and can be customized with the user's choice of content.

A tiie is different from an icon because it provides a real-time or near-real time view of the underlying information in that it contains continually refreshed content. A tile is different from a window because a tile will typically be smaller in size, allowing the user to view multiple tiles simultaneously if desired.

A tile provides an at-a-glance view of the current status of the program or file associated with it but does not necessarily have the large number of active areas associated with windows such as title bar, menu bar and scroll bars. Therefore tiles lead to a reduction in clutter on the display screen because many tiles may be displayed simultaneously without overlapping with one another in the way that windo s must necessarily do. Tiles are supenoi to icons because they give an immediate indication of the current state of the file or program and have utility functions associated with them, as described below Another advantage of usmg tiles is a umfoπnity of appearance between tiles which correspond to different programs and datastreams The display content of a tile will differ from application to application though its size and format need not

A tile is associated with a progiam, file or datastream, in the same way that an icon and a window aie A tile may present data any of a numbei of ways For example, m the preferred embodiment, the tiles may present a miniaturized, "thumbnail" view of the undei lying information, a "'porthole" view of a portion of the undei lying infonnation as viewed at full size, a symbol indicating whether the information has been updated since it was last viewed, or a custom interface designed to allow rapid access to the underlying infonnation The way m which a tile displays content may be independently configured foi ach tile

Figure 4 illustrates representative tiles Tile 402 displays a pictuic or giapluc such as may be stored m a bit-map, IPEG, TIFF oi GIF file, or on a w orld-wide web page The content of tile 402 is typically a miniaturized representation of a graphic or still-frame from datastream Tile 404 displays a portion of a text document or text of a world-wide web page In tins sense the tile functions as a transparent panel placed on top of a document, thus permitting a portion of the document to be displayed Tile 406 displays a further array of tiles that maybe displayed in full by expanding tile 406 to occupy the full area of the display Tile 408 has been configured to link to an electronic mail program An alarm setting associated with tile 408 has been configured so that the ale displays an envelope and the message "New Mail¹" when an unread message has been received by the mail progiam

Tile 410 displays a name, "FM 101", denoting the title of a broadcast signal, in this case audio, that is associated with the tile Tile 412 displays a "thumbnail" of the document viewed in a window such as 340. A "thumbnail", as used herein, is a miniaturized representation of an image tiiat retains sufficient characteristics to permit easy recognition of the image. For example, if the document displayed in a window were a multipage document, tile 412 may display a thumbnail of the first page of the document.

Tiles are selectable and live. When a tile is selected, whether by mouse click or otherwise, the tile instantly provides the user with access to the underlying information, whether that data be a hierarchical menumg system leading the user to a different level or tiles, a word processing file stored on a local area network, a spreadsheet stored locally on a user's computer, HTML file on the Internet, or a television signal The tiles are live in that each contains leal-time or near real-time information. hi a prefened embodiment a selection operation is associated with a tile. For example, clicking on a tile will cause it to immediately refresh its contents. Selecting tile

402 causes the most recent frame of the video stream to be displayed by the tile, or, if the picture were obtained from a static graphic file, causes the most recent version of the file to be displayed by the tile Selecting tile 404 or tile 412 causes the most recent version of the document to be displayed.

In a preferred embodiment, double-clicking a tile causes that tile to occupy a substantial fraction of the whole display area. In this embodiment, double clicking tile 402 or 412 causes the image to be expanded to fit the whole display area In an alternate embodiment, selecting a tile causes it to occupy an area m the middle of the display that is larger in area than a single tile but does not occupy the full display. If tile 404 were double- clicked, as much of the document as could be displayed m the enlarged tile in regular font size becomes visible. Double-clicking tile 408 causes the mailbox window of the e-mail utility to be sufficiently enlarged to allow new messages to be selected and read. Double- clicking tile 410 causes the audio stream to become audible over the appropriate channel of the system. A representative tile data structure 500 is shown m Figure 5. It is important to understand that the tile itself is an image that at any given instant is resident on the file system. This image is separate and distinct from the application program or file associated with the tile The tile data structuie 500 comprises two addresses: a tile address 502 that defines the location on the file system where the tile image is stored; and a target address

504 that is the location at which the file or application program associated with the tile can be found. Additionally, the tile data structure contains a name 506 that may be displayed on the tile in certain circumstances. Tiles of the present invention may be assigned at least seven functions, including but not limited to: an initialization function 508 tiiat is responsible for establishing connection with the target addiess 504; a refresh function 510 that handles updates to the tile image stored at the tile address 502, a screen-size function

512 that stores the dimensions orthe display aiea filled by the tile upon receipt of a request, an alarm function 514 that permits the tile to display an alarm or warning when the application program associated with the tile encounteis a designated event, an on mouseover function 518 and an on mouseout function 520 which control the behavior of the tile when a selection tool such as a mouse-controlled cursor is placed respectively on and off the tile, and a toolbar function 522 which may permit an aπay of special buttons to appear on or adjacent to the tile for the purposes of adjusting properties of the tile. In one embodiment, a tile is configured so that a right-click of a 2-or 3-button mouse while the cursoi is over the tile would activate the tool-bar function. In a prefened embodiment, right-clicking on a tile can reveal a menu of options that enable the user to ascertain properties of the tile, such as the bandwidth it is consuming

In one embodiment of the present invention, the tile is itself a document created in a markup language such as HTML or XML as shown m Figure 6 and is suitable for display in a web-browser. In this embodiment, a tile occupies a position in a table defined with elements of mark-tip language which will be familiar to one skilled the art. Tile-specific attributes are introduced which control the way in which a web-browser displays the tile n the example in Figure 6, the tile has a clickable map, i e , separate areas of the tile produce separate results when clicked Also, this tile has a toolbar, which in one embodiment may appear if the mouse is right-clicked when the cursoi is over tiie tile In another embodiment of the present invention, tiles communicate with one another and have conditional content. That is, the content of one tile depends upon the content of another.

Grid Object

The arrangement, layout and independent functioning of the tiles on the display and exemplaiy software for their control is now described with respect to Figures 7-13

The overall hieraichy of a graphical user interface embodied bv the present 5 invention is summarized in Figure 7 he Grid 700 is the top level functionality to which application piograms are subordinate 7. he grid can replace the functionality of a user's computer desktop and offers similar and additional featuies. Gnd 700 comprises a matrix Q of tiles of which tiles 702, 704, 706, 708 and 710 are representative The user can control the grid m such a way that the tiles 702, 704, 706, 708, and 710 present simultaneous information content from a plurality of somces The grid controls the layout and priorities of the tiles Each tile is associated with a data stream or application program and allows a 5 choice of displayed images, hi particular, different tiles can be associated with different contents For example, tile 702 is connected to a web-page viewer 712 such as a biowser Tile 704 is connected to a source of streaming video 714, such as Real Player Tile 706 is connected to an audio player 716 such as a CD-player program or a source of streaming 0 audio such as Real Audio. Tile 708 is connected to a file viewer 718 such as a text-editor or a word-processing package. Tile 710 is associated with another grid object 720, thereby permitting a "layenng" of information hierarchies In a prefened embodiment, a grid embodies a similar undei lying data structure to a tile 5 Each tile is separately associated with a source of information, for example, an application program, datastream or file, any one of which may be another grid object Such a hierarchical structure permits a user to organize programs and infoiτnalion through the graphical user interface For example, separate categones of infonnation can be displayed on separate grids allowing each grid to be associated with a theme

By using the native attributes of a Ule, a user may specify a presentation of the gnd, consisting ot its dimensions, (^■ e , the numbei of tiles to display and their arrangement), and the programs or files to be associated with each tile A smgle gnd, composed ot multiple tiles, may theiefore present a number of infonnation sources simultaneously

Together, the gnd and tiles compnse the application through which a usei can view simultaneously information from a multitude of his available sources mcluding multiple 5 sites on the Internet, receive signals from multiple broadcast channels, and open and view multiple files In its initial embodiment, the application may be run through conventional computer operating systems, whereupon it overlays the user's desktop and acts as if it were _Q a "borderless browser". Therefore the application resides ovci existing applications without replacing any of them; rather it enables them to be called from the grid itself The application, iherefoi e, becomes a graphical file manager in which the content of contmυously changing files, t e , datastreams, is being displayed in leal-time or near leal- 5 time, depending on the assigned pi lonty Effectively the application replaces the user's desktop with a more visually intuitive dynamic menuing system

In ά prefened embodiment, a gπd of tiles replaces the functionality of the computer

"desktop" utilized by many modern computer opei tmg systems Whereas a desktop is 0 typically populated by static icons and tool bars, a gnd of tiles instead pieseuts to the user an anay of snapshots of current programs and files The essence of the grid is that its content is dynamic and informative. As previously discussed, icons are inherently limited m the information that they can present and windows clutter the entire desktop. By contrast, 5 each tile on the grid can show the current status of the data or datastream associated with it

- IS - The fact that a tile may refer to a separate grid permits nesting of grids and consequently a hierarchy of organized information sources.

The grid also understands the interests of the user and acts as a repository for passwords and identifiers to subscription services. In this way, it is not necessary for a user to remember and enter a user name and password to access data requiring such a log-in. In this same vein, Internet sites of interest can be "bookmarked" and stored by the grid, each such site possessing its own tile. The grid is also nestable in that a tile in one grid may point to a separate grid and tiles in the separate grid may point back to the first grid or to yet more grids. If desired, a user can impose a "theme" on a grid and thereby categorize, group, and/or otherwise manage his data sources. In this manner, the user can group tiles relating to particular subject matters, Internet sites, documents, or otherwise in a ϋrid according to some speciality. A tile in such a grid could link to another grid to provide a connection between related categories. Equally, a tile on more than one grid can point to the same information source. In a prefened embodiment, the grid pennits a regular layout of tiles on the display screen such that the tiles are unifonn in size and shape, as depicted in Figures 8-1 1. Each tile is indexed by its position on the grid. For example 802-2-1 is the first tile in the second row. Tiles in the first row of the grid are 802-1-1, 802-1-2, 802-1-3 and so on, to 802-1- . Tiles on the second row are 802-2-1, 802-2-2, 802-2-3 and so on, to 802-2-N. And, tiles on the bottom row are 802-M-l, 802-M-2, 802-M-3 and so on to 802-M-N. There are no gaps between the tiles, the tiles are not permitted to overlap and the whole grid is covered by tiles. Figure 8 shows one embodiment in which all tiles are the same size and are presented in an array comprising M rows and N columns. There is no particular requirement that the arrangement consists of more than one row and more than one column. On the display of a mobile telephone or smart phone, for example, a single row of tiles may be apposite.

Figure 9 shows an arrangement in which there is a unit tile size, that of tile 802-1-1, but tile 802-1-2 and tile 802-N-l have each been configured to occupy regions of the grid equal to exact multiples of the unit tile size. Such an arrangement may be useful and important if one or more datastreams is of particular interest but the others are also to be monitored at the same time. Figure 10 shows an arrangement of tiles in which Tile 802-M-l is associated with a further grid 1000. The lower half of the figure shows an enlarged perspective of that tile showing a grid with Y columns and X rows.

Figure 11 shows an arrangement of tiles in which are depicted different application programs associated with three of the tiles. Tile 802-2-2 links to a file viewer displaying a specific file; tile 802-N-l presents streaming video; Tile S02-M-N depicts a page of information on the world wide web.

Figure 12 shows a schematic data structure of the attributes of one embodiment of grid object 700. The architecture shown in Fig. 12 applies to any grid, including a top-level grid 700 shown in Fig. 7 as well as to a grid that is contained within a tile.

Significantly, the grid manages the flow of infonnation to the tiles. For example, the grid can communicate with the display device in order to determine its current configuration and allocation of resources. In one embodiment the grid continually cycles around the currently displayed tiles, one by one, refreshing the content of a tile each time it is accessed. When a given tile is refreshed, the refresh operation is completed before refreshing the next tile in sequence. In this way, the cycling rate may be set so that the current content of all tiles are reasonably up to date. The cycling can be interrupted by a user selecting a given tile, so that that tile alone becomes continuously updated. In this way, the user does noi need to worry about manually refreshing multiple tiles.

In a preferred embodiment, according to priorities that may be applied to individual tiles on a tile by tile basis if desired, the grid manages the refresh rate of each tile in the grid.

For example, for locally stored word processing or spread sheet files, the user might configure the tiles to refresh only when the underlying data is written to the local hard drive.

Similarly, a user might configure tiles containing infrequently updated HTML data from rhe Internet to refresh at a certain time each day. At the other extreme, a user might configure an active tile to display a television channel at a refresh rate of 29 frames per second, while at the same time configuring inactive tiles to display different channels at a refresh rate of once every five seconds. In this way, a user could monitor many channels until program content of interest appeared in one of the tiles without the burden of actively refreshing each tile.

The grid itself has an address 1204 that specifies its location within the file system of the device in which the application program program runs. The grid has associated with it several utility programs: a configuration wizard 1206 that may be called by the user when setting up a new grid; a tile creation function 1208 utilized hy the configuration wizard when initializing new tiles; a tile amiihilation function 1210 utilized in case of error or when resizing the grid.

The grid object stores the number of rows 1212 and the number of columns 1214 of tiles that are present. The grid also stores a tile list 1216 containing attributes of each respective tile. In particular, the address of each tile, its priority and its refresh rate are stored by the grid program. The grid also stores other attributes of tiles such as their respective positions on the grid as given by their column and row number. The priority of a tile may be used to determine its refresh rate in one embodiment of the present invention. A tile can have a password feature built into it if it is desired to restrict access to the tile's content. The grid itself can have a toolbar by which its attributes may be accessed and modified. hi a prefened embodiment, a grid is a special form of a tile. It is a tile that can create and manage an anay of other tiles. Accordingly, its data structure also comprises those elements of a tile data structure shown in Figure 5 in addition to those shown in Fig.

12. If the grid has a parent grid, the address of the parent grid 1202 is stored. For example, grid 1000, associated with tile 802-M-l of Figure 10, has grid 700 as its parent. The grid can be configured to contain any number of tiles from one to as many as can reasonably fit on tire user's display.

In one embodiment of the present invention, the grid is a document created in a markup language such as HTML, SGML or XML, Figure 13 and is therefore suitable for display via a web-browser. In this embodiment, the addresses of the parent grid, the grid itself and each of the tiles is expressed as a Universal Resource Locator (URL). The various functions controlled by the grid are accessible through function calls devised according to methods familiar to one skilled in the art For example, "dynamic HTML", Java applets or simple CGI-scnpts could provide the technological basis for enabling vanous grid utilities.

The application may be downloaded from a pre-determined web-site and operates in a client-server mode. Users may download preconfigured grids from the predetermined 5 server A grid configuration "wizard" program which guides a user through a step by step set up of a custom-grid may also be downloaded Other web hosts are able to deliver content to end-users via the predetermined server. Some basic functions of tiie grid can be o carried out on the predetermined server and provided to the user.

Grid Configuration Wizard

In one embodiment of the present invention, the set up of a particular grid is 5 achieved through a grid configurauon progτam ("wizard") that is downloaded to the display device from a remote site. The grid configuration program permits a user to define and install one or more grids on the client system. When a tile is partitioned into a further anay of tiles, the grid configuration program can also be used. One embodiment of the user 0 interface of the grid configuration wizard is shown in Figure 14.

A first screen displayed by the grid configuration wizard comprises an application program logo 1404, button 1406 to guide the user to the next screen and a number of choices, such as 1408. The user is offered the choice of preconfigured, or "standard" grid 5 configurations, selected from a list. Examples of such grids include grids themed by content such as sport-related gnds or by type of data such as gnds whose content is video- based.

Additionally, the user is permitted to configure "customized" gnds in which each tile can be taken from a list of predefined samples or can be initialized according to the user's wishes. In a second screen 1412 displayed by the grid configuration wizard, a tiled area 1416 represents the gnd that the user is building. Sample tile categories 1418 such as "weather", "news", "stocks", or "sports" are listed. In an alternate embodiment, a grid can be filled by the "drag and drop" technique in which a selected document 1414 is moved on to the display area of the grid configuration program and automatically becomes a tile. A button 1410 offers the user the chance to go back one screen.

In a third screen 1420 displayed by the grid configuration wizard, the user can name the grid and, optionally, store it for future reference, for example in an archive of prefened gnds The user can elect to finish gnd construction by clicking the "Finish" button 1412, or launch the gnd immediately by activating button 1422 When launching the grid immedi tely, the grid is automatically constructed on the fly according to the content and tile types specified by the user dunng the set up procedure

Architecture of Application Program Software The hierarchy of the software in a prefened embodiment of the present invention is illustrated in Figure 15. The software comprises a number of modules. In Figure 15, an anow connecUng two modules means that one module uses an interface of the other. The anow comes from the module invoking the interface towards the module whose interface is bemg mvoked An interface may simply be a function call between the two modules or, for example, a call to a dynamic linked library (DLL).

The Surfcast application program 1500 takes its underlying data from two sources, metabase 1506 and a system library 1514 that resides on the device on which the application program is running. For example, the application program 1500 may be required to call functions from the system library while it is running. Actual tiles that a user visualizes can be spawned from the metabase 1506.

Data structures for user tiles 1502 are obtained from the XP core 1504 which itself also utilizes components from the system library 1514. The XP core is an abstraction layer for the operating system environment. Tiles that utilize components such as buttons take these components from the widget set 1508. A widget is a basic user interface element such as a button or a text input box. The metabase also uses an interface of the widget set 1508, and can therefore use functions within the widget set. The widget set requests functions from the XP core.

Objects in the metabase 1506 that retneve content from remote sources such as world wide web pages utilize a connection manager and bandwidth controller 151 A URL loader 1510 decides whether content should be obtained afresh by contacting the connection manager 1512, or from content previously stored m cache. Effectively, the URL loader manages the connection manager, and calls functions within it. Underlying all of the application program's operations are functions from the operating system library 1516 that is supplied with the device on which the application program is running.

Each of the objects shown m Figure 15 is now descnbed in further detail by reference to Figures 16-21.

The Surfcast application program 1500, in Figure 16, comprises a launcher 1618 and a framework 1620. The launcher opens the program from scratch whereas the framework is responsible for managing grids and tiles. The user interacts directly with the framework to set up a prefened anangement of tiles on the display. In one embodiment, the framework initially contains a prescribed set of tiles. The framework controls communication between tiles, for example, in the case of conditional content.

Tiles 1502, Figure 16, are the equivalent of an application in a conventional GUI system. They can be built in C++ using the Surfcast tile builder application program interface (API) usmg XP core classes, or via a utility such as a custom tile editor or via a scnpt file Some predefined tiles are included with the basic Surfcast system including a web browser tile called a surftile 1602, tiles for contacts and commumcations such as a chat tile 1604 and an e-mail tile 1608. Media-player tiles such as a video tile 1606 are also supplied, as are tiles that interface to commonly used desktop programs A word tile 1610 interfaces to a word processor; excel tile 1612 interfaces to a spreadsheet program A general content viewer that can compose pieces of content clipped from a variety of sources, layout tile 1616, is also provided Reg tile 1614 is a general purpose tile that permits a user to define his own tile type

In a prefened embodiment, all tiles have a common base class, and each specialized type of tile has its own class that builds upon the base class There are many ways in which specific tile classes can be derived from the common base class Functions for specific tile classes are readily apparent to one of skill in the art Tiles may additionally be repiesented by markup language files and viewed within a web-browser environment The data classes XP core 1504, Figure 16, comprise base classes and utility classes with which tiles and widgets are built In general, classes within the XP core descnbe how tiles can communicate with one another and with the overall application program framework 1 20 In particular, XP core classes are generic and portable, thereby permitting cross platform capability

XP core classes include tilebaεe 1622 for the genenc class that underlies all tiles, tile base view 1624, tile controller 1626, canvas 1628, and classes for event handling 1630

A view is what a tile uses to draw itself Tile base view is the base class for all views associated with a visual object A controller processes events, for example mouse moves, chcks, keyboard events and external events Tile controller is the base class for controllers associated with a tile A canvas is an area of screen on which to render some unage, for example a tile The metabase 1506, Figure 17, is a local store for platform-specific implementations of the descriptions of tiles, grids and other objects Tiles, grids and content are created, saved and restored via the metabase. The metabase contains tile type and content type registnes such as tile registry 1732, and a local database of grids and content such as content store 1734 and grid store 1736 Unknown tile and content types can be obtained from remote servers. Tile types are abstracted so that if a grid contains a particular type of tile, for example an e-mail tile, the metabase provides whatever is appropriate for the device the applicaUon is n ning on. Items in the metabase are "persistent", that is they are not saved explicitly but are preserved from session to session

The widget set 1508, Figure 17, comprises a platform-specific set of visual components that tiles can use Widget set contains the predefined widgets that are included with the system. It can be extended with new widgets. It mcludes such useful widgets as a button 1738, a label 1740, an edit widget 1742 that enables a user to enter text into an editable field and a list widget 1744 that enables a user to select from a set of options. Items within the widget set can be used with tile types such as text input tiles, web browser tiles, and a streaming video tile The term widget can also include more complex objects such as a video player that can be inserted mto a tile as easily as a button.

The URL loader 1510, Figure 17, provides a mechanism for retrieving content. The URL loader 1510 interacts with connection manager 1512 for tiles which need to make a network connection. Tiles and the metabase ask for content for a given URL and the content manager will attempt to retneve it. The metabase also contacts the connection manager through the URL loader to ascertain whether there is sufficient bandwidth for the transfer. In particular, the connection manager decides whether the URL loader should furnish tile content from the cache 1746, as would be the case if the content has been recently displayed and stored locally. Alternatively, if the content is not cached, the URL loader supervises loading of content from the location specified by the URL. The URL loader also comprises a file manager 1748 for organizing the cached content. The URL loader additionally comprises a DOM (data object model) renderer 1750 that administers the parsing of pages in markup languages such as XML and HTML. The URL loader may also comprise an implementation of an API for XML rendering such as SAX. In an alternate embodiment, such an API may reside in the XP core module.

The system library 1514 comprises commonly used utilities within the application program, including, but not limited to, code for string manipulations, file handling and server communication. The system library module comprises generic code and can be compiled for any operating system.

The operating system library 1516 comprises utilities that differ in their implementation from system to system but are needed for operation of the application program. For example, utilities that may be found in system library 1516 are those that provide support for threads and synchronization 1752, debugging tools 1754, graphics libranes 1756, further basic string manipulations 1760 and connections 1762. Additionally, not shown in Figure 17, useful items in the operating system library include utilities that permit definitions of objects, sockets, input devices and hardware devices. Items in the operating system library are accessible through classes with documented public interfaces.

The operation of the widget set 1508 is further described with respect to Figure 18. As previously mentioned, the widget set contains widgets for various functions such as buttons, text labels, etc. Illustrated in Figure 18 is a set of widgets for buttons.

Figure 18 shows the class hierarchy for the button widget. Base classes for widgets are grouped together in box 1809. Widget class 1810 is a container for other classes. All widgets use the base class widget 1810 stored in the XP core module and further described later. Accordingly, button 1800 is a specific class inherited from widget 1810.

Widget view 1812 is a class, also stored in XP core, that defines the look and feel of the widget. Button view 1802 inherits from widget view and controls how a button draws itself In the scheme of Figure 18, user tile 1816 comprises three objects, tile controller

1626, the base class tile 1622 and tile view 1624, each of which is found in XP core and is further described later. Tile view is responsible for drawing the tile and can employ one or more widgets.

A button is something that a user can click on. A button provides button events to a button event consumer. A button event consumer is also known as a client, i.e , clients that use buttons implement button event consumer 1808 to be notified of button events. For example, a button event consumer may be a "play" function in a video tile. The button event consumer interacts with a control stnicture tile controller 1626 associated with a user tile 181 by telling the tile that the button has been activated. Button event consumer 1808 is itself a class that inherits from an event consumer class, described later.

Button controller 1804 controls how button events are processed, for example mouse and keyboard events. It inherits class stnicture from widget controller 1814 stored in

XP core Not all button events aie recognized by the button controller: for example, a particular key-stroke may have no effect on the state of the button.

Other widgets follow a similar pattern. They include: textedit, for inputting text; textiabel, for displaying text; textspinner, for selecting from a choice of options; datewidget, for entering a date; list, for selecting from a choice of options; titlebar; and a toolbar. Metabase 1506 is further described with respect to Figure 1 . Anows between the components in Figure 1 denote interfaces The metabase compnses a registry 1900 and store 1902 of tile and gnd types, a content store 1918 and a content and tile link database

1920.

Content store 1918 is a cache that contains content of tiles for the previous session.

Content tile link database 1 20 copes with descriptions of relationships between tiles and also themes of content for related tiles. This database can also be used in the context of

"knowledge management", i.e., those operations that monitor a user's activities and attempt! o suggest further sources of content based on it. Both content store 1918 and the database

1920 interface with tile type registry 1 00.

The tile type registry 1 00 contains a list of tile and widget types along with information about how to create them and what they are called, along with other information such as pointers to objects from which tiles are created. The tile and grid store

1 02 contains a library of stored grids and tiles. Tiles can save themselves or be restored.

In a pTefened embodiment, grids are just special cases of tiles. Tiles from the tile library can be called and displayed on the screen by asking the registry to load tiles. The tile and grid stoie interfaces to an XML library 1 06.

Tiles from the tile and grid store can also be saved outside the metabase in a library of markup language files, e g., an XML library 1906.

Also in the metabase is metabase tile 1904 which utilizes tile base 1622 from the XP core module. AU tiles inherit from this class

DLLs can contain additional tiles and widgets that can be created by independent third parties. These can be implemented within the metabase through the tile factory 1916 and tile creator 1 14, both of which interface to the tile type registry 1900. The tile factory

1916 contains the description and classes necessary for someone to register a new tile type.

Tile creator 1914 is the code that does the tile creation at runtime. In general, independent creation of tiles is facilitated by supplying a tile toolkit to third parties.

Inquiry utility 1912 is an optional means for an outside user to interface to the metabase, for example to ascertain the class structures of stored tile classes.

The content store 1918 follows a similar pattern to the tile type registry and the tile store. The content and tile link database 1920 is a database of information about how tiles and grids are related, and how content is related between them.

Figure 20 shows a further description of classes found in the XP core and their interaction with the operating system library 1516. As discussed below, some anows between ob_jects within XP core denote inheritance of classes Canvas 1628 descnbes an area of screen that can be drawn to.

GfxContext 2002 is a set of graphics pnmitives, for example for line-drawing, colors and space filling It is a genenc version that encapsulates and abstracts operating system-specific features inhented from GfxContextFactory 2032 the operating system library 1516 Win32 GfxContext 2034 is an example of a graphics context used with the Windows operating system Other GfxContent 2036 includes alternative platform dependent graphics contexts

The foundation classes are tile base 1622, tile base view 1624 and tile base controller 2008 Tilεbase 1622 is the base class for all visual objects such as tiles, widgets and gnds The tile class 2010, and widget base class 2016, hent from tile base. A widget effectively functions as a special kind of tile that can be placed mside a tile. Widget base

2016 is not meant to be instantiated on its own but is a foundation for the widget class 1810 used by a genenc widget Tile base view 1624 is the base class for all views associated with a visual object A view is what a tile uses to draw itself Also inhenting from tile base view is tile view 2012, the base class for all views associated with tiles and widgetview 1812, the base class for all views associated with widgets Tile base view and tile base interface with canvas 1628. Tile base controller 2008 is the base class for all controllers associated with a visual object Inheriting from tile base controller are tile controller 1626, the base class for controllers associated with a tile and widget controller 2020, the base class for all controllers associated with widgets A controller processes all events Tile base interfaces with both tile base view 1624 and tile base controller 2008 Tile base controller interfaces to event system 2022 Finally, event system 2022 communicates between the operating system library 1516 and the tile box controller 2008

Event system 2022 is further descnbed with respect to Figure 21. An event can be any one of a mouse movement event, another mouse event such as a mouse-click, or a keyboard event such as a keystroke. In Figure 21 , event 2112 is the base class. Other classes such as mouse movement event 2106, mouse event 2108 and keyboard event 2110 derive from the base class by inhentance. The event consumer 2104 is a class responsible for directmg events to the controller. The event producer 2102 is interpreting system events into Surfcast events for an event consumer, the boxes 2122, 2124, 2126, 2128, 2130 an

2132 are multiplexes handling the case where midtiple clients are affected by multiple types of events. An event is communicated to tile base controller 2008

Managing Connections to more than one Datastream

When two or more tiles connect to sources of data available over a network, communication must be established in such a way that the rate at which updated data is transmitted to the grid can be controlled. In practice, for an embodiment of the application which resides on a user's computer, a flow control protocol such as TCP is required. In this way, each tile can communicate with the remote datastream to which it is linked and a determination can be made of available bandwidth at the time of data transfer.

Alternatively, in a client-server mode, flow control is not necessary because communication with the server suffices, as is described below.

It is not practical to fire up a separate browser program from each tile that wishes to download data from a site on the world wide web. A web-browser is very greedy on memory and resources and the user would have little or no control over the respective rates at which data was downloaded from di ferent sites

Instead, in a prefened embodiment of the present invention, a hierarchy of layers manages the simultaneous connection and allocation of resources to different world wide web sites, as shown in Figure 22. The layer structure applies to the way in which any given tile downloads content.

At the highest level there exists a widget, refened to as "Surf widget" 2200, which is the basic browser control withm the application program of the present invention. Ideally, this widget will operate m conjunction with any commonly used world wide web browser

It will typically be associated with a tile type such as surftile 1602

The surf widget communicates with a surf widget controller 2202 a control layer 2201 Also m the control layer is a web browser apphcation program 2204 Examples of such a program mclude Microsoft's Internet Explorer and Netscape's Navigator software

The surf widget controller 2202 handles the interaction between the surf widget 2200 and the web-browser 2204 The surf widget controller also passes on requests from the browser to die URL Manager 2206 in the next layer, the URL layer 2205 The surf widget controller then pipes back the results to be rendered to surf widget A typical example of this process in operation would be a user clicks on a hyper-link in a web-page, the web-browser makes a request for that page to surf widget controller 2202, the request gets handed to the URL manager 2206, once the page is loaded, the URL manager 2206 notifies the surt widget controller, which in turn sends the infonnation to the web-browser for rendermg

The responsibility for obtaining pages of content is that of the URL layer 2205 When a URL is requested, the URL manager 2206 issues a request for the page and any subsequent media to the connection manager 2210 The URL manager keeps track of the requested URL for future use, if it is requested again The URL manager also queues up

URL's that have been requested according to their focus, i e , the tile that a user has cunently selected and according to the respective priorities of the active tiles

In a prefened embodiment, a pre-fetch utility such as URL pre-fetch manager 2208 can be implemented It saves the user time if items can be pre-fetched instead of waiting for their download Several strategies can be used to try pre-fetch items for the user. Using a history of a user's previous browsing habits, it is possible to predict what the user will probably want next Another function of a pre-fetch utihty is to penodically check the validity of items in the cache and to make sure they are up to date As would be familiar to one skilled in the art, some of the new HTTP1 1 methods would prove very useful for this, namely the conditional gets Another strategy is to start loading links from the page that a user is browsing, regardless of whether the user has selected the links. Although such an approach could be very wasteful of resources if there are a lot of links and very few are ultimately accessed and also because a lot of Unks tend to be advertisers, in situations where very high capacity bandwidth exists, this approach could be effective.

The connection layer 2209 handles each individual request for download passed to it through the URL manager, regardless of whether it is an HTML page, a graphic or sound file. The connection manager 2210 understands the total bandwidth available for allocation, for example, whether the device is connected to a modem or a T-l line. It will also manage the connection to the requested site and maintain its own cache. Before making a network request for an item, connection manager 2210 first checks its cache, the connection manager cache 2212 If the item is not in the cache, the connection manager then passes the request 15 off to the HTTP protocol socket 2214 in the protocol layer 2215 The way m which HTTP protocols and caches work is familiar to one skilled in the art.

The protocol layer 2215 consists of a suite of different socket types, 2214, 2216 and

20 2218, intended to support different communication protocols, such as HTTP, FTP and also a client server protocol specific to the application program via the surfcast protocol socket

2218

The socket layer 2219 comprises at least one socket 2220 The socket layer wraps

25 up all the system implementation specifics for a given platfonn and allows genenc socket types us to be built on top. The socket keeps track of its bandwidth usage, which can then be queried at the connection layer. The socket layer then facilitates bandwidth management.

With all comrnumcations going through the same socket layer it s possible to easily

30 collect data about a socket's bandwidth usage. If, at the connection layer, it is noticed that the total bandwidth allocation has been exceeded, it is a simple case of blocking further data transfer until such time as total bandwidth usage falls back under what has been allocated.

35 As a user switches focus from one tile to another priorities can be dynamically re- allocated to ensure the fastest possible loading of the selected page. All other communications can then abide by the same rules, allowing for complete control. The sequence of events and functions in a "dynamic bandwidth allocation" feature of the present invention are descnbed as follows. The dynamic bandwidth allocation feature involves the URL loader, the connection manager and the bandwidth controller

The tiles that need access to the network resource for downloading content from a

URL, pass certain parameters to the URL loader which manages all such requests from the tiles. These parameters include the URL itself, the priority of the tile, the minimum bandwidth requirement if any, and the maximum bandwidth requirement, if any.

The URL loader detects the need for a connection to a network resource, as would be notified to it by the connection manager In the case of dial-up connections, the connection manager is responsible for allocating the modem resource and making the dial-up. Once a connection is made and the network resource is available, the URL loader requests the bandwidth controller to start delivering the required content, talcing into account the additional parameters for each request

The bandwidth controller 2300 is an object that comprises a number of functions, as shown in Figure 23. AddURL 2303 is a function used by tiie URL loader to add an additional connection request to those already considered. RemoveURL 2304 is used by URL loader when cancelling or aborting a request. GetURLStatus 2306 is used by URL loader to obtain a status report for a given request GetStatus 2308 is used to obtain a general status report for the overall bandwidth. The bandwidth controller is additionally able to execute a main cycle loop over the outstanding URL requests for the purposes of managing the bandwidth among them The following pseudocode descnbes steps within the main cycle loop, according to one embodiment of the present invention

while ( uncompleted requests outstanding ) calculate bandwidth obtained per request, check for need to postpone, stall or cancel lower pπonty requests, check for need to increase higher pnonty requests, detect completed requests and notify requestor, detect undehverable requests and reissue or cancel if necessary, carry out other service and statistics functions, as required;

}

Each of these steps is explained, as follows. One or more of the steps may be performed in a different ordei from that presented above without departing from the scope _{r ι} of tiie present invennon

The step of calculating the bandwidth obtained per request utilizes a function that calculates obtamed bandwidth for each of the managed requests, mcludmg requests that are stalled (or postponed) due to pnority issues This calculation takes place frequently because of the nature of the network The bandwidth obtained can vary drastically even dunng the course of the individual network transaction, and therefore a priority based dynamic system must continuously accommodate such fluctuations The result of calculating the bandwidth obtained is used both Tor feedback to users and for making decisions in the subsequent steps within the mam cycle loop

The step of checking for the need to postpone, stall or cancel lower pnority requests is another precautionary mechanism to use for the purposes of adjusting the cunently active connections If outstanding requests are not achieving the desired mimmum bandwidth, or an actual bandwidth in line with the pnority of a given request is not achieved, bandwidth must be made available to the higher pnority requests, by stalling, cancelling or postponing lower pnonty requests A throttle feature implemented consistent with the layer nature of the stack can be applied wherein the frequency of issuing requests can be decreased A complete cancellation of a request followed by reload from cache is usually the last resort.

The step of checking for the need to increase higher pnonty requests has the opposite effect If applied, higher pnority requests are increased by means of spawning additional simultaneous requests, or by increasing the throttle mentioned above Both the steps of checking the need to postpone and checking the need to increase a priority can provide feedback to the user in terms of their success or failure. Status parameters can additionally be collected and calculated. The step of detecting completed requests and notifying the requestor utilizes a function for handling successful completed requests. Conversely, the step of detecting undehverable requests followed by reissuing or canceling the request i necessary is the mechanism for avoiding undehverable requests, or retrying temporanly unavailable requests

Other service and statistics functions may also be called, as maybe necessary for supplying in ormation to the layers above the connection layer

Client-Server Interaction

In one embodiment of the present invention, Figure 24, the user at client device

2400 interacts with server software on a server 2402. The server stores locally a profile comprising user-specific content 2406 that can Teed customized data to the user. For example, the user may store pre-defined grid configurations on the server. Additionally, passwords for specific web-sites can be stored along with the user's profile. A gri generator 2404 on the server creates a grid of tiles according to user-specified content. Each tile has been created on the server by producing an image from the location specified. For example, tile creator 2408-1 produces a tile from content 2410-1. Thus, when a user logs on to the server, for example through a conventional web-browser, a grid of tiles is downloaded to the user's system.

The client-server embodiment provides a number of advantages. For example, individual users and the devices they use maybe differentiated. Therefore, the tile-content that the server delivers can be customized according to the rendering device.

Turning next to Figure 25, each time a user logs on to the server 2402, a "session" is initiated, step 2500. The server verifies the user's identity, step 2502 and acknowledges the log-in, step 2506. The client registers one or more resources, such as connection bandwidth, cost per transmission unit and properties of local playback device, step 2504.

From this information, the server identifies the client device type, for example, set-top box or personal computer, step 2508. At step 2510, the server retneves gnd settings specific to the user, if appropnate. Details of a session, as defined by tile content and priorities can be held over from one session to another, both for the purposes of permitting a user to continue with ongoing work and m order to protect against the adverse consequences of abnormal disconnections Additionally, targeted advertising and messaging can be delivered to subsets of users via the predetermined server.

Having rendered a grid, step 2512, and delivered it to the client device, the user can request datastreams, step 2518. Any number of datastreams can be requested at once, the conespondmg stream request information which defines parameters for each stream is communicated to the server.

Upon completion of these steps, the Server knows the client charactenstics and is able to distribute bandwidth available to the client among multiple content servers, hi this example, if the client has an incoming bandwidth of say 56Kbιts/second, and is requesting datastreams from three sources with equal priority, then the Server will respond for each request that a bandwidth of 56/3 = 18.7 Kbits/sec is available for each stream The requested datastreams are displayed on the client device, step 2520 and, i f the user changes the content of a tile or explicitly requests an update or refresh operation on the tile, step 2522, an update request is sent to the server, step 2524 Once the new content has been received, the grid is rendered anew, step 2526.

Intensive operations on each displayed tile are also channeled through the predetermined server. For example, refresh operations on a tile generate a refresh request that is sent to the predetermined server. Similarly, requests to thumbnail a given image can be earned out, by request, on the predetermined server and the resulting compressed image

13 transmitted to the user. In the foregoing embodiment, the Server component may reside locally on the client machine, in which case the Server is known as the "Resource Manager", or it may be a remote server. In a prefened embodiment of client server operation, shown m Figure 26, aspects of a user's gid profile are transmitted to third parties so that the third parties may then commumcate tile based content directly to the user, For example, a user's custom gnd may contain a tile that points to a third party web-site 2604. Content 2606 from the 3^,α party web-site is typically transfened to the server for disemination to the user. The server 2602 notifies the 3 ^** party web-site that the user requires tiled data by, for example, transmitting user information 2608 The third party then permits the tile based content of its web-site to be transmitted directly to the user 5

The use of Servers also allows for the latest versions of tiles to be downloaded and installed across all devices Users are then able to share gnds and tiles with other users The Sεi ver Side Technology utilized permits users of all client devices, from desktop PC's o to mobile telephones with a consistent expenence.

The majority of the Server Side code is written in lava, with C++ being used where necessary. Inter-Server communication also utilizes XML to provide consistency with other aspects of the invention 5 in a prefened embodiment, either Oracle 8ι or SQL Server 2000 are used to provide a relational database (RDB) functionality of the server Both of these RDB's now provide direct SQL to XML transformations. Databases are developed using the ANSI 92 SQL standard, which is usable by either of the RDB's 0

Thin Client Technology

An object of the application program of the present invention is that it should operate on a vanety of devices, including mobile telecommunications devices such as 5 cellular phones, handheld web-browsers, palm pilots, personal digital assistants and other devices that can communicate by wireless application protocol (WAP).

Accordingly, because most handheld or mobile devices do not have the same amount of local storage and processing power as desktop computers and set-top boxes, a special version of the application program of the present invention is envisaged for mobile devices. The special version embodies so-called 'thin client' technology in which a lot of the operations are perfonned by a server instead of the device itself.

An "n-tier" architecture is one that is designed generically for a multitude of platforms, for example PC's, PDA's WAP phones and UNIX systems. Accordingly, in order to maximize the use of mobile devices, the application program of the present invention employs an n-tier architecture allowing the majority of the processing to be canied out on the Server with the results being sent to the device for rendenng. This model allows for a reduced size system to be stored on mobile devices, with the system logic residing on remote servers. The features that are moved from client to server will be dependent upon the device in question. For example it is possible to provide client with more features on a personal digital assistant such as a Pal pilot than on a WAP Phone.

In order to provide a level of consistency between the devices and the servers, the markup language XML is used to wrap the data that is being transmitted. As previously described, the system of the present invention uses a metabase to store information about the user's cunent grid and tile configurations. A synchronization procedure allows the metabase to be stored on a server and queried by any device. In this way it is possible to provide consistent grid and tile implementations independent of the device and its location.

Figure 27 provides an example of the way in which wireless devices can interact with a server. A personal digital assistant (PDA) 2700 or a WAP phone 2702 communicate in a wireless manner with a dial-in bank 2704. The dial-in bank communicates data to a server farm 2706 which is connected to the internet 2710, optionally through a firewall 2708. Alternatively, another personal digital assistant such as 2714 can communicate with an internet service provider such as 2712, also connected to the internet 2710. The server farm 2706 is able to provide content directly to a PDA such as 2700 or indirectly, over the internet, to a PDA such as 2714.

Claims

What is Claimed is

1 A method executed by a computer under the control of a program, said computer including a memory for storing said program, said method comprising the steps of. parti tiotung a visual display of a computer into an anay of tiles, assigning a first refresh rate to a first tile of said array of tiles and a second refresh rate to a second tile of said anay of tiles, updating information presented to said first tile m accordance with said first refresh rate, and renewing information presented to said second tile in accordance with said second refresh rate

2 The method of claim 1 wherein said partitioning step includes the step of partitioning said anay of tiles m accordance with a user-defined anay size

3 The method of claim 1 wherein said partitioning step includes the step of partitioning said anay of tiles a non-overlappmg configuration wherein each tile of said anay of tiles is a uniform size and shape

4 The method of claim 1 wherein said assigning step includes the step of assigning said first refresh rate and said second refresh rate accordance with a first pnonty value of a first information source associated with said first tile and a second pnonty value of a second information source associated with said second tile

5 The method of claim 1 wherein said assigning step includes the step of selectively attnbutmg a selected state or an unselected state to said first tile and said second tile

6. An electronic readable memory to direct an electronic device to function m a specified manner, comprising: a first set of instructions to control simultaneous communication with a plurality of information sources: a second set of instructions to anange a display into an anay of tiles; a third set of instructions to associate a first information source of said plurality of information sources to a first tile of said anay of tiles and a second information source of said plurality of infonnation sources to a second tile of said anay of tiles, a fourth set of mstnictions to retneve information from said first information source m accordance with a first retrieval rate and retrieve infonuation from said second infonnation source in accordance with a second retrieval rate, and a fifth set of instructions to present information to said first tile in accordance with said first retrieval rate and present information to said second tile in accordance with said second retrieval rate.

7 The electromc readable memory of claim 6 further comprising a set oT instructions to process a network datastream from a network data source.

8. The electronic readable memory of claim 6 further comprising a set of instructions to process a tuner signal from a tuner device.

9. The electronic readable memory of claim 6 wherein said second set of instructions anange said anay of tiles on a display of a mobile telephone.

The electronic readable memory of claim 6 wherein said second set of instructions anange said anay of tiles on a display of a television.

The electronic readable memory of claim 6 wherein said second set of instructions anange said anay of tiles on a display of a computer

The electronic readable memory of claim 6 wherein said second set of instructions arrange said anay of ules on a personal digital assistant

The electronic readable memory of claim 6 further compnsmg a set of instructions to assign a password to a selected tile of said anay of tiles

The electronic readable memory of claim 6 wherein said second set of instructions produce an airay of non-oveilapping tiles wherein each tile has a uniform size and shape

The electronic readable memory ot claim 6 wherein said fourth set of instructions assign said first retneval rate and said second retrieval rate accordance with a predetermined pnonty scheme

The electronic readable memory of claim 6 further comprising a set of instructions to selectively assign a selected or unselected state to specified tiles of said anay of tiles

The electromc readable memory of claim 6 further comprising a set of instructions to interrupt said first retrieval rate and present said first tile with static information from said first datastream

18. The electronic readable memory of claim 6 further comprising a set of instructions to deliver selected textual content from said fust information source to said first tile.

19. The electronic readable memory of claim 6 further compnsing a set of instructions to deliver a video signal to a selected tile of said array of tiles.

20 The electronic readable memory of claim 6 further comprising a set of instructions to deliver a frame of a broadcast TV signal to a selected tile of said anay of tiles

21. The electronic readable memory of claim 6 further comprising a set of instructions 15 to deliver information from a network document to a selected tile of said anay of tiles.

20 22 The electronic readable memory of claim 6 further comprising a set of instruction to deliver a web page to a selected tile of said anay of tiles.

25

30

35