WO2003090118A1

WO2003090118A1 - Database search/query system using customisable tangible user interface

Info

Publication number: WO2003090118A1
Application number: PCT/GB2003/001642
Authority: WO
Inventors: Christopher John Vernall; Robert Flint
Original assignee: Central Research Laboratories Limited
Priority date: 2002-04-18
Filing date: 2003-04-16
Publication date: 2003-10-30
Also published as: GB0208873D0

Abstract

A database search/query system includes a customizable tangible user interface comprising objects (1) which are labeled with an ID tag (2) which can be read when the objects are placed adjacent a reader (3). The tag is preferably an RFID tag, and represents a term for use in a search statement. The position of the tag relative to the reader can indicate the relative importance of the term in the search statement.

Description

DATABASE SEARCH/QUERY SYSTEM USING CUSTOMISABLE

TANGIBLE USER INTERFACE

The present invention relates to user interfaces for computers. It relates particularly to Tangible User Interfaces (TUIs).

When we interact with the world we rely on several natural abilities, among them our tangible manipulation ability. From an early age, we can move, manipulate, assemble and disassemble a seemingly endless variety of physical objects with very little cognitive effort. Yet, current Human-Computer Interfaces (HCIs) use only a limited range of our abilities and rely on a limited variety of physical objects. The 30- year-old Keyboard-Mouse-Monitor interface and the Window-Icon-Menu-Pointer (WIMP) interaction metaphor prevail as the dominant HCI. Many of our natural abilities are blocked by this, forcing complexity on what could otherwise be simple (even intuitive) HCI tasks. More specifically, we believe that novel Tangible User Interfaces (TUIs) can be employed to substantially improve existing tasks like searching for information on the WWW, and can also open the way for other applications not currently supported by computers.

TUIs give physical form to information, using objects both as representations of information and as controls to manipulate the underlying platform. They are genuinely multimodal, combining this physical element with digital representations (e.g. graphics, audio) that provide a dynamic element - showing the changing information resulting from the physical manipulations, for example. Users can directly observe the results of abstract actions initiated by manipulating concrete representations.

There is no doubt that Graphical User Interfaces (GUIs) have brought huge benefits, during a period when a more diverse population than ever before is using computers. However, there is growing evidence that direct manipulation is far more important than pictorial metaphors (e.g. the 'desktop' metaphor followed by most PCs) for assisting end users. We propose an alternative to GUIs that places the greatest priority on direct manipulation - users control the system and navigate through information by selecting and positioning physical objects, not just representations on a computer screen.

According to the present invention, there is provided a system as specified in the claims.

Embodiments of the invention will now be described, by way of example only, with reference to the accompanying diagrammatic drawings, in which:-

Figure 1 shows an overview of the system,

Figure 2 shows an example of a TUI in use, and

Figure 3 shows the overall system design in more detail.

Previous work on TUIs has tended to focus on their use for end-user programming - an example being "AlgoBlocks" (Suzuki H and Kato H, 'Interaction- level support for collaborative learning: AlgoBlock - an open programming language', Online: http://www.cscl95.indiana.edu/cscl95/suzuki.html). We contend that the topic of searching and retrieving information, particularly from the WWW, is a new area which is highly applicable and where substantial benefits can be obtained. Hence, this invention combines novel multimodal interfaces with semantic web technology.

The system can have the following highly desirable characteristics:

• Generic, to support lots of diverse applications

• Inclusive, so it can be used by multiple user groups with wide ranging mental and physical abilities

• Intuitive, so it can be rapidly learnt by new users

• Extensible, so it can be used in simple and complex environments

• Flexible, so it can be used with multiple platforms including desktop,

PDA, mobile etc • Cheap, robust and reliable - and fun to use

More specifically, the present invention provides a series of TUI applications for information search and retrieval.

This is achieved by creating a combined physical (object-based) (10) and virtual (screen-based) (11) user interface (12) to support a range of interactive tasks. This will run on the user's local machine (e.g. a PC (14)) in order to allow rapid interactions, but will communicate with a remote server (15) in order to retrieve information stored centrally or on the WWW (see Figure 1). This architecture permits a flexible approach to information storage and presentation; the user can logon from any terminal, make a connection with the server and retrieve the data associated with the objects they are using.

The object-based interface is implemented using for example RFID (radio frequency identification) tags which use radio frequency modulation to transmit data between two parts: the tag itself, which may be embedded into an object; and a reader which is placed in close proximity to the tag. The technology to be used has already been employed in recent 'talking' toys (e.g. Star Wars figures where tags trigger short samples of compressed speech when a figure is placed on a platform). These tags are widely available, are sold in the millions, and cost very little. As an alternative an optically read bar code or a data store comprising a magnetic tape could be used.

As a simple example, imagine a small plastic tree-shaped object. Placing the tree on an interactive table triggers an application to retrieve and present a predefined set of WWW data on plants. We employ this technology to show how new interface paradigms can be constructed, which are simple yet powerful, and can be integrated into many disparate applications and device types - a truly generic technology.

A suite of backend technologies is integrated, to interpret the commands and queries constructed by the user interface, and execute searches on various content sources over the WWW. In order to maximize the semantic content of the physical objects, the system also needs to build a database profiling the way in which each user interacts with the system. This is a 'learning' resource that gets richer as the system gains experience of its users.

Some concrete examples of this are:

• A business user carries an object (e.g. a tag built into their key ring or

PDA) that enables them to 'hot-desk'; placing it on a reader next to a terminal logs the user on and brings up their personal desktop and active documents.

• A music magazine distributes tags embedded in plastic icons as 'cover items'; when placed on a reader these direct the user's browser to a set of links with further information on the band in question. The same idea could also be used for business cards, which contain a tag pointing to the company's webpage and contact details.

• A teenager sends a message to their best friend by selecting a small plastic tag with their photo on, throwing it into the reader and dictating a quick voice mail, which is automatically sent to the correct recipient.

• A traveller carries a tag representing a complex profile of interests; news, sport, music and others. When placed in the reader in his hotel bedroom, the in-house system checks this profile online, and compiles a multimedia newspaper from WWW items which is shown to him on his TV.

• A child uses their personal tag (a photo of themselves) to log on at school, make contact with their home computer, and synchronize a workfile for further development at home.

• A child uses his two favourite character dolls (which have tags embedded in them) as players in an online game. Another child invites other children to play an on-line game by presenting their personal tag to the reader, while her friend agrees remotely to play the game by giving their own personal tag to the reader on their computer. • A class of children indicates their individual responses to the teacher's question by presenting either their "yes" tag or their "no" tag to the reader; the results are displayed on an interactive whiteboard.

• A user can arbitrarily define a tag to represent anything they like; a message; the results of a web search; their homework etc. All represented by a single object.

Multiple appliances and services are served by developing interfaces which can be applied equally to PCs, PDAs and even mobile devices; because the technology is simple it can be integrated with processing and display systems of varying ability. The physical metaphors stimulate and encourage fluidity between workplace, schools, homes and other venues; they are inherently mobile - even 'pocketable' - and are robust and cheap enough to survive the journey.

Overall, the present invention contributes to the objective of 'easier and more effective interactions, better usability, efficiency, quality, enjoy ability and an increased sense of realism ... ' .

The present invention, shown in Figure 2, includes a novel TUI whereby the user can associate information with small physical objects (1) TnfoGlobes'. This is implemented using simple RFID tags (2). Placing the InfoGlobe onto the reader (3) triggers a signal unique to that globe, which can be used to link to information held on a server. In other words, the InfoGlobe does not contain the information, but merely provides a link to it. The link can be to a single file, or to an entire ontology.

InfoGlobes can have almost any physical format, but iconic styling that symbolizes their content may be most appropriate. Globes can be pre-programmed with material (e.g. a dinosaur to represent a collection of media on ancient reptiles). Or, users can also use 'empty' globes to make associations with whatever material they choose. The globes can then be labeled, painted or otherwise identified, and used to capture that material indefinitely.

Because the data itself is only associated rather than stored, these devices are very cheap and need no batteries or disposable parts. The globes can be taken home, passed to friends or stored for future use. Losing or damaging a globe costs little, and can be replaced without losing the associated information by ensuring that a record of all InfoGlobes is kept on the server. A new globe (and unique ID) can simply be associated with the previous information set.

The term 'tangible user interface' or TUI should be credited to Hiroshi Ishii from the Tangible Media Group at the MIT Media Lab. TUIs have been investigated for many years, but are only recently emerging as realistic alternatives to standard mouse, keyboard and GUI systems. One of the earliest works was Bishop's influential answering machine; this interface coupled voice messages with physical marbles, allowing these messages to be replayed, their callers to be redialed, and messages to be stored through manipulation of the physical marbles.

The toy industry has several examples of tangible interaction tools between children and computerized applications or 'huggable interfaces'. However, perhaps the most well-known children's application is the work done at MIT, for example the 'tangible computation bricks' project where microprocessors were embedded into Lego bricks, so children could construct arbitrary objects with in-built intelligence. A further example is the Zowie system, a commercially available product where physical models of game characters are moved around a toy ship to drive an interactive PC application.

Examples of the tactile representation and manipulation of information are discussed in the literature (Fitzmaurice G, Balakrishnan R and Kurtenbach G, 'Sampling, Synthesis and Input Devices', Communications of the ACM, 42(8), pp.54- 63, New York, ACM, August 1999).

On a technical front, the present user interface is designed to operate with a variety of devices and rendering platforms. The state of the art until the advent of the UIML (User Interface Markup Language) specification was that interfaces needed to be specifically designed and implemented for a single platform or rendering system. UIML2 enables the present invention to integrate a single interface design for the wide variety of current and future platforms (PCs, mobile phones etc). The present invention uses an approach to the search and retrieval of information from the WWW, which builds upon and extends the above physical metaphor. The user is considered as an information source and the tools developed here as receivers and filters that transform the input (the query) into an appropriate response (the result set). The cost of information transfer is minimised at both ends.

The user should first be able to formulate his/her query with minimal redundancy.

In particular, some memory of invariance within the user query should be kept and recalled at query time. Further, the response should be presented (or rather represented) in a way that makes the information easy to grasp by the user.

To define a search, the user drops InfoGlobes onto an RF reader pad, each of which symbolizes topics or themes such as biology, history, geography, art, movies etc. These are used as input criteria; by combining a number of globes on the reader, complex searches with multiple criteria are created. An important feature of the invention is the contextual association of the topics embodied by multiple globes, e.g.:

• Does the order in which the globes are placed on the reader imply hierarchy?

• Does the same globe have different meanings when associated with other globes?

The physical location of the globes on the reader can also help define the search. By defining sensitive zones on the reader pad, more important criteria can be placed near the center of the reader, whilst criteria with lower importance are placed further away. The position can be altered by the user in response to the search results, changing the weighting of criteria and modifying the results.

A simple example would be a child researching history, who first places a

"history" globe onto his search pad and then combines this with an "art" globe to signify that they are interested in the history of art. They then add an "image" globe that instructs the agent to return only pictures as results. Finally, they add a variable globe in the shape of a clock, which symbolizes "time". They then view a sequence of images returned by the agent, and tune the search forwards and backwards in time interactively to find the sequence they require.

An objective is to allow the user to very simply and quickly compile and vary their searches, but also have the ability to explore and discover new and relevant information themselves. It allows them to be able to create interconnections between subjects, periods of time etc. rather than just learning in a conventional "linear" fashion.

Nowadays, content-based information retrieval systems suffer from the contrast between the plethora of forms in which the information is made available, and the limited possibilities to express the semantics of required information. Even simple scenarios can lead to complex, repetitive and even unmanageable search activities. The main reason is the way in which users may construct queries: this is predominately limited to text descriptions that suffer from ambiguity of language semantics. There are no easy and intuitive possibilities for an average person, not being an information retrieval specialist, to express their requirements by a free description using querying by complex positive and negative examples. Widely available "concept search" and "query expansion" technologies are cumbersome and can easily lead the naive user in a wrong direction.

The present invention provides solutions to these problems by investigating at least two directions: user profiling (or system personalisation) and the design of physical devices or metaphors for information retrieval. Both encompass a high level representation of the information and therefore are directed towards semantics rather than basic contents.

The definitions of efficient languages and interfaces necessary to implement appropriate solutions in a distributed environment are the key for such a development. The most widely used standard, the aged ANSI Z 39.50, is restricted to client-server paradigms and supports only the querying and retrieval of bibliographic information. Although used outside its originally intended scope, it deserves a successor better supporting currently used technologies (three tier architecture, Web harvesting, XML-compatible structured multimedia information, etc.)

The Multimedia Retrieval Markup Language (MRML) is seen as a first step in these directions. MRML is an XML-based communication protocol that enables the separation of the query processor (i.e. the part that transforms the query into a response) and the client (i.e. the interactive part where the query is formulated and the results are shown). MRML is designed to be as simple as possible to integrate into any software environment. The client can be any MRML-compliant user interface (e.g. JAVA applet, PHP interface, plug-in for GIMP or Konqueror). The advances made in relation to the SOAP (Simple Object Access Protocol) standard from the XML Protocol Working group of the W3C .are also relevant. This will be of importance to help maximize system compatibility.

There is the ability within the scope of the present invention to develop a form of Physical Query Language facilitating the development of innovative applications that bind the interface and backend information retrieval processes together. In the Web context, such a language is necessary to develop novel approaches to personalized information retrieval, using specialized intelligent agents, which enable profiling of individual preferences and weighting of interest criteria.

In particular, a complex information retrieval process compiled using a combination and juxtaposition of physical objects, can be disambiguated and translated into queries suitable for conventional search engines.

This function may be executed by a Query Processor (22) that will intervene between the user interface and the search engines that perform the retrieval.

The overall system architecture is illustrated in Figure 3. This shows the following aspects:

• A distributed architecture of clients 14 and servers 15, communicating via the XML-based MRML communication protocol 16. The MRML traffic itself is analyzed and used to provide semantic profiling of the user.

• A user profiling database 17 that takes account of user dynamics. This allows the system to more accurately gauge the needs of the user in terms of content, and will be able to change over time (e.g. a 10 year-old girl will not want the same type of results to a search for toys, as a 5 year-old boy). This database contains both explicit and implicit data. Explicit data includes: Age; Gender; Interests; Geographic location; Language; Display capabilities; School level etc. Implicit data includes high- level features extracted from statistical analysis of the MRML logs that reflects user relevance feedback interaction.

• A personal mediator agent 18 to handle the search of external database sources. This interposes between the user interface and the media sources. The complexity of the search variables will necessitate the use of an agent system that is capable of searching in various sources and using variables such as the weighting given by the interaction between the user and the user interface. This will comprise a Query Processor. The agent will also extract the appropriate user profile from the database in response to user interactions.

• A mobile agent system to search through the searches of other users held on the central servers of the system. If there are a large number of users there may be a large amount of information which can be recycled by other users and which may also have been customized to better suit their needs. Those users that do not want their information shared can opt out of the service at any time. The mobile agent search facility has advantages over the personal mediation search facility in that it can continue searching even while the user has their machine turned off and can therefore be used to find material on a daily basis. This is even more useful if it is used on a homework server at schools such as are available in some areas of the US. The disadvantage of intelligent agents is that they need to travel between host servers which are compatible with them, this is why the present invention uses the FIPA 2000 standard so as to ensure the widest possible compatibility.

• A framework containing a plug-in mechanism for query engines 19. Two approaches are described below for implementing this.

Query translators to convert generic queries into specific queries suited to 3^rd Party search engines (e.g. Google, Yahoo!) which will allow browsing and searching on the WWW 20. > bespoke query engines, based on customized versions of freeware engines to minimize resource expenditure. These are used to attempt to gain greater relevance and more highly targeted search results.

• Because of the size of the WWW and the difficulty of ensuring the suitability of material the invention will initially provide a "walled garden" service where information will only be retrieved from a select number of sites. Later, there will be facilities for information browsing as well as searching, through the implementation of two services :

> A web spider service which will constantly search on the web for suitable material.

> A web directory 21 to be maintained by humans to ensure the suitability of content.

Claims

1. A database search/ query system having a customizable tangible user interface (TUI), characterised in that the interface includes one or more objects (1) which are labeled with an ID tag (2) which can be read when the objects are placed adjacent a reader (3).

2. A system as claimed in claim 1 in which the tag is an RFID tag.

3. A system as claimed in claim 1 or claim 2 in which the tag represents a term for use in a search statement.

4. A system as claimed in claim 1 or claim 2 in which the tag represents the location of data associated with said one or more objects.

5. A system as claimed in any preceding claim in which the one or more objects is/are adapted to be moveable by hand relative to said reader.

6. A system as claimed in claim 3 in which the position of the tag relative to the reader indicates the relative importance of the term in the search statement.