WO2010023348A1

WO2010023348A1 - Interactive displays

Info

Publication number: WO2010023348A1
Application number: PCT/FI2008/050475
Authority: WO
Inventors: Tommi Ilmonen
Original assignee: Multitouch Oy
Priority date: 2008-08-26
Filing date: 2008-08-26
Publication date: 2010-03-04
Also published as: EP2332027A4; EP2332027A1; CN102132239A; CN102132239B

Abstract

An interactive display (100) is provided by providing a user with a perceivable visible image on an image layer (102) configured to and configured to pass through infrared light, capturing an infrared image through the image layer (102), and detecting (303) from the infrared image contact areas where a pointing object (112) is located at the image layer (102) and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object (112) and a central region surrounded by the peripheral region. The detecting (303) of the contact area is performed under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions (310) of the shadow.

Description

INTERACTIVE DISPLAYS

FIELD OF THE INVENTION

The present invention generally relates to interactive displays.

BACKGROUND OF THE INVENTION

There are interactive displays which comprise a projector or a Liquid Crystal Display panel to form an image on a surface, a camera behind the surface and computer vision circuitry for detecting user input.

US 2001/0012001 A1 discloses one such display system. This system comprises a semi-transparent screen and an infrared LED (light emitting diode) panel, a CCD (charge coupled device) camera and a projector provided on the back side of the semitransparent screen. The camera operates on infrared wavelengths and detects infrared light reflected from objects on the other side of the semitransparent screen, when the objects are relatively close to the screen or when they touch the screen. The projector projects visible image light onto the screen. Infrared component of the image projected by the projector may be filtered out. Thereby the projected image does not disturb the camera.

ACM publication 'ThinSight: Integrated Optical Multi-touch Sensing through Thin Form-factor Displays" by Izadi et al. discloses another type of interactive displays.

In this publication individual infrared emitter/detector pairs placed behind backlight of an LCD (liquid crystal display) display are used to detect objects on or in the vicinity of the LCD display, the LCD display thus operating as an interactive display. The use of an LCD display may overcome many of the problems of the projector-based interactive screens. The disclosed technology however requires number of infrared emitter/detector pairs to be mounted on the system, if input is to be detected on a larger area, whereby production of such interactive displays is laborious and thus high costs may be involved. In the absence of ambient IR light, the display may illuminate objects on or near its surface by means of IR illumination, but it is very difficult to counter excessive ambient IR illumination. Detecting an object on the surface of the display with an IR camera is complicated by the presence of sunshine or other strongly IR- emitting illuminators such as halogen lamps. Namely, in this case the infrared- emission that the system produces may not be strong enough to cause near-by objects to appear as highlights. For instance, one known technology for detecting a finger touching the display surface is to detect when an image of the finger becomes crisp. A diffusive screen is attached onto the display surface so that a touching finger blocks IR light from being diffused by the diffusive screen under the finger and a relatively sharp shadow is formed conforming to the shape of the contact. However, under strong, coherent light such as sunshine, any object casts a sharp shadow on the camera even from a distance. Hence, it may be impossible to detect a contact simply from the sharpness of the shadow. Moreover, under strong ambient IR light the camera signal may be flushed with measured light such that it is generally impossible or at least very difficult to distinguish any meaningful picture.

US20050064936A1 describes a method in which a finger may be recognized from its shadow and distinguished from other shadows by using parameters which are characteristic to a finger, such as color, shade, reflectance, particular grey color change at the perimeter of the shadow caused by the skin of a finger, and by making use of the expected use based on the layout of usable radio buttons, see paragraphs 239 to 246. This publication lists some optical parameters without explaining how they should be used and moreover requires very accurate color cameras in order to detect fine effects such as the grey boundary through the skin at the edge of the finger. This publication deals with particular problems present in a car equipment, but does not address a situation in which there is a strong single backlight behind the finger.

WO0184251 A2 presents a panel display with which movement of a finger is being detected by using histograms, see e.g. page 16 and Figs. 10a-11f. The use of histograms is a known image processing method in which pixels are compared. The disclosure of this publication does not disclose whether it could be deduced whether a shadow is caused by a proximate or distant shadow. The publication neither discloses how to address problems caused by ambient light. Further, the use of histograms of this publication loses the shape of the finger, whereas the shape would be needed for computing the width of a finger. The publication is focused on light areas rather than shadows. This technique detects spot-like objects that touch the screen and are illuminated from under the screen so that a touching object reflects a clear illuminated spot and appears as a light area in captured images. WO0184251 further requires computationally intensive histogram calculations and database comparisons so that its implementation requires relatively expensive and complex equipment.

WO2004091956A2 discloses a vehicle display device panel. This publication discusses using of ambient light i.e. back illumination of a finger such that the finger is seen as a shadow, as described in the paragraph bridging pages 56 and 57. The size of a shadow caused by a finger may also be used for determining whether a particular pressure is applied against the screen, if a dedicated piezoelectric sensor is not used (page 51 to 52). Clearly, in order to identify the force of a finger tip pressing against the display, the finger should be pressed in a consistent angle against the display. In a car, the geometry between the driver's hand and a display mounted on the dashboard may determine that the angle of a finger is relatively consistent so that the width of the finger may indicate the force of the finger as the image caused by the finger then broadens. However, the inventors have realised that if the geometry is not that well defined, the finger may be placed in a widely varying angle in comparison to the display such that the shape and size of the shadow may vary in a very broad range. Hence, the disclosed detecting of the force of a finger against the screen by measuring the size of the shadow is not universally usable for detecting the point of touch by a finger.

It is an object of the present invention to avoid problems associated with prior art and/or to provide an alternative to existing technology. SUMMARY

According to a first aspect of the invention there is provided an interactive display device comprising: an image layer configured to provide a user with a perceivable visible image and configured to pass through infrared light; an image capture unit configured to capture an infrared image through the image layer; a recognizing unit configured to detect from the infrared image contact areas where a pointing object is located at the image layer and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object and a central region surrounded by the peripheral region; wherein the recognizing unit is configured to detect the contact areas under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions of the shadow.

According to a second aspect of the invention there is provided a method in an interactive display device, comprising: providing a user with a perceivable visible image on an image layer configured to and configured to pass through infrared light; capturing an infrared image through the image layer; detecting from the infrared image contact areas where a pointing object is located at the image layer and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object and a central region surrounded by the peripheral region; wherein the detecting of the contact areas is performed under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions of the shadow.

According to a third aspect of the invention there is provided a computer program configured to cause when executed by a computer a method according to the first aspect of the invention.

According to a fourth aspect of the invention there is provided a computer readable memory medium embodied with a computer program which when executed by a computer causes a computer to perform a method according to the first aspect of the invention.

Various embodiments of the present invention have been illustrated only with reference to certain aspects of the invention. It should be appreciated that corresponding embodiments may apply to other aspects and embodiments as well to produce further non-limiting examples.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention will be described, by way of example only, with reference to the accompanying drawings, in which:

Fig. 1 shows a block diagram of a system according to an embodiment of the invention;

Fig. 2 shows a simplified block diagram of the structure of a control unit shown in Fig. 1 according to an embodiment of the invention; Fig. 3 shows a schematic flow chart according to an embodiment of the invention for illustrating a first phase in a process of determining a pointing object from an Infrared (IR) image captured by an IR image capture unit from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light;

Fig. 4 illustrates according to an embodiment of the invention for illustrating a second phase in a process of determining a pointing object from an IR image captured by an IR image capture unit from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light;

Fig. 5 shows one graph demonstrating a shadow gradient width test;

Fig. 6 shows another graph demonstrating a shadow gradient width test; and

Fig. 7 shows an example of an infrared image of a pointing object when partly in contact with a display.

DETAILED DESCRIPTION

In the following description, like numbers denote like elements.

In the following examples of various embodiments of the invention an LCD screen is used as an example of an image layer. The LCD screen may comprise a film that forms a visible image and optionally also other elements such as background illumination, infrared (IR) illumination, incoming IR intensity detection across the LCD film, and/or one or more speakers for outputting sound. It is appreciated that the invention may likewise be applied with any other types of image layers as well, or screens for simpler expression. Generally, an image layer is any means for producing a visible image for a user, such as an electric display, a screen displaying a projected image or a substance carrying printed, dyed image, or woven image. However, the image layer should be sufficiently transparent or translucent for IR light that is used for detecting a pointing object through the image layer as will be explained in more detail in the following. Fig. 1 shows a block diagram of a system 100 according to an embodiment of the invention. Fig. 1 also shows a user 113 for facilitating the description of particular order of different elements. The system 100 is suited for use as an interactive user interface device e.g. as a built in dynamic menu in a restaurant, as a display screen at a ticket office, or generally wherever an interactive display and optical pointing recognition is desired.

The system 100 comprises as an outmost element or as facing to the user 113 a touching layer 101 such as a toughened glass plate, then an LCD film 102 and a diffusion layer 103 behind the LCD film. Preferably, the touching layer, the LCD film 102 and the diffusion layer 103 are all in a compact stack such that the distance from the touching layer to the diffusion layer is as low as possible for machine view performance reasons that will be described in more detail in this description.

The purpose of the diffusion layer 103 is to spread the light coming from a background light source (described in the following), so that an image displayed on the LCD film appears even from many directions. This spreading can be achieved with a diffusion film or with a holographic rear-projection film. By placing the diffusion layer 103 behind the LCD film 102, significant advantages may be achieved in comparison to placing the diffusion layer 103 in front of the LCD film 102 or to omitting the diffusion layer 103. Namely, the diffusion layer 103 reducing reflections from the typically glossy backside of the LCD film 102, which reflections may interfere with the recognizing of pointing objects. It is also advantageous to locate the diffusion layer 103 behind the LCD film 102 when seen by a user 113, because otherwise it provides an additional reflecting surface between the LCD film 102 and the viewer thus impairing the image quality or contrast especially.

In order to support the aforementioned optical elements, there is provided a casing 104. The casing 104 comprises a back wall 105 attached to side walls 106. The side walls are attached from one end to the back wall 105 and from their opposite ends to the touching layer 101 , LCD film 102 and diffusion layer 103. A background light source 108 may be located in the casing 104 for background illumination of the LCD film 102. The background light source 108 may comprise, for example, one or more of: LED light, light conductor, fluorescent light, and luminescence light.

In an alternative embodiment, the diffusion layer 103 is omitted. This may particularly be the case when an evenly illuminating background light is provided by a luminescence light that in itself provides an even field of illumination on the LCD film 102.

The side walls 106 may be coated from the inner side with some reflective material in order to deliver maximum amount of light to the LCD film and finally to users of the system. By means of the reflective side walls one may avoid or at least reduce shadows that may be caused to the background light in the IR image captured by a camera behind the LCD film 102. The reflective side walls may also help in delivering the background light to the LCD film in a way that the light can be spread to the users without allowing users to see details inside the system through the LCD film and at the same time improving the viewing angle of the system. In addition to the reflective side walls (or reflective casing) the diffusion layer 103 may help to achieve this effect. More particularly, the side walls may be for example mirror walls, in which case a particularly good consistency of an image on the image layer may be achieved.

Inside the casing, there is an image capture unit 107 that is in this particular case an IR camera configured to see through the diffusion layer 103, LCD film 102 and the touching layer 101 any objects placed near or at the touching layer 101. Further in Fig. 1 , there is drawn as a dotted area an IR light reception space 107 for the IR camera. The camera is configured to detect signals that are outside the visible light wavelengths. There may be for example a filter in front of the lens of the camera providing this effect. The camera may be based on for example CCD (charge-coupled device) or CMOS (complementary metal-oxide-semiconductor) technology. Moreover, the image capture unit 107 may comprise more than one camera e.g. for better resolution, for smaller distance between the screen and the cameras, or for covering larger LCD screens 102. The distance between the background light source 108 and the diffusion layer 103 may generally depend on the space that the camera setup requires. The distance may be shortened for more compact sized system e.g. by moving the camera forward and/or turning the camera around (to point away from the screen) and using a mirror or prism in front of the camera to capture the activities on the screen.

Alternatively, the image capture unit 107 may be formed of a planar sensing structure that has a number of sensors spread over a plane to recognize intensity of incoming light through each pixel or pixel block of the LCD film 102. Such a structure may also double as a visible light and/or IR illumination behind the LCD film 102 for the purpose of background illuminating the LCD screen and/or illuminating objects in front of the LCD screen under weak ambient IR illumination. In case of a planar sensing structure operating as an image capture unit 107, there is no IR light reception space 107 as that drawn in Fig. 1 but instead there is merely a small parallel space between the LCD film and the image capture unit. Further, the image capture unit may be integrated directly into the image plane, for example the LCD film.

The camera 107 and the IR LEDs 110 may not be seen from the outside of the system if their shadow is negligent e.g. due to the highly diffused illumination coming from the background lights and as the reflective inner side of the system provides evens further illumination of the display surface. Further, the diffusion layer 103 may prevent the users from seeing any details from inside the system.

A camera as the IR image capture unit typically provides a more economical solution than a plurality of emitter/detector pairs that are typically used in a planar illumination and sensing element. First, the area covered by one camera typically requires a relatively high number of emitter/detector pairs thus incurring a relatively high number of components and wires. Second, production of the system may be simpler when a camera is used. On the other hand, by using emitter/detector pairs in a planar image capture unit, the size of the system may be reduced and there is no need for accurately positioning and directing the camera in a desired angle with regard to the LCD film 102.

The system 100 may further comprise an IR reflection inhibitor 109 configured to reduce reflection of IR light from the back wall and again from any elements in front of the camera 107. The IR reflection inhibitor 109 may comprise, for instance, a black plate such as a sooted aluminum plate placed around the background light source 108 or behind the background light source 108.

Alternatively, or additionally, IR reflection inhibitor 109 may comprise, for instance, a directionally refractive or reflective element such as one or more prisms configured to direct IR light to such a direction that the amount of IR light reflection to the camera 107 is significantly reduced.

Further still, the IR reflection inhibitor 109 may comprise additionally or alternatively a selective screen between the background light source 108 and the IR light reception space 107.

A screen that filters out IR light may be simply arranged e.g. in a plane configuration behind the camera parallel to the back wall 105 in sake of simple installation. Alternatively, the screen may be conically arranged partly on the front side of the camera 107 when seen from the direction of the user 113. While a planar configuration is economical and simple to arrange, conical or different curved arrangements may further enhance the operation of the IR reflection inhibitor 109, whether implemented using a filter or an absorbing surface. Namely, there is always an amount of reflection and by suitably shaping the IR reflection inhibitor 109 it may be possible further reduce reflections of IR light all the way from the outside of the case 104 through the LCD and back and forth inside the case 104 to the camera 104.

The IR reflection inhibitor 109 represents an advantageous implementation that may be used in the present invention, but this document is more particularly focused on the way in which a pointing object 112 is determined from the IR image. To this end, there is provided a control unit 11 , which is configured to control operation of the system and/or to detect the pointing object 112.

Additionally the system 100 may comprise IR light sources 110 enabling input detection on IR wavelengths. The IR light sources may be for example IR LEDs placed outside the angle of view of the image capture unit. In case that a camera is used as the image capture unit, the IR light sources 110 may be located outside a cone formed by the view area of the camera. On the other hand, if a planar image capture unit behind or integrated with the LCD screen is used, the LCD screen itself may be configured to provide the IR illumination across its view area or the IR illumination may be configured to be produced such that it illuminates objects at the LCD screen without first passing through the LCD screen.

The system 100 may further comprise an audio speaker 114 for providing audible signals to the user 113. The system may be configured to e.g. provide a tapping sound to indicate determined tapping on the touching surface 101 for enhancing user experience of an operable system and to teach users that there is no need for applying substantial force against the touching surface when the recognition of the pointing object 112 is optically performed.

Fig. 2 shows a simplified block diagram of the structure of the control unit 111. The control unit 111 may be based on, for example, a general purpose computer supplied with suitable software and / or on a particularly adapted computing device. While it is possible to implement the control unit 111 by purely hardware based a device, typically it is more economic and faster to produce by making use of software.

In Fig. 2, the control unit 111 is drawn to comprise a memory 201 that comprises a work memory 202, a non-volatile memory 203 that is configured to store software 204, presentation information 205 describing content to be presented by the system 100 and/or how pointing at different areas on the screen should be treated, and settings 206 needed e.g. for manual or automatic calibration of the system 100. The software 204 may comprise any one or more of the following items: operating system, device drivers, display presentation application, hypertext markup language parser, image processing software, and drivers for different external equipment that may be connected to the system such as printers, further displays, further interactive systems 100, audio systems, and external IR illumination equipment (not shown).

The control unit 111 further comprises a processor 207 configured to control the operation of the control unit 111 according to the software 204 by executing computer executable program code contained by the software in the work memory 202. Alternatively, the control unit may be configured to execute the software in place in the non-volatile memory in which case the work memory may not be necessary. The control unit further comprises an input/output unit (I/O) 208 for exchanging signals with other elements of the system 100 and optionally also with external equipment. The I/O 208 may comprise e.g. any one or more of a universal serial bus port, a local area network port, an ISA bus, a PCI express port, an IR port, a Bluetooth element, and a parallel port. Alternatively to being configured capable of communicating with external equipment, the system 100 may be provided with a transferable memory reception unit 209 such as a cd-rom or dvd- rom drive, memory card reader or memory stick reader which enables replacing part of the non-volatile memory e.g. for updating information to be displayed on the LCD screen 102.

In order to control the operation of various components of the system and to obtain the captured image, there are connections between the control unit or particularly its input/output unit 208 and other components of the system 100, while not shown in sake of clarity of the drawing. The control unit has generally the task of receiving a signal from the camera 107, detecting if and where the touching layer 101 is pointed at and typically also outputting the determination in a standard way e.g. emulating a computer drawing tablet, mouse or other known pointing device.

Generally, the control unit operation may comprise following acts: - controlling the LCD film to show desired images to the user 113 - controlling the IR lights 110 to produce IR light on demand for showing a pointing object 112 such as a user^'s 113 finger when brought close to the LCD film

- obtaining signals corresponding to received IR light from the image capture unit 107

- detecting from the received signals the pointing object at the touching surface 101

- performing a predefined action based on the detected input, e.g. changing the image displayed on the LCD film 102 or following a hyperlink associated with the area at which the pointing object is detected

- detecting the amount of ambient IR light controlling the IR lights 110 accordingly

It is appreciated that while the control unit may consist of one separate unit, the control unit 111 may alternatively be integrated with any other element or comprise two or more discreet elements each for one or more of the aforementioned acts.

Fig. 3 shows a schematic flow chart according to an embodiment of the invention for illustrating a first phase in a process of determining a pointing object from an IR image captured by an IR image capture unit 107 from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light. In the process a finger tip is sought as an example of a typical pointing object 112 at the touching layer 101 by the control unit 111. It is appreciated that the system 100 operates by continually repeating the following process to determine a pointing object when brought at the touching layer 101. However, in sake of simplicity, the flow chart presents the process for one single time of verifying if and where there are pointing objects at the touching layer 101. It is further appreciated that while in the following process more than one pointing objects may be identified, the process may alternatively be stopped once any one pointing object is found if so desired. It is an advantage of attempting to identify all pointing objects that if a user e.g. places many fingers on the touching layer 101 , many different contacts may be identified and all or none of respective areas may be detected as being pointed depending on the implementation. The process starts 301 when the camera operates and also typically the LCD film

102 is presenting some still or motion picture to the user 113 and the control unit receives a digitized IR image from the camera or from an analogue-digital converter (possibly comprised by the control unit) in case that the image capture unit 107 has an analogue output. The digital IR image comprises a set of pixels typically such that the area corresponding to the touching layer 101 is divided into a matrix of x columns and y rows. In the matrix, each cell represents one pixel with a value that is typically of 8 bits depth i.e. has 256 different numerical values representative of the IR luminosity detected by the image capture unit for that cell.

Next, in step 302 an initial pixel is selected. The initial cell may be chosen e.g. near an area corresponding to one corner of the touching layer such that the distance to nearest edges is about half the thickness of a thin finger tip.

In step 303 it is then checked whether the luminosity value of the selected cell is significantly below its neighboring pixels. This test may be carried out e.g. by dividing the luminosity value of the selected cell by the highest luminosity value in the cells at a given distance around the selected cell or by subtracting from the highest luminosity value of cells at a given distance around the selected cell and comparing the result to a predetermined threshold. Alternatively any other linear or non-linear comparison method may equally be applied.

If the result the checking in step 303 is negative, then the selected cell is not likely one representing a finger tip and the process advances to step 304 to verify whether all feasible cells have been tested i.e. whether the testing should be finished. If yes, the process ends at step 305, alternatively the process advances to step 306 in which a next pixel is selected for testing. The next pixel may be selected by stepping onwards along a slice of the image by a distance of half the width of a narrow finger tip or less. Alternatively, coarser sampling may be employed to reduce required computation or denser sampling may be employed for better accuracy. Next, the process resumes to step 303 for again checking whether a promising starting point would be selected for verifying whether it belongs to a shadow of a finger tip.

In step 305, once the selected pixel is sufficiently darker than its neighbors, a threshold luminosity value is defined between the background luminosity and the pixel luminosity. To define the value one may estimate the background luminosity with various methods. For example one can search the near-by pixels for the highest value, which would then be the luminosity value. Alternatively one can choose a subset of near-by pixels, for example by shooting short rays into different directions and collecting the brightest pixel along the rays. Further, one can take histogram of the near-by pixels and instead of taking the very brightest pixel, use a pixel value that is selected from the histogram at a given positioning. To then find the threshold value lth for step 305 one can calculate a weighted average between the selected pixel brightness lpi and environment brightness len, so that lth = a ^■ lpi + b ^■ len, where a and b have a sum of 1.

In step 308, the control unit next shoots rays in different directions preferably with even angle offset e.g. into 8 different directions and checks the luminosity at each pixel along the rays, one by one until a pixel is found along the ray with a luminosity above a given threshold indicates that an edge region has been reached in the shadow of a suspected finger tip.

In step 309, it is tested whether a given number, here at least two, of the shadow lengths exceed or reach a given threshold. If yes, then the selected pixel is not likely to belong to a finger tip region and the process resumes to step 304. If no, the lengths of opposite rays are summed up to form the length of a shadow of given darkness in four different directions with 45 degree offset (in case of eight evenly distributed rays). The width of the finger tip is initially estimated 310 by choosing from the summed lengths of rays of opposite directions i.e. lengths along a direct line, by rejecting summed lengths which exceed a predetermined maximum length threshold and then taking the shortest of the remaining lengths. It is appreciated that the order of the processing steps may be varied e.g. so that already in step 309 the summed lengths would be used. If all of the summed lengths exceed the predetermined maximum length threshold, the process may jump to step 304 in an embodiment in which such a finding is held as an indication that the selected pixel is not near an edge of a pointing object.

After step 309, when the estimated width of the finger tip (shadow) is calculated, the estimated width is stored and the selected pixel is marked as a potential finger tip pixel. The process then continues from step 401 in Fig. 4.

Fig. 4 illustrates according to an embodiment of the invention for illustrating a second phase in a process of determining a pointing object from an IR image captured by an IR image capture unit 107 from behind of the pointing object under clear daylight or strong and relatively coherent ambient IR light. Following step

311 , the process continues from step 401 at which potential finger tip pixels are merged into groups by using any commonly known flood fill technique, for example recursive fill, right-hand fill or scan-line fill. The flood fill produces a group of pixels representing an arbitrary shaped estimate area on the touching layer 101.

Next, width of the estimate area is calculated 402 in four adjacent directions with 45 degrees steps and an average of these four widths is calculated as a refined width estimate for the finger tip shadow. This calculation is done by averaging the length values of each of the pixels in the group.

After step 402 or between steps 401 and 402 the merged area is tested 403 by comparing the number of pixels within the estimate area resulting from the flood fill to a given maximum area and if the estimate area is too small (or in one embodiment if the area is too large), the selected pixel is rejected from consideration as a likely pixel representing a finger tip, and the process resumes to step 304 of Fig. 3. Otherwise, the estimate area size testing is passed and the process may continue.

It is appreciated that a number of different methods can be developed to detect finger tips from the infrared image. For example one can use a two-dimensional filter-bank to detect fingertips (a process that can be optimized with the use of fast Fourier transformation). Likewise, one detection mechanisms such as Bayesian estimators may be used. For processing purposes, the image can be presented in a number of ways. The methods may work on raw pixel data (as the algorithm in figure 3) or use for example low-pass (and/or high-pass) filtered versions of the image, or perform a global or local optimization or normalization on the image. Also any other commonly known image processing methods may be applied.

In step 404 a maximum allowable luminosity l_max is calculated based on the refined width estimate. The maximum allowable luminosity may be derived for example by dividing the maximum luminosity value of the nearby pixels (that are clearly outside group) with some numeric constant, for example 0,3. Further l_max may be limited so that it is never less than certain percentage of the maximum luminance value of camera image in the whole tracing area. In effect, this step dynamically adjusts the threshold for determining whether a shadow cast by the finger tip is dark enough. Naturally other strategies may be applied for this purpose. For example, a histogram of the near-by pixel values may be calculated and the histogram may be used as an input to numeric analysis tools such as Bayesian estimator or artificial neural network that computes the value for l_max. Also other linear- and non-linear methods may be used to calculate the value for l_max.

In step 405, a central pixel or an average of a predetermined number of central pixels of the merged set of pixels is identified and a center luminosity value is derived from the central pixel or pixels.

In step 406, the center luminosity is compared to the maximum allowable luminosity and if the center luminosity does not exceed the maximum allowable luminosity l_max, the process advances to step 407 wherein the merged group is accepted as a finger tip shadow and the process may in advance to step 304 to check whether the captured IR image is completely scanned or whether still further finger tips should be sought from the image. If, however, in step 406 the center luminosity is found to exceed the maximum allowable luminosity l_max, the merged group of pixels is not considered as a finger tip shadow and the operation proceeds to step 304.

It should be appreciated that the method takes implicitly into account the sharpness of the shadow. This sharpness information may be used in steps 308 and 309 in the following way: if the shadow is not sharp, then the lengths measured in 308 will be relatively long and the non-fingertip will be rejected in step 309, due to the long rays measured in step 308.

There are also alternative ways to check if the shadow is sharp enough to be caused by finger contact: For example, a shadow gradient width test may be added. The shadow gradient width test may verify that the shadow is sharp enough so that it is not caused by far-away objects, under diffuse ambient illumination. One way to perform this test is to shoot rays into different directions from the estimated center of the fingertip as before the rays can be shot into eight directions. For each ray one then calculates two luminosity threshold values: start value Gb for the beginning of the gradient and end value Ge for the end of the gradient, based on the luminosity at the center of the finger and luminosity outside the center of the finger. Examples of how the luminosity may behave along the ray are presented in figures 5 and 6. As the ray progresses away from the fingertip it is assumed that the luminosity will first exceed the start value Gb at a distance d1 from the finger tip center, and the luminosity of exceeds the end value Ge at a distance d2 from the finger tip center. In figure 5 shadow gradient width i.e. the distance between the start value Gb and the end value Ge crossing is short, so the shadow is sharp (thin gradient around the shadow center), while in figure 6 the distance between respective Gb and Ge crossings at respective distances dT and d2' is substantially longer, so the shadow gradient is weak along the ray. One should note that the gradient may be weak towards many of the rays, but strong towards other directions for example if the finger is not perpendicular to the screen. When one has the gradient widths for all rays, one can test if most of the rays have a long gradient, in which case the fingertip is not touching the surface, but further away. It shall be appreciated that in this description various thresholds are applied for classifying different parameter values and that is merely a matter of implementation whether the threshold is set to require exceeding or reaching some threshold or being smaller or not larger than a threshold. It is further understood that in digital computing, it makes no difference whether higher values are used to indicate higher luminosity of a given pixel or vice versa, but in this description, the examples assume that lower IR intensity of pixels results in lower luminosity values.

Further, it is appreciated that one can implement the luminosity comparisons using methods that work on abstracted data structures, for example by calculating radial histograms starting from the center of the potential finger tip pixels and analyzing the histograms with statistical tools. Such methods may be configured to present the darkness of the center of the potential fingertip and other relevant parameters such that calculations may identify which shadows are caused by objects in contact with the display, and which are caused by objects further away from the surface.

The control unit may also be configured to automatically detect the edges of the LCD screens and to calibrate the parameters that govern the transformation from camera co-ordinates to screen-co-ordinates. Additionally the software may be configured to automatically eliminate the parts of the image that are not part of the active LCD film (for example LCD margins and any support structures) from the input processing.

After the control unit has finished the process described in connection with Figs. 3 and 4, the control unit preferably provides identification of pointing object regions to an application that controls the content being showed on the LCD screen 102, or more generally to an application that is configured to make use of the pointing object location as a user input. Fig. 7 shows an example of an infrared image of a pointing object when partly in contact with a display. Here, the pointing object 112 is a hand and five fingers are in contact with an external surface i.e. touching layer or image layer depending on implementation. The picture has been taken through a diffusive screen with near- infrared range light or radiation.

It should be appreciated that in this document, words comprise, include and contain are each used as open-ended expressions with no intended exclusivity. Moreover, term light here is interchangeable with radiation. While infrared light has in occasions been used, this terming is merely for convenience of explanation the term light is not intended to imply suitability for perception by means of a human eye.

The foregoing description has provided by way of non-limiting examples of particular implementations and embodiments of the invention a full and informative description of the best mode presently contemplated by the inventors for carrying out the invention. It is however clear to a person skilled in the art that the invention is not restricted to details of the embodiments presented above, but that it can be implemented in other embodiments using equivalent means without deviating from the characteristics of the invention.

Furthermore, some of the features of the above-disclosed embodiments of this invention may be used to advantage without the corresponding use of other features. As such, the foregoing description shall be considered as merely illustrative of the principles of the present invention, and not in limitation thereof. Hence, the scope of the invention is only restricted by the appended patent claims.

Claims

1. An interactive display (100) comprising: an image layer (102) configured to provide a user with a perceivable visible image and configured to pass through infrared light; an image capture unit (107) configured to capture an infrared image through the image layer (102); a recognizing unit (111 ) configured to detect from the infrared image contact areas where a pointing object (112) is located at the image layer (102) and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object (112) and a central region surrounded by the peripheral region; wherein the recognizing unit (111 ) is configured to detect the contact areas under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing object (112) in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions of the shadow.

2. An interactive display (100) according to claim 1 , further comprising a diffusion layer (103) adjacent to the image layer (102) between the image layer (102) and the image capture unit (107).

3. An interactive display (100) according to claim 1 or 2, further comprising an infrared light reflection inhibitor (109) inside the interactive display, configured to inhibit infrared light reflection from within the interactive display to the image capture unit (107).

4. An interactive display (100) according to any one of the preceding claims, wherein the recognizing unit (111 ) is configured to test whether a particular pixel of the infrared image belongs to a shadow of a pointing object by performing a process comprising: detecting (303) whether the luminosity of the pixel is below a given first threshold luminosity.

5. An interactive display (100) according to any one of the preceding claims, wherein the recognizing unit (111 ) is configured to test whether a particular pixel of the infrared image belongs to a shadow of a pointing object by performing a process comprising: determining (305) a second threshold luminosity based on a the luminosity of the particular pixel and a background luminosity in the infrared image around the particular pixel.

6. An interactive display (100) according to any one of the preceding claims, wherein the recognizing unit (111 ) is configured to test (308) whether a particular pixel of the infrared image belongs to a shadow of a pointing object (112) by performing a process comprising: testing rays (308) around the selected pixel in the infrared image against a given second threshold luminosity by calculating lengths of shadows in different directions through the particular pixel and verifying (309) whether at least two of said calculated lengths are within a given threshold length for an acceptable candidate of a shadow of the pointing object (112).

7. An interactive display (100) according to any one of the preceding claims, wherein the recognizing unit (111 ) is configured to test (308) whether a particular pixel of the infrared image belongs to a shadow of a pointing object (112) by performing a process comprising: calculating dimensions (310) of a shadow surrounding the particular pixel, merging (401 ) potential finger tip pixels around the particular pixel and checking (402,403) whether the merged area has an area not exceeding a threshold area size.

8. An interactive display (100) according to claim 7, wherein the recognizing unit (111 ) is configured to test (308) whether a particular pixel of the infrared image belongs to a shadow of a pointing object (112) by performing a process comprising: determining a maximum allowable luminosity (404) based on the size of the merged area, determining a center luminosity (405) of the merged area and checking (406) whether the center luminosity is within the determined maximum allowable luminosity and if yes, determining (407) that the particular pixel belongs to a pointing object.

9. An interactive display (100) according to any one of the preceding claims, wherein the pointing object (112) is a finger tip.

10. A method in an interactive display, comprising: providing a user with a perceivable visible image on an image layer configured to and configured to pass through infrared light; capturing an infrared image through the image layer; detecting from the infrared image contact areas where a pointing object is located at the image layer and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object and a central region surrounded by the peripheral region; wherein the detecting of the contact areas is performed under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions of the shadow.

11. A method according to claim 10, further comprising testing whether a particular pixel of the infrared image belongs to a shadow of a pointing object by performing a process comprising detecting (303) whether the luminosity of the pixel is below a given first threshold luminosity.

12. A method according to claim 11 , wherein said process further comprises determining (305) a second threshold luminosity based on a the luminosity of the particular pixel and a background luminosity in the infrared image around the particular pixel.

13. A method according to claim 11 or 12, wherein said process further comprises testing rays (308) around the selected pixel in the infrared image against a given second threshold luminosity by calculating lengths of shadows in different directions through the particular pixel and verifying (309) whether at least two of said calculated lengths are within a given threshold length for an acceptable candidate of a shadow of the pointing object (112).

14. A method according to any one of claims 1 1 to 13, wherein said process further comprises calculating dimensions (310) of a shadow surrounding the particular pixel, merging (401 ) potential finger tip pixels around the particular pixel and checking whether the merged area has an area not exceeding a threshold area size.

15. A method according to any one of claims 1 1 to 14, wherein said process further comprises determining a maximum allowable luminosity (404) based on the size of the merged area, determining a center luminosity (405) of the merged area and checking (406) whether the center luminosity is within the determined maximum allowable luminosity and if yes, determining (407) that the particular pixel belongs to a pointing object.

16. A computer readable memory medium embodied with a computer program for controlling a interactive display, the computer program comprising: computer executable program code configured to cause the interactive display when executed to provide a user with a perceivable visible image on an image layer configured to and configured to pass through infrared light; computer executable program code configured to cause the interactive display when executed to capture an infrared image through the image layer; computer executable program code configured to cause the interactive display when executed to detect from the infrared image contact areas where a pointing object is located at the image layer and casts in the infrared image a corresponding shadow, the shadow comprising a peripheral region corresponding to the shape of the pointing object and a central region surrounded by the peripheral region, and the detecting of the contact areas being configured to be operable under daylight, or corresponding strong and coherent ambient infrared illumination, from at least two of the following characteristics present in the shadow of the pointing in the infrared image: darkness of the shadow; gradient of the darkness of the shadow at the edge region; sharpness of the edge region; and dimensions of the shadow.