WO2016180914A1 - Home-automation system - Google Patents

Home-automation system Download PDF

Info

Publication number
WO2016180914A1
WO2016180914A1 PCT/EP2016/060639 EP2016060639W WO2016180914A1 WO 2016180914 A1 WO2016180914 A1 WO 2016180914A1 EP 2016060639 W EP2016060639 W EP 2016060639W WO 2016180914 A1 WO2016180914 A1 WO 2016180914A1
Authority
WO
WIPO (PCT)
Prior art keywords
viewer
content
behaviour
media player
media
Prior art date
Application number
PCT/EP2016/060639
Other languages
French (fr)
Inventor
Laurent MOLAC
David Servignat
Guillaume REILLON
Philippe Stransky-Heilkron
Original Assignee
Nagravision S.A.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nagravision S.A. filed Critical Nagravision S.A.
Publication of WO2016180914A1 publication Critical patent/WO2016180914A1/en

Links

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42201—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/4104—Peripherals receiving signals from specially adapted client devices
    • H04N21/4131—Peripherals receiving signals from specially adapted client devices home appliance, e.g. lighting, air conditioning system, metering devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/41—Structure of client; Structure of client peripherals
    • H04N21/422—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS]
    • H04N21/42202—Input-only peripherals, i.e. input devices connected to specially adapted client devices, e.g. global positioning system [GPS] environmental sensors, e.g. for detecting temperature, luminosity, pressure, earthquakes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
    • H04N21/4331—Caching operations, e.g. of an advertisement for later insertion during playback
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/433—Content storage operation, e.g. storage operation in response to a pause request, caching operations
    • H04N21/4333—Processing operations in response to a pause request
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/436—Interfacing a local distribution network, e.g. communicating with another STB or one or more peripheral devices inside the home
    • H04N21/43615—Interfacing a Home Network, e.g. for connecting the client to a plurality of peripherals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
    • H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
    • H04N21/43—Processing of content or additional data, e.g. demultiplexing additional data from a digital video stream; Elementary client operations, e.g. monitoring of home network or synchronising decoder's clock; Client middleware
    • H04N21/442—Monitoring of processes or resources, e.g. detecting the failure of a recording device, monitoring the downstream bandwidth, the number of times a movie has been viewed, the storage space available from the internal hard disk
    • H04N21/44213—Monitoring of end-user related data
    • H04N21/44218—Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program

Definitions

  • the present disclosure relates to the domain of home automation and covers topics including the use of sensors, usually those used in the field of home security, such as presence sensors or face recognition sensors and the like, especially when such sensors are used in consumer devices such as televisions, set-top boxes, tablet computers and the like.
  • Home automation systems exist which allow users to visualize and control a number of electronic devices and pieces of electronic equipment in their homes. Such systems may be used to perform centralized control of lighting, HVAC systems (heating, ventilation and air conditioning systems), various household appliances, blinds, multi-room-audio, home cinema, security locks on gates or doors and so on, to provide improved convenience, comfort, energy efficiency and/or security.
  • HVAC systems heating, ventilation and air conditioning systems
  • various household appliances blinds, multi-room-audio, home cinema, security locks on gates or doors and so on, to provide improved convenience, comfort, energy efficiency and/or security.
  • Sensors may use sensors. Sensors of various types are known and are used in many different situations. Sensors include movement detectors for security systems, face recognition for cameras and facial expression detectors and emotion detectors, which are sometimes used in computer equipment for marketing purposes. Sensors provide a means for inputting data to a computing system, the data to be input being based on some activity or phenomenon of interest, which is detected by the sensor. The activity or phenomenon of interest usually occurs outside of the computing system and data related to that activity or phenomenon has to be entered into the computing system to affect the behaviour of the computing system.
  • a sensor is a particular type of transducer.
  • a transducer transforms one kind of energy to another.
  • a sensor is a transducer whose purpose is to sense (that is, to detect) some characteristic of its environment or within its detectable range.
  • a sensor is used to detect a parameter in one form and report it in another form of energy, often an electrical signal.
  • a sensor may therefore be used to measure a characteristic or a change in characteristic of its local environment and to convert the measurement to a parameter.
  • the parameter may be an electrical parameter such as a voltage or a current. The particular value of the voltage or current may be of significance in an analogue environment.
  • the parameter in a digital environment the parameter may be taken to represent a state of 1 or 0, whereby a parameter having a value of 5V for example may represent a one and a parameter having a value near 0V may represent a zero.
  • the parameter in the digital environment, the parameter may easily be expressed as a group of digital values.
  • a parameter may correspond to a computer instruction or group of instructions or a value (group of bits) from a computer memory or a pointer to a location in a computer memory.
  • sensors include a temperature sensor, configured to detect the temperature of its environment.
  • the same temperature sensor can detect a change of temperature over time. Depending on the characteristics of the temperature sensor, it may be capable of following the change in temperature of its environment when the temperature changes slowly or more quickly.
  • An accelerometer is another example of a sensor which can detect changes in speed or changes in either magnitude or direction of velocity or both.
  • a visual sensor may detect the presence of a given object. It may be configured to detect displacement of objects and so changes in position of the objects.
  • One obstacle to the wide acceptance of automation systems in general is the perceived level of intrusion associated with the equipment that is needed in such automation systems, especially the various sensors that provide the information needed for the systems. In some situations a user may have a negative impression of such equipment, leading to a certain reticence as to its acceptance. Another obstacle is that the equipment is usually associated with the prevention of the occurrence of hostile events, or at least associated in some way with events which may be perceived by users as being hostile events. For some users therefore, such equipment may have negative connotations, albeit on a subconscious level, thereby further tainting such equipment with a less-than enthusiastic reception among those users.
  • One such system includes an interactive television set for receiving and displaying a television signal.
  • Interactive television sets allow viewers to do more than just view broadcast content, and as such, they are usually configured to receive inputs from other external sources such as the Internet or a telephone line. Interruptions can therefore be received from these other external sources while the viewer is viewing content.
  • the system In order to prevent the viewer from missing part of the content while he attends to the interruption (e.g. answering a video call), the system begins to buffer the content when the interruption is received and continues buffering until the interruption has been dealt with (e.g. end of the video call). Such systems then generally play the content from the buffer and the viewer can therefore catch up on the content that he would otherwise have missed due to the interruption.
  • the interactive TV set is the gateway for reception of both the content and the source of the interruption and so it is easy for the system to handle the simple phone call (or video call) interruption by buffering the content, because interactive TV sets are already designed to be able to handle interruptions coming from a well-defined, pre-determined source.
  • benefits are to be gained from such networking of sensors in the following ways, for example: reduced hostility because consumers are becoming used to the presence of sensors in the home environment;
  • home automation may be experienced by all classes of consumers, not only those that are wealthy enough to purchase dedicated home automation equipment.
  • a consumer's media playing equipment is able to predict, based on one or more environmental conditions detected by at least one sensor associated with the equipment, which part of a particular media content the consumer would be likely to want to review, thereby allowing for facilitated reviewing of the content even if the consumer had not been aware at the time that he might want to review that part or those parts.
  • Media playing equipment configured to perform such tasks may include one or more sensors to detect likely interruption indicators, an analyser to determine the likelihood of an interruption indicator actually corresponding to an interruption, a time-shift buffer to store content when an interruption is detected, a bookmark generator for generating bookmarks relative to positions in the content likely to be selected for review and a memory for storing the bookmarks.
  • a device for playing media content within a first local environment the media content comprising timing information
  • the device comprising:
  • a bookmark inserter for inserting bookmarks in a buffer
  • bookmark buffer for storing one or more bookmarks, each bookmark having a time attribute
  • a local processor configured at least to receive signals from the interface and to control: the bookmark inserter; the bookmark buffer; and the time-shift buffer;
  • the interface is configured to:
  • the local processor is configured to perform at least a first action comprising: correlating the received input with at least one first behaviour; comparing a first behaviour against a set of at least one first predetermined behaviours in order to determine whether at least one from the set of at least one first predetermined behaviours was detected;
  • controlling the bookmark inserter to store a bookmark corresponding to a time, according to the media content timing information, where the first predetermined behaviour was detected;
  • a local network system comprising
  • the first device comprising a first sensor and further comprising a first network interface module for connecting to the local network;
  • a central processor operably connected to the local network
  • the central processor is configured to receive the signals from the first sensor via the local network, the central processor being configured to perform at least part of the first action instead of the local processor.
  • FIG. 1 illustrating a situation in which a method according to an embodiment of the present invention may be implemented
  • FIG. 2 showing an example of a process according to another embodiment of the present invention.
  • FIG. 3 showing a system in which an embodiment of the present invention may be deployed.
  • Sensors are very widely used in the domain of security and surveillance.
  • the types of sensors used in this domain are sometimes video-based and include systems which use cameras, motion detectors, facial recognition systems and so on.
  • Such sensors can sometimes appear to the general public as being intrusive, depending on how, and for what reasons, they are deployed. Nonetheless, people are getting more and more used to the presence of such sensors almost everywhere in everyday life and even welcome their presence in the home environment where such sensors may be used to provide useful services such as switching lights on and off or as part of an alarm system.
  • the proliferation of the use of sensors within the home is therefore becoming more and more widely accepted.
  • sensors can be deployed in standard household appliances such as a set-top box (an STB), fridge, coffee-machine or a television set (TV) for example. These appliances may be connectable to a network such as the Internet for example. When such connectivity is possible on a household appliance or any other device, it may be referred to as a network-enabled device.
  • a network-enabled device Various ways of integrating sensors into network-enabled devices are possible, and these may be used in embodiments of the present invention, such as: on the front of the appliance, facing the living room;
  • an extender like a small cable, so that it can be ideally placed facing the living room or any place of interest;
  • the system described herein therefore has at least one household device or appliance, and relies on the fact that several of these devices can be linked together via a home network.
  • the device is then equipped with a sensor, which then takes advantage of the fact that the devices to which they are attached are connected to a home network.
  • a central computer within the home network (usually in the gateway device for example) can then be fed with information about observations which are made by one or more of the sensors since the sensors are connected to the network via their respective home appliance (network-enabled devices).
  • the sensor is connected to its household device via a first interconnection having a first protocol, while the household device is connected to the home network via a second interface, having a second protocol, usually different from the first.
  • Sensors which can find use in systems in which an embodiment of the present invention may be deployed do not necessarily have to be only integrated on existing consumer devices or communications devices such as mobile phones or tablet computers and so on; instead, stand-alone sensors may also be used according to some embodiments.
  • Sensors of different types may be used depending on what event is to be detected or to enable detection under various different conditions or to provide various levels of sensitivity (person moving into the room vs. person sitting on the sofa and barely moving). These sensors are configured to detect passive user's behaviors i.e. the user does not need to carry a particular device (such as a remote control or a smartphone) to interact with the sensors.
  • a particular device such as a remote control or a smartphone
  • Embodiments of the present invention can make use of various different types of sensors which may be integrated with domestic consumer devices, especially those which are used for playing digital media content (e.g. televisions, set-top boxes, tablet computers, desktop computers, media players etc).
  • digital media content e.g. televisions, set-top boxes, tablet computers, desktop computers, media players etc.
  • advantage is taken from the fact that such domestic consumer devices are present in many homes to the extent that multiple such devices are present in each home, sometimes with one or more devices per room.
  • a network such as a home network or any other local network for example.
  • a network of sensors is created. This can allow for a person's movement throughout the home or local premises to be tracked and analysed in order to estimate or predict a probable behaviour of that person, thereby allowing for different actions to be taken as a consequence, such as turning off different appliances or turning on or off different lights in different rooms.
  • independent sensors i.e.
  • sensors other than those associated with media playing equipment can be added to the network.
  • Mobile devices and their associated sensors may also be included in a network in which an embodiment of the present invention is deployed.
  • portable telephones usually have sensors whose functionality is compatible with the goals to be achieved by the system, in which case they may also be included in the network described.
  • sensors may be added to other equipment and included in the network, thereby improving the user's experience.
  • independent sensors can be added to the network, which may be considered either by using sensors already available in mobile devices, or adding stand-alone sensors, thereby improving the user experience.
  • the idea relies on the fact that widely used equipment, like TV and STB, is present in most homes, and sometimes in multiple occurrences.
  • Implementing the sensors in the STB and/or TV provides a good vector for educating consumers on the benefits of these sensors.
  • the sensors can also cooperate with each other and be used to detect dynamic behaviour, like a person moving from one room to another room for example.
  • Fig. 1 illustrates an example situation where a method according to an embodiment of the present invention may find use.
  • a first media playing equipment (TV1 ) in a first room (ROOM1 ) is equipped with a sensor (SENS1 ) and a second media playing equipment (TV2) in a second room (ROOM2) is equipped with another sensor (SENS2).
  • the two pieces of media playing equipment are interconnected via a gateway device (GWAY) to form part of a network (COM).
  • GWAY gateway device
  • a viewer (VWR) who is initially present in the first room (ROOM1 ) performing a first activity (ACT1 ) while consuming media content presented on the first media playing equipment (TV1 ), decides to move to the second room (ROOM2) to perform another activity (ACT2).
  • the networked sensors detect this movement and so the system is thus able to present the viewer (VWR) to continue to view the media content using the second media playing equipment (TV2) while performing the new activity (ACT2) in the second room (ROOM2).
  • VWR viewer
  • ACT2 new activity
  • a plurality of networked sensors was shown to detect the viewer moving from one room to another.
  • a single sensor may be used to detect a behaviour and initiate an appropriate action, without necessarily involving inputs from other sensors in the network.
  • a behaviour may be matched with a particular event related to the content being played, like the viewer walking out of the living room when advertising content starts playing.
  • An appropriate action to take when the viewer leaves the room might be to stop recording the content, so that the recorded version of the content has no advertisements.
  • the system may be programmed to take appropriate action depending on the user's requirements. For example, if the content does not include advertisements, then the fact that the viewer leaves the room might cause the system to continue recording the content so that the viewer can see the parts he missed at a later time.
  • the programming of the system may take account of different contexts within the home. For example, different states may be defined, such as "owner present” or "owner absent”. In this way, depending on the context, a signal received from a detector will be interpreted in different ways.
  • a person enters the living room and the TV automatically switches on to a predetermined channel;
  • the home is in an "owner absent” state; a person enters the living room thereby triggering an alarm condition since no movement should be detected when the home is in "owner absent” state;
  • the TV in the bedroom switches on automatically and tunes to the same channel as in the living room; after a few minutes of no movement detected in the living room, the TV in the living room switches off automatically;
  • all TVs switch on automatically, possibly with IP cameras; all TVs display a pre-recorded message warning the intruder that the police have been alerted and the IP cameras stream the picture of the intruder on all TVs;
  • a currently-watched TV program is interrupted by advertising; the viewer leaves the room and comes back when the program resumes; the timing of this event can be matched with the program schedule;
  • the viewer moves to the other side of the room and the volume control (a control parameter) is automatically increased to make sure the viewer can still hear; the sensor detects temperature above a predetermined normal value; an alarm is triggered and a message is displayed on all TVs indicating in which room the temperature anomaly is detected.
  • the volume control a control parameter
  • Any of the actions described above, or such actions as switching a TV on or off, triggering an alarm condition, tuning a TV to a given channel, actuating a recording camera, displaying a message or specific video content and so on, may be achieved by actuators configured to perform the operations when signaled by the processor.
  • An actuator can be a switch or a selector for example.
  • a system in which an embodiment of the present invention is deployed may be configured to react in an appropriate manner to any from a wide range of different scenarios.
  • such sensors may be paired with one or more servers or appliances adapted to perform as servers in a home network.
  • a sensor may be configured to behave as a DLNA client connecting to a DLNA server.
  • a sensor or sensor system within a particular piece of media equipment may be used to detect what a viewer is doing during the time that he watches a video for example, and to decide, depending on the viewer's behaviour, which parts of the video appear to be of particular significance to the viewer. These parts may then be considered to be parts to which the viewer is likely to want to return.
  • the system then automatically places bookmarks at these places, thereby allowing the viewer to conveniently return to positions of significant interest to him.
  • particular significance it may mean either parts which stimulate a raised level of interest in the viewer or on the other hand parts which the viewer is likely to have either missed or been distracted from, due to some external interruption like a telephone ringing, the doorbell ringing or having to temporarily move to another room to attend to another task such as making coffee.
  • the bookmarking it is not always apparent to the viewer at the time of (first) viewing that such parts will be of particular significance and so one requirement of this aspect of the invention is that the bookmarking be automated. In other words the system has to be able to reliably predict what may be of particular significance, even if the viewer does not himself recognise at the time that it may be of particular significance.
  • This embodiment may be described as an automated bookmark inserter for automatically inserting navigation reference points within digital media content while the content is being consumed.
  • a viewer While watching a piece of video content, a viewer may be distracted by an event (e.g. a door bell ringing, a telephone call, falling asleep etc.) or certain points during the playback of the content may provide particular interest to the viewer to an extent where he may feel encouraged at some later time to review that point or those points. Examples of where a viewer might be motivated to further review a part of a content are when that point presents a feature having heightened significance, such as a goal being scored during a football match, or a particularly funny joke during a comedy sketch.
  • a particular point may become a candidate for possible review merely because the viewer gets distracted at that point and misses it. Again, this becomes a point that the viewer cannot know that he might want to review in future in order to try to understand what he has missed.
  • automated bookmarking of a given content which is being watched, based on certain detected events is achieved when a sensor provides information to a processor allowing the processor to decide that the event has particular significance.
  • the processor signals to a tag inserter to place a tag (bookmark) in a time-shift buffer relative to the thus-identified event.
  • a system thus configured may thereby provide for detection of any from the following events, for example: a third party (person) appearing in the vicinity of a TV while a first person is watching, may distract the first person such that a replay of the content from or near that point might be welcomed;
  • a viewer leaves the vicinity of the TV momentarily to attend to something else (e.g. make coffee); a phone rings or a door bell rings while the viewer watches the TV (i.e. with a microphone as a sensor, with digital signal processing to detect when the monitored sound corresponds to the phone or doorbell ringing);
  • something else e.g. make coffee
  • a phone rings or a door bell rings while the viewer watches the TV (i.e. with a microphone as a sensor, with digital signal processing to detect when the monitored sound corresponds to the phone or doorbell ringing);
  • a viewer loses concentration and begins to fall asleep (closing or closed eyes for more than a predetermined threshold time are detectable by sensors); or a viewer shouts, laughs or makes a particular expression with his face, any of which are detectable using emotion detection techniques (i.e. with a microphone as a sensor, with digital signal processing to estimate the significance of certain parts of the recording, or an emotion detector).
  • emotion detection techniques i.e. with a microphone as a sensor, with digital signal processing to estimate the significance of certain parts of the recording, or an emotion detector.
  • the system can create a tag that the user may or may not decide to use later on to bring him back to a particular point in the content.
  • any of the events described above are detectable using a camera of an integrated digital television (iDTV) or a camera connected to a set-top box (STB).
  • iDTV integrated digital television
  • STB set-top box
  • An automated bookmark system therefore comprises a media player such as a TV, set-top box (STB), telephone (smart-phone) or computer device having at least one sensor.
  • the system comprises a time-shift buffer to receive and play out the content received by the media player.
  • a time-shift buffer is a storage medium (usually a temporary storage medium) which is operable to record content as it is being played, thereby enabling the viewer to pause the playing of the content, while continuing to record, and later to resume playing of the content since, upon resumption of play, the content from one end of the time-shift buffer is viewed while the other end continues to be filled with the original (non time-shifted or live) content (i.e. recorded).
  • the sensor can be of a type which detects movement to detect when someone arrives in front of the media player or it could be a camera-based sensor which can detect facial expressions of the viewer or can analyse the viewer's emotion.
  • the sensor could also be audio- based, thus an emotion could be detected when the viewer laughs for example.
  • the sensor could be configured to detect a combination of audio and visual phenomenon.
  • a doorbell ringing could be detected by an audio sensor.
  • Other detectors could be included to detect specific events. For example instead of using an audio detector to detect a door-bell ringing, the sensor could directly (electronically) detect when someone depresses the door-bell button.
  • a movement detector can be used to detect when a first viewer arrives at or leaves a predetermined perimeter surrounding the media player with which the sensor is associated, or when a further viewer appears within the perimeter, thereby suggesting a point where the first viewer is likely to become distracted. By placing a mark in the content at this position, the viewer can easily jump back to that position if it does turn out that he was distracted and decides to review the content. Facial recognition detectors can detect when the viewer falls asleep or smiles or laughs.
  • the media player is configured to be able to include bookmarks in the content and to jump to particular bookmarks of the viewer's choice.
  • a personal video recorder is a known media player which incorporates book-marking features such as this, such technology being simply adaptable for any media player.
  • the media player may be configured to receive an external sensor and so has an interface for receiving input from the external sensor.
  • the senor may be of any of the types known in the art as long as it can detect a characteristic or a change in a characteristic within its local environment or detection field.
  • a processor of the media player receives input from the sensor. The input indicates the characteristic or change in characteristic of the sensor's environment. The input is fed to a processor of the media player. The processor correlates the input with a behaviour. In other words it translates the input to a physical parameter or group of physical parameters indicative of a behaviour.
  • a sensor observing a person whose eyes are closing would send corresponding signals to the processor, which would analyse the particular signal or set of signals as being a person closing his eyes.
  • the processor compares this behaviour with a set of predetermined behaviours, i.e. behaviours of particular interest. If a person falling asleep happens to be one of the behaviours of interest, then the processor will signal this.
  • the processor then can instruct the bookmark inserter to store a bookmark corresponding to the time that the behaviour was observed. Bookmarks have timing attributes.
  • the media content usually has some kind of timing reference within it.
  • the processor initiates the storage of the content, or at least part of it, in the time-shift buffer at a time starting from when the falling asleep (closing eyes) was detected.
  • an interface preferably a graphical interface, the media player can then give the person an opportunity to review the part of the content he missed.
  • a media player in which an embodiment of the present invention is deployed may be included in any of the local network configurations described above.
  • the time-shift buffer need not be activated all the time and would be activated based on an input from the sensor.
  • the time- shift buffer can be configured to only begin recording whenever a behaviour of interest is detected by the sensor.
  • the media player simply displays the received content, while when the sensor detects a behaviour deemed to be of interest, it begins to buffer the received content using the time-shift buffer.
  • the time-shift buffer functions all of the time and, whenever the sensor or sensors detect an event which is analysed and judged to be of particular significance, a mark is included in the time-shift buffer at the time when the event was detected.
  • this mark becomes a bookmark, albeit automatically generated, to which the viewer may jump back using the controls of the media player.
  • an integrated digital television (iDTV) or a set-top box having a camera and adapted to detect movement, recognise facial expressions and/or emotions and having a processor for analysing what is being detected could be configured to include a mark for each event considered to be of significance (i.e. exceeding a pre-determined significance threshold) and to provide the viewer with an interface for easily accessing the bookmarks from the time-shift buffer as desired.
  • Fig. 2 illustrates some of the steps which could take place within a system adapted to perform a process according to an embodiment of the present invention.
  • a media player device receives content (CT) and passes the content through a time-shift buffer (TSB).
  • TTB time-shift buffer
  • the media player is connected to a sensor and has a processor for analysing what the sensor detects. From the processing of the sensor data, the media player estimates that at given time (t-x) the viewer exhibits a behaviour (BHV1 ) which is of particular significance and therefore creates a bookmark in the content in the time-shift buffer at time t-x.
  • BHV1 behaviour
  • the system estimates that the exhibited a further behaviour (BHV2) of particular significance and therefore creates a bookmark at time t-y.
  • BHV2 further behaviour
  • the viewer At a further given time (t-z) the viewer is estimated to have exhibited yet another behaviour (BHV3) having particular significance and so places a bookmark at time t-z.
  • the viewer can now easily perform trick-play commands (e.g. rewind) to quickly review the content at the times indicated by any of the thus automatically inserted bookmarks (t-x, t-y, t-z).
  • Fig. 3 illustrates a system in which an embodiment of the present invention may be deployed.
  • the system comprises a media player (STB) for receiving and playing content (CT).
  • the media player (STB) has an integrated camera (SNS) which is configured to recognise emotions of a viewer of the media player (STB).
  • SNS integrated camera
  • the media player (STB) further comprises a time-shift buffer through which the received content passes.
  • the time-shift buffer may be able to record from a few minutes of content up to a number of hours of content depending on its chosen size.
  • the sensor detects that the viewer is beginning to fall asleep.
  • the sensor signals to a mark insertion module (MKR) within the media player (STB) and the mark insertion module (MKR) inserts a mark into the content at the position in the content indicated by the sensor.
  • MKR mark insertion module
  • Various methods may be used for placing a mark, for example the marker may manipulate some of the metadata within the transport stream in a predetermined manner or it may manipulate one or more bits within the header of one or more packets of content within the transport stream.
  • Bookmarks may be stored in a bookmark memory along with corresponding data such as the corresponding position in the content and data relative to a level of significance a particular position appears to represent to the viewer. This allows for the bookmark positions to be presented in an order of ranking to help the viewer decide whether or not he wants to review that part of the content.
  • a user interface can be suitably programmed to present the bookmarks to the viewer in a way which makes the most sense.
  • a processor (not shown), preferably a programmable processor, can be used to analyse the signals from the sensor or sensors to determine whether or not a perceived behaviour corresponds to one having particular significance in the sense of the invention.
  • Any of the known means can be used to help in this decision such as performing a comparison with reference behaviours already stored in the system or by performing statistical analyses of all of the behaviours detected throughout a viewing session or over a number of viewing sessions for example.
  • the processor may also be used to create a suitable user interface for presenting the bookmarks to the viewer.
  • Processors may be used in any of the embodiments of the local network systems described herein.
  • the sensors themselves may have processing capability and therefore can be described as comprising a processor and the system itself may have a central processor for analysing signals or combinations of signals received from one or more sensors either directly or via one or more of the network-enabled devices connected in the system.
  • the analysis of the signals generated by the sensors, or part of such analysis may be performed by the processor in the sensor itself.
  • a remaining part of the analysis may be performed by the system's central processor. In other embodiments it is the central processor which performs the analysis.
  • Embodiments of the system described provide for the action to be taken, depending on the results of one or more of the analyses, to be carried out or at least initiated by the central processor.
  • An example of the central processor initiating the action is sending instructions to one or more of the appliances or devices on the network to perform a particular task.
  • any of the embodiments above may be deployed in a local network system within a local premises.
  • a home network with the local premises being the whole household in some cases (e.g. detect moving from room to room) or a single room in other cases (e.g. detect an intruder) or even a part of a room (e.g. detect viewer falling asleep).

Abstract

The present invention relates to a system and a device for automatically detecting certain behaviours of an occupant or occupants of a particular designated space, usually within the home environment, and to perform action depending on the detected behaviour. The behaviours may be detected using sensors associated with standard electronic equipment available in most homes, like media content receivers for example. According to one embodiment of the invention a set-top box comprises a sensor to monitor the viewer and to analyse the viewer's behaviour in order to estimate when the viewer loses interest in the content. The set-top box further comprises a time shift buffer for buffering the received content and a mark inserter module which is configured to place a mark in the content within the time-shift buffer at places where the viewer is deemed to have lost concentration, thereby allowing for the viewer to conveniently replay the content from where he lost interest. According to another embodiment, there is provided a system comprising a plurality of sensors associated with a plurality of appliances or electronic devices within the household. The system is configured to take appropriate actions depending on how the information received from the sensor or network of sensors is interpreted.

Description

HOME-AUTOMATION SYSTEM
TECHNICAL DOMAIN
The present disclosure relates to the domain of home automation and covers topics including the use of sensors, usually those used in the field of home security, such as presence sensors or face recognition sensors and the like, especially when such sensors are used in consumer devices such as televisions, set-top boxes, tablet computers and the like.
STATE OF THE ART
Home automation systems exist which allow users to visualize and control a number of electronic devices and pieces of electronic equipment in their homes. Such systems may be used to perform centralized control of lighting, HVAC systems (heating, ventilation and air conditioning systems), various household appliances, blinds, multi-room-audio, home cinema, security locks on gates or doors and so on, to provide improved convenience, comfort, energy efficiency and/or security.
The popularity of home automation has been increasing greatly in recent years due to much higher affordability and simplicity through connectivity features provided by smart phones and tablet, or otherwise mobile, computers. Possibilities now exist for web-based visualization - from within the house via a web browser using a personal computer or from anywhere in the world using any Internet-enabled device. Interconnectivity of devices and appliances, facilitated by Internet-enabled devices, is sometimes referred to as the "Internet of things".
Home automation systems may use sensors. Sensors of various types are known and are used in many different situations. Sensors include movement detectors for security systems, face recognition for cameras and facial expression detectors and emotion detectors, which are sometimes used in computer equipment for marketing purposes. Sensors provide a means for inputting data to a computing system, the data to be input being based on some activity or phenomenon of interest, which is detected by the sensor. The activity or phenomenon of interest usually occurs outside of the computing system and data related to that activity or phenomenon has to be entered into the computing system to affect the behaviour of the computing system.
A sensor is a particular type of transducer. A transducer transforms one kind of energy to another. A sensor is a transducer whose purpose is to sense (that is, to detect) some characteristic of its environment or within its detectable range. A sensor is used to detect a parameter in one form and report it in another form of energy, often an electrical signal. A sensor may therefore be used to measure a characteristic or a change in characteristic of its local environment and to convert the measurement to a parameter. The parameter may be an electrical parameter such as a voltage or a current. The particular value of the voltage or current may be of significance in an analogue environment. Otherwise, in a digital environment the parameter may be taken to represent a state of 1 or 0, whereby a parameter having a value of 5V for example may represent a one and a parameter having a value near 0V may represent a zero. Furthermore, in the digital environment, the parameter may easily be expressed as a group of digital values. In this case a parameter may correspond to a computer instruction or group of instructions or a value (group of bits) from a computer memory or a pointer to a location in a computer memory.
Examples of sensors include a temperature sensor, configured to detect the temperature of its environment. The same temperature sensor can detect a change of temperature over time. Depending on the characteristics of the temperature sensor, it may be capable of following the change in temperature of its environment when the temperature changes slowly or more quickly. An accelerometer is another example of a sensor which can detect changes in speed or changes in either magnitude or direction of velocity or both. A visual sensor may detect the presence of a given object. It may be configured to detect displacement of objects and so changes in position of the objects.
One obstacle to the wide acceptance of automation systems in general is the perceived level of intrusion associated with the equipment that is needed in such automation systems, especially the various sensors that provide the information needed for the systems. In some situations a user may have a negative impression of such equipment, leading to a certain reticence as to its acceptance. Another obstacle is that the equipment is usually associated with the prevention of the occurrence of hostile events, or at least associated in some way with events which may be perceived by users as being hostile events. For some users therefore, such equipment may have negative connotations, albeit on a subconscious level, thereby further tainting such equipment with a less-than enthusiastic reception among those users.
Various systems exist within the state of the art to allow a viewer who uses a media player for viewing content to be able to catch up on the viewing of parts of the content that he might have missed due to an interruption received during his viewing of the content.
One such system includes an interactive television set for receiving and displaying a television signal. Interactive television sets allow viewers to do more than just view broadcast content, and as such, they are usually configured to receive inputs from other external sources such as the Internet or a telephone line. Interruptions can therefore be received from these other external sources while the viewer is viewing content. In order to prevent the viewer from missing part of the content while he attends to the interruption (e.g. answering a video call), the system begins to buffer the content when the interruption is received and continues buffering until the interruption has been dealt with (e.g. end of the video call). Such systems then generally play the content from the buffer and the viewer can therefore catch up on the content that he would otherwise have missed due to the interruption. In systems such as this, the interactive TV set is the gateway for reception of both the content and the source of the interruption and so it is easy for the system to handle the simple phone call (or video call) interruption by buffering the content, because interactive TV sets are already designed to be able to handle interruptions coming from a well-defined, pre-determined source.
In view of the prior art, there remains a need for a convenient, non-intrusive, non- hostile automated method of recognising various different situations within a household and performing appropriate actions depending on the situation. Furthermore, there remains a need for a convenient automated method of recognising various different types of interruptions while a viewer is watching content and to allow the viewer to conveniently resume viewing after the interruption is over. BRIEF SUMMARY OF THE INVENTION
Nowadays it is not uncommon to find up to three or more televisions within a single household. It is also not uncommon for modern television sets to incorporate some kind of a sensor. According to aspects of the present invention, it is recognised that other leisure-related devices within the home may also incorporate one or more sensors, and that these devices can be networked together. Consequently, this provides a possibility of having a significant number of opportunities to use a number of readily-available interconnected sensors within the home in order to realise different systems in which various embodiments of the present invention may be deployed. Such systems will be non-threatening to users given that the user is already accustomed to having the sensors around and does not have to install a dedicated, and possibly cumbersome or intrusive, system of sensors or the like to perform any of the desired goals.
For example, with an average of 3.7 TVs per household in the United States, there is already a potential for 3.7 sensors per household to be installed without any specific effort and without potentially arousing a householder's suspicion or reticence towards installing hardware in his home which could otherwise be seen to be invasive of his privacy. Furthermore, the fact that the sensors can be interconnected via a network contributes towards building a system in which various embodiments of the present invention may be deployed.
According to a first aspect, benefits are to be gained from such networking of sensors in the following ways, for example: reduced hostility because consumers are becoming used to the presence of sensors in the home environment;
consumers may enjoy a new lifestyle without requiring them to purchase separate devices for their first steps in home automation;
consumers who develop a positive perception through use may wish to further improve the experience by purchasing additional sensors to add to the network; and home automation may be experienced by all classes of consumers, not only those that are wealthy enough to purchase dedicated home automation equipment.
According to another aspect, a consumer's media playing equipment is able to predict, based on one or more environmental conditions detected by at least one sensor associated with the equipment, which part of a particular media content the consumer would be likely to want to review, thereby allowing for facilitated reviewing of the content even if the consumer had not been aware at the time that he might want to review that part or those parts. Media playing equipment configured to perform such tasks may include one or more sensors to detect likely interruption indicators, an analyser to determine the likelihood of an interruption indicator actually corresponding to an interruption, a time-shift buffer to store content when an interruption is detected, a bookmark generator for generating bookmarks relative to positions in the content likely to be selected for review and a memory for storing the bookmarks.
Accordingly, provision is made for a device for playing media content within a first local environment, the media content comprising timing information, the device comprising:
an interface for receiving input from a sensor;
a time-shift buffer;
a bookmark inserter for inserting bookmarks in a buffer;
a bookmark buffer for storing one or more bookmarks, each bookmark having a time attribute; and
a local processor configured at least to receive signals from the interface and to control: the bookmark inserter; the bookmark buffer; and the time-shift buffer;
characterised in that:
the interface is configured to:
receive an input from the sensor, said input indicating a characteristic or a change in a characteristic within the first local environment; and
make the received input available to the local processor; and the local processor is configured to perform at least a first action comprising: correlating the received input with at least one first behaviour; comparing a first behaviour against a set of at least one first predetermined behaviours in order to determine whether at least one from the set of at least one first predetermined behaviours was detected;
controlling the bookmark inserter to store a bookmark corresponding to a time, according to the media content timing information, where the first predetermined behaviour was detected; and
storing at least part of the content in the time-shift buffer starting from when the first predetermined behaviour was detected.
According to another aspect, there is provided a local network system, the system comprising
a first device as described above, the first device comprising a first sensor and further comprising a first network interface module for connecting to the local network; and
a central processor operably connected to the local network;
wherein the central processor is configured to receive the signals from the first sensor via the local network, the central processor being configured to perform at least part of the first action instead of the local processor.
BRIEF DESCRIPTION OF THE DRAWINGS
The present invention will be better understood thanks to the detailed description which follows and the accompanying drawings, which are given as non-limiting examples of embodiments of the invention, namely:
Fig. 1 , illustrating a situation in which a method according to an embodiment of the present invention may be implemented;
Fig. 2, showing an example of a process according to another embodiment of the present invention; and
Fig. 3, showing a system in which an embodiment of the present invention may be deployed.
DETAILED DESCRIPTION
Sensors are very widely used in the domain of security and surveillance. The types of sensors used in this domain are sometimes video-based and include systems which use cameras, motion detectors, facial recognition systems and so on. Such sensors can sometimes appear to the general public as being intrusive, depending on how, and for what reasons, they are deployed. Nonetheless, people are getting more and more used to the presence of such sensors almost everywhere in everyday life and even welcome their presence in the home environment where such sensors may be used to provide useful services such as switching lights on and off or as part of an alarm system. The proliferation of the use of sensors within the home is therefore becoming more and more widely accepted.
Within the home, sensors can be deployed in standard household appliances such as a set-top box (an STB), fridge, coffee-machine or a television set (TV) for example. These appliances may be connectable to a network such as the Internet for example. When such connectivity is possible on a household appliance or any other device, it may be referred to as a network-enabled device. Various ways of integrating sensors into network-enabled devices are possible, and these may be used in embodiments of the present invention, such as: on the front of the appliance, facing the living room;
on an extender, like a small cable, so that it can be ideally placed facing the living room or any place of interest;
on the front, back and/or side of the appliance, so that better detection of particular events can be achieved; or
deployed within a CI+ CAM module or CableCard, to add sensing capability to a legacy device.
The system described herein therefore has at least one household device or appliance, and relies on the fact that several of these devices can be linked together via a home network. The device is then equipped with a sensor, which then takes advantage of the fact that the devices to which they are attached are connected to a home network. A central computer within the home network (usually in the gateway device for example) can then be fed with information about observations which are made by one or more of the sensors since the sensors are connected to the network via their respective home appliance (network-enabled devices). The sensor is connected to its household device via a first interconnection having a first protocol, while the household device is connected to the home network via a second interface, having a second protocol, usually different from the first.
Sensors which can find use in systems in which an embodiment of the present invention may be deployed do not necessarily have to be only integrated on existing consumer devices or communications devices such as mobile phones or tablet computers and so on; instead, stand-alone sensors may also be used according to some embodiments.
Sensors of different types may be used depending on what event is to be detected or to enable detection under various different conditions or to provide various levels of sensitivity (person moving into the room vs. person sitting on the sofa and barely moving). These sensors are configured to detect passive user's behaviors i.e. the user does not need to carry a particular device (such as a remote control or a smartphone) to interact with the sensors.
Embodiments of the present invention can make use of various different types of sensors which may be integrated with domestic consumer devices, especially those which are used for playing digital media content (e.g. televisions, set-top boxes, tablet computers, desktop computers, media players etc).
According to a first aspect, in a system in which an embodiment of the present invention is deployed, advantage is taken from the fact that such domestic consumer devices are present in many homes to the extent that multiple such devices are present in each home, sometimes with one or more devices per room. By incorporating sensors on these devices and interconnecting them via a network such as a home network or any other local network for example, a network of sensors is created. This can allow for a person's movement throughout the home or local premises to be tracked and analysed in order to estimate or predict a probable behaviour of that person, thereby allowing for different actions to be taken as a consequence, such as turning off different appliances or turning on or off different lights in different rooms. According to a variant of this embodiment, independent sensors (i.e. sensors other than those associated with media playing equipment) can be added to the network. Mobile devices and their associated sensors may also be included in a network in which an embodiment of the present invention is deployed. For example, portable telephones usually have sensors whose functionality is compatible with the goals to be achieved by the system, in which case they may also be included in the network described. Similarly, sensors may be added to other equipment and included in the network, thereby improving the user's experience. Of course, independent sensors can be added to the network, which may be considered either by using sensors already available in mobile devices, or adding stand-alone sensors, thereby improving the user experience.
The idea relies on the fact that widely used equipment, like TV and STB, is present in most homes, and sometimes in multiple occurrences. Implementing the sensors in the STB and/or TV provides a good vector for educating consumers on the benefits of these sensors. As TVs and STBs are more and more interconnected, the sensors can also cooperate with each other and be used to detect dynamic behaviour, like a person moving from one room to another room for example.
Fig. 1 illustrates an example situation where a method according to an embodiment of the present invention may find use. A first media playing equipment (TV1 ) in a first room (ROOM1 ) is equipped with a sensor (SENS1 ) and a second media playing equipment (TV2) in a second room (ROOM2) is equipped with another sensor (SENS2). The two pieces of media playing equipment are interconnected via a gateway device (GWAY) to form part of a network (COM). A viewer (VWR), who is initially present in the first room (ROOM1 ) performing a first activity (ACT1 ) while consuming media content presented on the first media playing equipment (TV1 ), decides to move to the second room (ROOM2) to perform another activity (ACT2). As he moves from one room to the other, the networked sensors detect this movement and so the system is thus able to present the viewer (VWR) to continue to view the media content using the second media playing equipment (TV2) while performing the new activity (ACT2) in the second room (ROOM2).
In the above example, a plurality of networked sensors was shown to detect the viewer moving from one room to another. In another example a single sensor may be used to detect a behaviour and initiate an appropriate action, without necessarily involving inputs from other sensors in the network. For example, a behaviour may be matched with a particular event related to the content being played, like the viewer walking out of the living room when advertising content starts playing. An appropriate action to take when the viewer leaves the room might be to stop recording the content, so that the recorded version of the content has no advertisements. The system may be programmed to take appropriate action depending on the user's requirements. For example, if the content does not include advertisements, then the fact that the viewer leaves the room might cause the system to continue recording the content so that the viewer can see the parts he missed at a later time.
The programming of the system may take account of different contexts within the home. For example, different states may be defined, such as "owner present" or "owner absent". In this way, depending on the context, a signal received from a detector will be interpreted in different ways. Here are some example scenarios: a person enters the living room and the TV automatically switches on to a predetermined channel;
according to certain pre-determined criteria, the home is in an "owner absent" state; a person enters the living room thereby triggering an alarm condition since no movement should be detected when the home is in "owner absent" state;
a person exits the living room and enters the bedroom; the TV in the bedroom switches on automatically and tunes to the same channel as in the living room; after a few minutes of no movement detected in the living room, the TV in the living room switches off automatically;
a person leaves the living room and no more movement is detected; the TV switches off automatically;
the home is in an "owner absent" state and movement is detected; all TVs switch on automatically, possibly with IP cameras; all TVs display a pre-recorded message warning the intruder that the police have been alerted and the IP cameras stream the picture of the intruder on all TVs;
a currently-watched TV program is interrupted by advertising; the viewer leaves the room and comes back when the program resumes; the timing of this event can be matched with the program schedule;
the viewer moves to the other side of the room and the volume control (a control parameter) is automatically increased to make sure the viewer can still hear; the sensor detects temperature above a predetermined normal value; an alarm is triggered and a message is displayed on all TVs indicating in which room the temperature anomaly is detected.
Any of the actions described above, or such actions as switching a TV on or off, triggering an alarm condition, tuning a TV to a given channel, actuating a recording camera, displaying a message or specific video content and so on, may be achieved by actuators configured to perform the operations when signaled by the processor. An actuator can be a switch or a selector for example.
Depending on the type of sensors available, e.g. temperature, movement, IP camera, etc., a system in which an embodiment of the present invention is deployed may be configured to react in an appropriate manner to any from a wide range of different scenarios.
According to a particular embodiment in which one or more independent sensors are employed, such sensors may be paired with one or more servers or appliances adapted to perform as servers in a home network. For example, a sensor may be configured to behave as a DLNA client connecting to a DLNA server.
According to further aspect, in a system in which another embodiment of the present invention may be deployed, rather than relying on a whole network of sensors, a sensor or sensor system within a particular piece of media equipment may be used to detect what a viewer is doing during the time that he watches a video for example, and to decide, depending on the viewer's behaviour, which parts of the video appear to be of particular significance to the viewer. These parts may then be considered to be parts to which the viewer is likely to want to return. The system then automatically places bookmarks at these places, thereby allowing the viewer to conveniently return to positions of significant interest to him. By "particular significance" it may mean either parts which stimulate a raised level of interest in the viewer or on the other hand parts which the viewer is likely to have either missed or been distracted from, due to some external interruption like a telephone ringing, the doorbell ringing or having to temporarily move to another room to attend to another task such as making coffee. As such, it is not always apparent to the viewer at the time of (first) viewing that such parts will be of particular significance and so one requirement of this aspect of the invention is that the bookmarking be automated. In other words the system has to be able to reliably predict what may be of particular significance, even if the viewer does not himself recognise at the time that it may be of particular significance.
This embodiment may be described as an automated bookmark inserter for automatically inserting navigation reference points within digital media content while the content is being consumed. While watching a piece of video content, a viewer may be distracted by an event (e.g. a door bell ringing, a telephone call, falling asleep etc.) or certain points during the playback of the content may provide particular interest to the viewer to an extent where he may feel encouraged at some later time to review that point or those points. Examples of where a viewer might be motivated to further review a part of a content are when that point presents a feature having heightened significance, such as a goal being scored during a football match, or a particularly funny joke during a comedy sketch. The viewer usually enjoys the feature when it is played the first time but is sufficiently absorbed by the content not to be consciously aware that he may want to review that point some time later. In other cases, rather than a point in the content providing particular interest or having heightened significance to the viewer, a particular point may become a candidate for possible review merely because the viewer gets distracted at that point and misses it. Again, this becomes a point that the viewer cannot know that he might want to review in future in order to try to understand what he has missed.
According to an embodiment, automated bookmarking of a given content which is being watched, based on certain detected events, is achieved when a sensor provides information to a processor allowing the processor to decide that the event has particular significance. The processor signals to a tag inserter to place a tag (bookmark) in a time-shift buffer relative to the thus-identified event. A system thus configured, according to an embodiment, may thereby provide for detection of any from the following events, for example: a third party (person) appearing in the vicinity of a TV while a first person is watching, may distract the first person such that a replay of the content from or near that point might be welcomed;
a viewer leaves the vicinity of the TV momentarily to attend to something else (e.g. make coffee); a phone rings or a door bell rings while the viewer watches the TV (i.e. with a microphone as a sensor, with digital signal processing to detect when the monitored sound corresponds to the phone or doorbell ringing);
a viewer loses concentration and begins to fall asleep (closing or closed eyes for more than a predetermined threshold time are detectable by sensors); or a viewer shouts, laughs or makes a particular expression with his face, any of which are detectable using emotion detection techniques (i.e. with a microphone as a sensor, with digital signal processing to estimate the significance of certain parts of the recording, or an emotion detector). For any of the events mentioned above, the system can create a tag that the user may or may not decide to use later on to bring him back to a particular point in the content. If the viewer does decide that he wants to review part of the content, then he can rewind easily using trick-play features as he would do using technology already available in most personal video recorders (PVR) by jumping to the parts indicated by the tags or bookmarks. Any of the events described above are detectable using a camera of an integrated digital television (iDTV) or a camera connected to a set-top box (STB).
An automated bookmark system according to an embodiment of the present invention therefore comprises a media player such as a TV, set-top box (STB), telephone (smart-phone) or computer device having at least one sensor. The system comprises a time-shift buffer to receive and play out the content received by the media player. A time-shift buffer is a storage medium (usually a temporary storage medium) which is operable to record content as it is being played, thereby enabling the viewer to pause the playing of the content, while continuing to record, and later to resume playing of the content since, upon resumption of play, the content from one end of the time-shift buffer is viewed while the other end continues to be filled with the original (non time-shifted or live) content (i.e. recorded). The sensor can be of a type which detects movement to detect when someone arrives in front of the media player or it could be a camera-based sensor which can detect facial expressions of the viewer or can analyse the viewer's emotion. The sensor could also be audio- based, thus an emotion could be detected when the viewer laughs for example. According to an embodiment, the sensor could be configured to detect a combination of audio and visual phenomenon. A doorbell ringing could be detected by an audio sensor. Other detectors could be included to detect specific events. For example instead of using an audio detector to detect a door-bell ringing, the sensor could directly (electronically) detect when someone depresses the door-bell button. A movement detector can be used to detect when a first viewer arrives at or leaves a predetermined perimeter surrounding the media player with which the sensor is associated, or when a further viewer appears within the perimeter, thereby suggesting a point where the first viewer is likely to become distracted. By placing a mark in the content at this position, the viewer can easily jump back to that position if it does turn out that he was distracted and decides to review the content. Facial recognition detectors can detect when the viewer falls asleep or smiles or laughs. The media player is configured to be able to include bookmarks in the content and to jump to particular bookmarks of the viewer's choice. A personal video recorder is a known media player which incorporates book-marking features such as this, such technology being simply adaptable for any media player. According to an embodiment, the media player, or other such device, may be configured to receive an external sensor and so has an interface for receiving input from the external sensor. As with the other embodiments, the senor may be of any of the types known in the art as long as it can detect a characteristic or a change in a characteristic within its local environment or detection field. A processor of the media player receives input from the sensor. The input indicates the characteristic or change in characteristic of the sensor's environment. The input is fed to a processor of the media player. The processor correlates the input with a behaviour. In other words it translates the input to a physical parameter or group of physical parameters indicative of a behaviour. For example, a sensor observing a person whose eyes are closing would send corresponding signals to the processor, which would analyse the particular signal or set of signals as being a person closing his eyes. The processor compares this behaviour with a set of predetermined behaviours, i.e. behaviours of particular interest. If a person falling asleep happens to be one of the behaviours of interest, then the processor will signal this. The processor then can instruct the bookmark inserter to store a bookmark corresponding to the time that the behaviour was observed. Bookmarks have timing attributes. The media content usually has some kind of timing reference within it. Finally, the processor initiates the storage of the content, or at least part of it, in the time-shift buffer at a time starting from when the falling asleep (closing eyes) was detected. By using an interface, preferably a graphical interface, the media player can then give the person an opportunity to review the part of the content he missed.
A media player in which an embodiment of the present invention is deployed may be included in any of the local network configurations described above.
According to one embodiment, the time-shift buffer need not be activated all the time and would be activated based on an input from the sensor. In this manner the time- shift buffer can be configured to only begin recording whenever a behaviour of interest is detected by the sensor. Thereby, in a standard mode of operation the media player simply displays the received content, while when the sensor detects a behaviour deemed to be of interest, it begins to buffer the received content using the time-shift buffer. According to other embodiments the time-shift buffer functions all of the time and, whenever the sensor or sensors detect an event which is analysed and judged to be of particular significance, a mark is included in the time-shift buffer at the time when the event was detected. In all embodiments, this mark becomes a bookmark, albeit automatically generated, to which the viewer may jump back using the controls of the media player. For example, an integrated digital television (iDTV) or a set-top box having a camera and adapted to detect movement, recognise facial expressions and/or emotions and having a processor for analysing what is being detected could be configured to include a mark for each event considered to be of significance (i.e. exceeding a pre-determined significance threshold) and to provide the viewer with an interface for easily accessing the bookmarks from the time-shift buffer as desired.
Fig. 2 illustrates some of the steps which could take place within a system adapted to perform a process according to an embodiment of the present invention. A media player device receives content (CT) and passes the content through a time-shift buffer (TSB). The media player is connected to a sensor and has a processor for analysing what the sensor detects. From the processing of the sensor data, the media player estimates that at given time (t-x) the viewer exhibits a behaviour (BHV1 ) which is of particular significance and therefore creates a bookmark in the content in the time-shift buffer at time t-x. At another given time (t-y) the system estimates that the exhibited a further behaviour (BHV2) of particular significance and therefore creates a bookmark at time t-y. At a further given time (t-z) the viewer is estimated to have exhibited yet another behaviour (BHV3) having particular significance and so places a bookmark at time t-z. The viewer can now easily perform trick-play commands (e.g. rewind) to quickly review the content at the times indicated by any of the thus automatically inserted bookmarks (t-x, t-y, t-z).
Fig. 3 illustrates a system in which an embodiment of the present invention may be deployed. The system comprises a media player (STB) for receiving and playing content (CT). The media player (STB) has an integrated camera (SNS) which is configured to recognise emotions of a viewer of the media player (STB). The media player (STB) further comprises a time-shift buffer through which the received content passes. The time-shift buffer may be able to record from a few minutes of content up to a number of hours of content depending on its chosen size. During the playing of received content, the sensor detects that the viewer is beginning to fall asleep. The sensor signals to a mark insertion module (MKR) within the media player (STB) and the mark insertion module (MKR) inserts a mark into the content at the position in the content indicated by the sensor. Various methods may be used for placing a mark, for example the marker may manipulate some of the metadata within the transport stream in a predetermined manner or it may manipulate one or more bits within the header of one or more packets of content within the transport stream. When the viewer wakes up and decides he wants to catch up on what he missed when he fell asleep, he can use his remote control to control the media player (STB) to skip back to a thus-placed bookmark of his choice within the content.
Bookmarks may be stored in a bookmark memory along with corresponding data such as the corresponding position in the content and data relative to a level of significance a particular position appears to represent to the viewer. This allows for the bookmark positions to be presented in an order of ranking to help the viewer decide whether or not he wants to review that part of the content. A user interface can be suitably programmed to present the bookmarks to the viewer in a way which makes the most sense. A processor (not shown), preferably a programmable processor, can be used to analyse the signals from the sensor or sensors to determine whether or not a perceived behaviour corresponds to one having particular significance in the sense of the invention. Any of the known means can be used to help in this decision such as performing a comparison with reference behaviours already stored in the system or by performing statistical analyses of all of the behaviours detected throughout a viewing session or over a number of viewing sessions for example. The processor may also be used to create a suitable user interface for presenting the bookmarks to the viewer.
Processors may be used in any of the embodiments of the local network systems described herein. The sensors themselves may have processing capability and therefore can be described as comprising a processor and the system itself may have a central processor for analysing signals or combinations of signals received from one or more sensors either directly or via one or more of the network-enabled devices connected in the system. According to some embodiments the analysis of the signals generated by the sensors, or part of such analysis, may be performed by the processor in the sensor itself. When only part of the analysis is done in the processor in the sensor itself, a remaining part of the analysis may be performed by the system's central processor. In other embodiments it is the central processor which performs the analysis. Embodiments of the system described provide for the action to be taken, depending on the results of one or more of the analyses, to be carried out or at least initiated by the central processor. An example of the central processor initiating the action is sending instructions to one or more of the appliances or devices on the network to perform a particular task.
Any of the embodiments above may be deployed in a local network system within a local premises. For example, a home network, with the local premises being the whole household in some cases (e.g. detect moving from room to room) or a single room in other cases (e.g. detect an intruder) or even a part of a room (e.g. detect viewer falling asleep).

Claims

1 . A device for playing media content, the media content comprising timing information, the device comprising:
a processor;
a time-shift buffer for providing delayed playback of the media content; and an interface for receiving at least one parameter from a sensor configured to derive said parameter from a measurement of a characteristic or a change in characteristic of an environment of the sensor;
characterised in that the device further comprises a mark insertion module, the device being configured to:
using the processor, correlate the received parameter with a behaviour and compare the behaviour with a set of predetermined behaviours of a viewer of the media content; and
store the playing content in the time-shift buffer and, using the insertion module, insert at least one mark in the stored media content at a time when the behaviour matches at least one of the predetermined behaviours.
2. The device according to claim 1 , further comprising a bookmark buffer, the device being further configured to store at least one bookmark in the bookmark buffer, the bookmark corresponding to the mark inserted in the stored media content.
3. The device according to claim 2, further comprising a programmable user interface configured to present the bookmarks to a viewer for selection, the device further configured to replay the media content from the time-shift buffer from an inserted mark corresponding to a bookmark selected by the viewer thereby replaying the media content which was playing at the time when the behaviour was detected.
4. The device according to claim 3, wherein each of the behaviours from the set of predetermined behaviours are attributed a respective ranking level, the user interface being configured to present the detected behaviours in order of the ranking level of their corresponding predetermined behaviours.
5. A system comprising:
a gateway device for receiving media content from an external network and redistributing the received media content to one or more media players within a local network; and
a media player device according to claim 1 , connected to the gateway device via the local network.
6. The system according to claim 5, further comprising a further media player device according to claim 1 , the further media player device connected to the gateway device via the local network, the gateway device configured to receive the detected behaviours from the media players and to derive a new behaviour of a viewer based on the received behaviours.
7. The system according to claim 6, the gateway device configured to provide one or other of the media player devices with the media content depending on whether the derived new behaviour involves the viewer being present within the vicinity of one or other of the media player devices.
9. The system according to claim 6, the gateway device configured to switch off one or both of the media player devices if the derived behaviour involves the viewer being absent from the vicinity of one or both of the media player devices, respectively.
10. The system according to either of claims 5 or 6, the gateway device being configured to switch on one or both of the media player devices if the behaviour or the behaviour involves the viewer being present in the vicinity of one or both of the media player devices, respectively.
1 1 . The system according to either of claims 5 or 6, one or more of the sensors being a video camera, the gateway device being configured to instruct one or other of the media player devices to record video using the video camera if the received behaviour from the respective media player device involves a viewer being present in the vicinity of the media player device.
PCT/EP2016/060639 2015-05-13 2016-05-12 Home-automation system WO2016180914A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
EP15167693 2015-05-13
EP15167693.9 2015-05-13

Publications (1)

Publication Number Publication Date
WO2016180914A1 true WO2016180914A1 (en) 2016-11-17

Family

ID=53181111

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2016/060639 WO2016180914A1 (en) 2015-05-13 2016-05-12 Home-automation system

Country Status (1)

Country Link
WO (1) WO2016180914A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3661221A1 (en) * 2018-11-30 2020-06-03 Sagemcom Broadband Sas Method for tracking an audiovisual transmission and device enabling implementation of same

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030097659A1 (en) * 2001-11-16 2003-05-22 Goldman Phillip Y. Interrupting the output of media content in response to an event
US20100122277A1 (en) * 2006-07-12 2010-05-13 Koninklijike Phillips Electronics, N.V. device and a method for playing audio-video content
US20110142411A1 (en) * 2008-05-01 2011-06-16 Sony Computer Entertainment Inc. Media reproduction for audio visual entertainment
US8464299B1 (en) * 2004-11-17 2013-06-11 Rockstar Consortium Us Lp Resource conservation for packet television services

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20030097659A1 (en) * 2001-11-16 2003-05-22 Goldman Phillip Y. Interrupting the output of media content in response to an event
US8464299B1 (en) * 2004-11-17 2013-06-11 Rockstar Consortium Us Lp Resource conservation for packet television services
US20100122277A1 (en) * 2006-07-12 2010-05-13 Koninklijike Phillips Electronics, N.V. device and a method for playing audio-video content
US20110142411A1 (en) * 2008-05-01 2011-06-16 Sony Computer Entertainment Inc. Media reproduction for audio visual entertainment

Cited By (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP3661221A1 (en) * 2018-11-30 2020-06-03 Sagemcom Broadband Sas Method for tracking an audiovisual transmission and device enabling implementation of same
FR3089379A1 (en) * 2018-11-30 2020-06-05 Sagemcom Broadband Sas Method for monitoring an audiovisual program and equipment allowing its implementation
CN111263213A (en) * 2018-11-30 2020-06-09 萨基姆宽带简易股份有限公司 Method for following audiovisual transmissions and device capable of implementing said method
US11259077B2 (en) 2018-11-30 2022-02-22 Sagemcom Broadband Sas Method for following an audio-visual transmission and equipment enabling the implementation thereof
CN111263213B (en) * 2018-11-30 2023-08-29 萨基姆宽带简易股份有限公司 Method for following an audiovisual transmission and device capable of implementing the method

Similar Documents

Publication Publication Date Title
US10154295B2 (en) Method and system for analysis of sensory information to estimate audience reaction
US20220114880A1 (en) Securing Property
CN107741698B (en) Linkage control method, device and system for intelligent household appliance and entrance guard equipment
US8689249B2 (en) In-home monitoring method and system having a video camera connected to a set-top box
US20170064412A1 (en) Device-based event detection and notification surfacing
US10540884B1 (en) Systems and methods for operating remote presence security
US8561095B2 (en) Affective television monitoring and control in response to physiological data
US9167312B2 (en) Pause-based advertising methods and systems
JP6930421B2 (en) Information processing systems, information processing methods, recording media, and programs
KR20050013249A (en) Method, system and apparatus for monitoring use of electronic devices by user detection
US20170199934A1 (en) Method and apparatus for audio summarization
WO2014120438A1 (en) System and method for user monitoring and intent determination
EP2362582A1 (en) contextual domotique method and system
WO2016180914A1 (en) Home-automation system
KR100565583B1 (en) method for controlling refrigerator with multimedia function
US10264330B1 (en) Scene-by-scene plot context for cognitively impaired
CA2957181C (en) Scene-by-scene plot context for cognitively impaired
WO2017077791A1 (en) Notification control device, detection device, notification control system, notification control method, and detection method
CN105474652B (en) For user monitoring and it is intended to the system and method judged
WO2020250080A1 (en) System and method for context aware digital media management

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16722210

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16722210

Country of ref document: EP

Kind code of ref document: A1