WO2010104480A1 - System and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks - Google Patents

System and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks Download PDF

Info

Publication number
WO2010104480A1
WO2010104480A1 PCT/SI2009/000050 SI2009000050W WO2010104480A1 WO 2010104480 A1 WO2010104480 A1 WO 2010104480A1 SI 2009000050 W SI2009000050 W SI 2009000050W WO 2010104480 A1 WO2010104480 A1 WO 2010104480A1
Authority
WO
WIPO (PCT)
Prior art keywords
psychological state
psychophysiological
user
psychological
built
Prior art date
Application number
PCT/SI2009/000050
Other languages
French (fr)
Inventor
Mihelj Matjaz
Marko Munih
Domen Novak
Original Assignee
Univerza V Ljubljani
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Univerza V Ljubljani filed Critical Univerza V Ljubljani
Publication of WO2010104480A1 publication Critical patent/WO2010104480A1/en

Links

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/68Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/6813Specially adapted to be attached to a specific body part
    • A61B5/6825Hand
    • A61B5/6826Finger
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/16Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/16Devices for psychotechnics; Testing reaction times ; Devices for evaluating the psychological state
    • A61B5/165Evaluating the state of mind, e.g. depression, anxiety
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/68Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/683Means for maintaining contact with the body
    • A61B5/6831Straps, bands or harnesses
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/68Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient
    • A61B5/6801Arrangements of detecting, measuring or recording means, e.g. sensors, in relation to patient specially adapted to be attached to or worn on the body surface
    • A61B5/683Means for maintaining contact with the body
    • A61B5/6838Clamps or clips
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

The invention relates to a system for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks. The system and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks consists of the measurement subsystem (1), which comprises rings with sensors for photoplethysmography, skin conductance skin temperature (21, 22, 23, 24), a bracelet (3), connections (41, 42, 43, 44), a belt with ECG and respiration sensors (5), a microphone, camera, encoders (K1-K6) and wireless transmitters (01- 06). The system also includes a wireless receiver (6) connected to a computer subsystem (7). The computer subsystem (7) continuously stores the signals from the wireless transmitter (6) into memory or onto the hard drive. Signals in memory are filtered by the computer subsystem and used to calculate the psychophysiological parameters needed to estimate psychological state. The system allows the user to choose among different algorithms for psychological state estimation. It is designed modularly so that different sensors can be added or removed. The estimate of psychological state can be transmitted to a different location over various networks.

Description

System and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks
Field of the invention
The invention relates to a system for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks.
Description of the technical problem
The technical problem solved by this invention is the design of an integrated system for psychophysiological measurement, computer-aided processing of the measured responses, conversion of the processed data into an estimate of psychological state, and transmission of this estimate to a different location over various networks. The measurement method must record several different psychophysiological responses with sufficient accuracy and precision, but must not prevent the user from performing everyday tasks. It is sensible to use complex nonlinear methods to estimate psychological state since the connections between psychophysiological responses and psychological state are also largely nonlinear. The system must allow the user to choose among different estimates of psychological state, and must also output an estimate if not all of the possible psychophysiological responses are available for measurement. The system should be designed modularly so that different sensors can be added or removed at will.
State of the art
Methods of measuring psychophysiological responses have been described by many different patents and scientific articles. The wireless transmission of psychophysiological data was described by patent WO0201478. However, psychophysiological responses by themselves are difficult to interpret. For practical use of such a system, psychophysiological data must first be converted into a simple, understandable estimate of psychological state. Two principal models for psychological state estimation exist: the discrete model and the dimensional models. The discrete model is generally considered to be less useful in practice. Such a model allows a person to only feel one specific emotion (e.g. anger, sadness, joy, boredom, surprise) at a time, without the possibility of emotions occurring simultaneously. In real life, a person can be both surprised and happy at once, but this is not possible in a discrete model. The discrete model is described in the paper An argument for basic emotions, Cognition and Emotion, 6(3/4), 169-200, by Ekman, P. (1992). The dimensional model is described in the paper A circumplex model of affect, Journal of Personality and Social Psychology, 39, 1161-1178, by Russell, J. A. (1980).
Patent US6021346 describes the estimation of psychological state using electroencephalography (EEG) or functional magnetic resonance imaging (fMRI). These two methods are impractical as they require expensive equipment and hinder the user. There are also many patents (e.g. US6846106 and WO2008099320) that estimate psychological states based only on one psychophysiological response. Such an approach gives a very limited estimate. To obtain an accurate estimate, it is necessary to measure several different psychophysiological responses.
Patent application WO9733515 describes measurement of psychophysiological responses and estimation of psychological state using the dimensional model and statistical z-values. The z-value is defined as the difference between a specific measurement and the population mean, divided by the standard deviation of the population. The device described by this patent measures EMG, heart rate and skin conductance. It converts the measured data to z-values and uses each z-value to estimate a single psychological dimension. This patent does not describe data transmission and also has several weaknesses. EMG measurements are intrusive, as electrodes must be attached to the face in order to obtain psychophysiological data. The connection between heart rate and the »arousal« psychological dimension is also questionable, as a number of studies have failed to find any correlation. One such study is »Similar patterns of cardiovascular response during emotional activation as a function of affective valence and arousal and gender«, Journal of Psychosomatic Research, 50, 245-253, by Neumann, S. A. and Waldstein, S. R. (2001). The statistical z-value is also a relatively inaccurate method that does not take nonlinear connections between psychophysiological measurements and psychological states into account.
Patent application US2003139654 describes the measurement of psychophysiological responses and their conversion to psychological state using the discrete emotion model and support vector machine (SVM) algorithms. SVM algorithms have been described by, for example, Vapnik, V. N. (1999) in the paper An overview of statistical learning theory, published in IEEE Transactions on Neural Networks, 10, 988-999. The psychophysiological measurement device is integrated into a watch and can measure the electrocardiogram (ECG), photoplethysmogram (PPG), skin conductance and skin temperature. The device converts ECG and PPG into heart rate variability and additionally calculates heart rate variability (HRV). SVM algorithms are much more suitable for psychological state estimation using psychophysiological measurements than z-values. However, this invention does not describe data transmission and does not support the dimensional model of psychological state. Another weakness is the construction of the psychophysiological measurement device, which is integrated into a wristwatch. A wristwatch cannot measure the ECG, as that would require attaching multiple electrodes to different parts of the body. The PPG can be measured, but gives a less accurate estimate of heart rate than the ECG, as the peaks in the PPG signal are much wider than in the ECG signal. An inaccurate estimate of heart rate also results in an inaccurate estimate of heart rate variability. Measurement of skin temperature and conductance on the wrist is also not optimal, as a majority of psychophysiological studies measure these two responses on the fingers.
Patent application US2008221401 solves the problem of ECG measurement by placing the ECG sensors into a vest. This method allows measurement of both ECG and respiration. All other psychophysiological measurements described by this patent application (e.g. EEG, electrooculogram, EMG) are unsuitable for use in everyday life due to their intrusiveness. To accurately estimate psychological state, it is necessary to measure other psychophysiological signals in addition to ECG and respiration. This invention also uses only the discrete model of psychological state and does not describe data transmission.
Neural networks are described in, for example, B. J. A. Ruse & P. P. Van der Smagt (1995), An Introduction to Neural Networks, Seventh edition, The University of Amsterdam, Netherlands. Genetic algorithms are described in, for example, D. E. Goldberg (1989), Genetic algorithms in search, optimization and machine learning, Addison-Wesley Publishing Inc., Reading, Massachusetts, USA.
Description of the solution of the technical problem
The system and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks according to the invention is characterised in that sensors for measurement of psychophysiological responses are built into rings, a bracelet, a belt, a microphone, and a camera for tracking eye movements. The measured responses are converted into an estimate of psychological state using discrete or dimensional models of psychological states as well as nonlinear classification and decision-making methods. This estimate is transmitted to a different computer in a different location over a network, lead or protocol. The system and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks according to the invention will be described in more detail using figures that show:
Figure 1 - sketch of the invention
Figure 2 - example of sensor placement on the hand and body
Figure 3 - block diagram of the invention
Figure 4 - block diagram of calculations for estimation of psychological state
The system and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks includes the measurement subsystem 1, which comprises the rings 21, 22, 23, 24, the bracelet 3 and the connections 41 , 42, 43, 44. Rings 21 and 22 contain two electrodes that are part of the skin conductance sensor. Ring 23 contains a skin temperature sensor, which can be a thermistor or any other temperature sensor with a rapid response time. Ring 24 contains a photoplethysmogram sensor in the form of a pulse oximeter. Rings 21 and 22 are necessary to measure skin conductance. Each contains one integrated electrode, and a stable voltage source generates a current through the electrodes and the hand. Skin conductance is calculated as the ratio between the measured current and the voltage of the source. The voltage source is built into or attached to bracelet 3, which is connected to rings 21 , 22, 23 and 24 via wire connections 41 , 42, 43 and 44. Connections 41, 42, 43 and 44 can also be wireless. Voltage sources can also be built into the individual rings. The number of rings depends on the application. The minimal number is two rings, but this increases the size of each ring. If four rings are used, they are relatively small, but the larger number of rings can be intrusive for some users. The measurement subsystem 1 also includes a belt 5 around the body of the user. This belt contains an ECG sensor, consisting of three or four electrodes built into different parts of belt 5. The ECG is measured by measuring the voltages between these electrodes. Respiration is measured by measuring the circumference of, the body using strain gauges built into belt 5. Instead of belt 5, a vest may be used. The vest and belt are functionally equivalent, as both contain ECG and respiration sensors.
The measurement subsystem 1 also contains a microphone that records the user's voice.
In addition to the aforementioned components, measurement subsystem 1 also contains a camera that measures eye movement and pupil dilation. Any sufficiently high-quality camera may be used. It can also be connected directly to the computer subsystem 7.
Instead of being integrated into the belt, vest or rings on the user's hand, the psychophysiological sensors can be built into different objects that are not affixed to the user's body. For instance, the sensors can be built into devices that the user holds in his or her hands: computer mouse, joystick, telephone, haptic robot etc. The sensors can also be built into objects that are only briefly in contact with the user. One such object is a door handle. For such an implementation, it is possible to use sensors for skin temperature, skin conductance and PPG. Each of these sensors consists of electrodes that must be in contact with skin. While working with the object or device, the user lays his or her fingertips on the electrodes built into the object. The measurements proceed while the hand is in contact with the electrodes. Further transmission and analysis of the measured data is identical to that in the version with rings and belt/vest. A few examples of integrating sensors into different objects follow. The computer mouse is held by the user so that the forefinger, middle finger and ring finger rest on the three buttons while the thumb and little finger are pressed against the sides of the mouse. Electrodes for skin conductance measurement can be built into the buttons while the skin temperature and PPG sensors can be built into the sides of the mouse. Thus, the user's fingers are in contact with the sensors while using the mouse. The sensors automatically measure psychophysiological responses. The positions of the different sensors can also be exchanged. A similar implementation can be used for the joystick, which is held by the user with the entire hand. Some fingers rest on the buttons while others rest on the casing itself. Once again, the sensors are built into the buttons and casing so that the user's fingers are in contact with the sensors while using the joystick. In this way, psychophysiological sensors can be built into any device grasped by the user's hand. For some objects, the location of the user's fingers is clearly set. For instance, when using the computer mouse, the user always places his or her fingertips on the buttons. For other objects, it is necessary to add markings or depressions that indicate where the user's fingers should be placed. The sensors are then built underneath these markings or depressions. One such example would be a door handle, which can be held differently by different users. The sensors can be powered by the same energy source that powers the device. Alternatively, a separate power source can be used. A door handle, for instance, does not ordinarily require electricity, so a separate power source must be provided.
Signals from the sensors in the rings 21, 22, 23, 24, bracelet 3 and belt 5 are converted by encoders K1 , K2, K3, K4, K5 and K6 into digital signals suitable for wireless transmission. The simplest possible encoders are analog-digital converters, but encoders based on frequency modulation can also be used. Encoded signals from encoders K1, K2, K3, K4, K5 and K6 are transmitted via wireless transmitters 01, 02, 03, 04, 05 and 06 to wireless receiver 6, which is connected to the computer subsystem 7. The receiver 6 and computer subsystem 7 can be connected using USB, Bluetooth, Wi-Fi or other method. Encoders K1 , K2, K3, K4, K5 and K6 as well as wireless transmitters 01, 02, 03, 04, 05 and 06 are built into belt (or vest) 5, rings 21 , 22, 23, 24 and/or bracelet 3. If the sensors are built into a separate object that is only briefly in contact with the user (instead of into the belt, vest, rings and bracelet), it is possible to use data transmission methods other than the wireless transmitters O1 , O2, O3, 04, 05 and 06. As an example, if the sensors are built into a computer mouse, it is possible to use the USB cable built into the mouse to transmit data from the sensors to the computer subsystem. If the sensors are built into a mobile phone, the physiological data can be transmitted via the telephone network. The method of data transmission thus depends on the object the sensors are built into.
The computer subsystem 7 regularly stores the data received from wireless receiver 6 onto the hard drive. Only data required for estimation of the current psychological state is kept in memory. Older data is saved on the hard drive or deleted. The amount of data needed for estimation of the current psychological state can range from zero to 30 minutes of data, depending on the application and settings of the device. The computer subsystem filters the data in memory using digital band-pass filters. A separate filter is used for each psychophysiological signal. From the filtered data, the computer subsystem calculates the psychophysiological parameters required for estimation of psychological state. Instead of the computer subsystem, it is also possible to use any other electronic device capable of the required processing techniques, wireless signal receiving and data transmission over a network. For instance, it is possible to use a mobile phone, portable music player or an electronic device created specifically for this purpose.
Psychophysiological parameters differ from psychophysiological responses as follows: a psychophysiological response is the raw signal measured by the sensor (e.g. a raw ECG signal) while a psychophysiological parameter is a quantitatively estimated feature of the psychophysiological response (e.g. heart rate is a psychophysiological parameter calculated from the raw ECG signal). Psychophysiological parameters are described in the following paragraph. From the measured ECG and PPG signals, the computer subsystem calculates heart rate and different estimates of heart rate variability using the methods described by the Task Force of the European Society of Cardiology, The North American Society of Pacing and Electrophysiology in a 1996 paper titled "Standards of measurement, physiological interpretation, and clinical use", published in the European Heart Journal, volume 17, pages 354-381. The filtered skin conductance signal is already a usable psychophysiological parameter, but the computer subsystem also calculates the frequency and amplitude of the so-called skin conductance responses (SCRs), transient increases in skin conductance that have been well-described in literature. The subsystem also calculates various estimates of changes in skin conductance such as mean absolute derivative, variance, or the third and fourth central moments. From the signal obtained from the strain gauges in the belt/vest 3, the computer subsystem calculates different respiration parameters such as respiratory frequency, changes in lung volume, inspiratory time, expiratory time etc. Skin temperature is already by itself a useful parameter for estimation of psychological state.
The computer subsystem uses one of several possible algorithms for estimation of psychological state. Connections between psychophysiological parameters and psychological states can be defined by the user manually by entering mathematical or statistical equations or rules that describe these connections. One possible method of manually defining connections is via statistical z-values. For each possible psychological state (or, in the dimensional model, for each possible dimension of psychological state), the user first defines a range within which z-values of psychophysiological parameters must be located. The computer subsystem calculates all psychophysiological parameters and z-values, then determines which range (i.e. which psychological state or dimension of psychological state) the z-values lie in. Another method of manually defining connections is using fuzzy logic. The user defines ranges of psychophysiological parameters (for instance, for heart rate the range »low« can be defined as »50-60 beats per minute«), then defines simple rules (in the form of »if x, then y«) that connect psychophysiological ranges with psychological states. An example rule would be »if heart rate is low and respiratory variability is high, then psychological arousal is low«. The computer subsystem calculates all psychophysiological parameters, determines the range they lie in (e.g. »low«, »high«), and uses the defined rules to estimate the current psychological state.
It is also possible to merely provide the computer subsystem with a database that contains typical examples of psychophysiological responses to various psychological states. A database for a discrete model of psychological state would, for instance, contain typical psychophysiological responses for anger, fear, satisfaction, boredom etc. A database for a dimensional model of psychological state would, for instance, contain typical psychophysiological responses for low and high values of a particular psychological dimension (e.g. low/high arousal, low/high emotional valence). These typical responses are measured in advance on multiple people. Based on this data, the computer subsystem automatically develops an algorithm for psychological state estimation. There are many possible approaches for automatic learning: neural networks, genetic algorithms, support vector machines (SVM) etc. In a simple neural network, the algorithm uses only three mathematical operations: addition, multiplication with a constant, and a threshold function. In the beginning, the constants in the algorithm are not defined. The user provides input data in the form of psychophysiological parameters and output data in the form of numerical values corresponding to different psychological states. The computer subsystem then automatically calculates the constants in the algorithm in order to obtain the output data from the input data. Advanced neural networks operate on multiple levels and also use other mathematical operations. Genetic algorithms operate similarly to neural networks. The computer subsystem randomly generates thousands of possible algorithms for estimation of psychological state, then calculates the output that these algorithms provide in response to the input data. The algorithms whose outputs are closest to the desired outputs are combined with each other. The outputs of the resulting combined algorithms in response to the input data are once again compared with the desired outputs. This continues iteratively (for up to thousands of iterations) until the outputs of the final algorithm are very close to the desired outputs.
The computer subsystem of the system for estimation of psychological state can include one or more of the above algorithms for converting psychophysiological parameters to psychological states. If the system includes several algorithms, the user can choose among them freely using, for example, buttons or menus in the computer software. Alternatively, the system can contain a database in addition to predefined algorithms. The user can thus choose the database entries that should be used for automatic learning by the computer subsystem. The user can both add and remove entries from the database. A new entry can be added by specifying the user's psychological state at a given time. The computer subsystem then records the current value of psychophysiological parameters and saves it in the database together with the specified psychological state. This allows the system to adapt to a specific user.
The system must also be able to provide an estimate of psychological state if it does not have access to data from all the sensors. In this case, the system provides only a partial estimate: either a list of several possible discrete states or only some of the possible psychological dimensions.
After psychophysiological parameters have been converted into an estimate of psychological state, the resulting estimate can be used by a different program on the same computer. However, the main purpose of the system is the transmission of the estimate of psychological state to a different computer or device 8 over different networks, leads or protocols 9. For transmission in the same building, it is possible to use LAN, Wi-Fi or a similar approach. For transmission over large distances, the Internet or telephone network can be used. These transmissions can either proceed in real time, in regular intervals, or only when changes in psychological state occur. The algorithm for transmission of the estimate of psychological state can be either independent or built into other programs such as MSN Messenger, Facebook, Second Life, World of Warcraft etc. Three possible applications of the system follow. Each of these three can use any of the listed methods of estimating psychological state or any other method.
Version I of the system is used for virtual-reality-assisted psychotherapy. The patient is placed into a virtual environment that can stimulate one or more senses. The simplest environments only stimulate vision while more advanced environments also stimulate hearing or touch. In this virtual environment, the user encounters some kind of stressful situation. This is usually something stressful that the user regularly encounters in real life. Examples of such situations are flight and public speaking. However, it is also possible that the patient encountered an extremely stressful situation only once and suffered long-term consequences. Examples of such situations are war, accidents or deaths in the family. The virtual environments are not dangerous for the patient, but do induce stress. This allows the user to experience a stressful environment that would be dangerous, expensive or difficult to replicate in real life. After repeatedly entering a virtual environment, the patient becomes accustomed to it and the level of stress decreases. At the end of the therapy, the patient can successfully cope with the same situation in real life. This method of psychotherapy has been successfully used to treat various disorders. The principal weakness of the method is that it requires the psychotherapist to constantly monitor the virtual environment and terminate therapy if it becomes too stressful for the patient. A possible solution is for the therapist to remotely monitor the patient using the system for estimation and transmission of the patient's psychological state. The patient could thus, for instance, experience the virtual environment on his or her home computer without the therapist's physical presence. However, this would require very detailed information about the patient's state. Incorrect information about psychological state could result in ineffective or even harmful therapy. For this application, the system comprises all of the listed hardware components. The patient is not allowed to choose the algorithm for conversion of psychophysiological responses to psychological state. The algorithm is either predetermined or chosen by the therapist during therapy.
Version Il of the system is aimed at socializing and entertainment. Two main applications are the so-called massively multiplayer online (MMO) virtual worlds and social networking websites. In MMOs, each user assumes the form of a virtual character ('avatar'). This avatar can be shaped according to the user's wishes. Avatars can explore the world, talk to other characters, learn etc. When two avatars meet, a user can immediately obtain certain information about the other user and avatar. If every user is equipped with a system for estimating psychological state, users can obtain information about other users' psychological states. This information can be provided as text. Alternatively, the avatar that represents the user can change. As an example, the facial expression of an avatar can reflect the psychological state of the user that controls the avatar.
On social networking websites, each user has a profile with different information. Basic websites include only a small amount of text and a photograph, but more advanced websites allow users to add applications such as quizzes, music players, interactive maps, virtual pets etc. A user who is connected to a system for estimation of psychological state would be able to display his or her own psychological state on his or her profile. This information would be regularly updated and visible to all other users. It could be shown as text or graphically (e.g. a face that laughs, cries, frowns...).
When using version Il of the system for socializing and entertainment, high accuracy is not required. An incorrect estimate of psychological state cannot have serious consequences. However, the system must be as unobtrusive as possible. For version Il of the system, only the sensors on the hand, the microphone and the camera are used. These components can be attached or removed very quickly and are relatively unobtrusive. The algorithm for conversion of psychophysiological responses to psychological state can be either chosen in advance or set by the user.
Version III of the system is a personal psychological state monitor. For this, the estimate of psychological state does not need to be transmitted to a different location. The estimate is used primarily by the person whose psychophysiological responses are being measured. Thus, the estimate of psychological state remains on the computer where it was calculated. The algorithm for conversion of psychophysiological responses to psychological state is selected by the user. Any of the previously listed sensors can be used. With such a system, the user can monitor his or her own level of stress, allowing him or her to more easily avoid stressful situations or cope with phobias.
The work leading to this invention has received funding from the European Community's Seventh Framework Programme (FP7/2007-2011) under grant agreement n° 215756.

Claims

Patent claims
1. A system and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks, characterised in that it comprises a measurement subsystem (1) consisting of rings (21 , 22, 23, 24), a bracelet (3) and connections (41, 42, 43, 44), that two electrodes for a skin conductance sensor are built into rings (21 and 22), that a skin temperature sensor is built into ring (23), that this temperature sensor can be either a thermistor or any other temperature sensor, that a pulse oximeter for photoplethysmography is built into ring (24), that rings (21 and 22) are necessary for skin conductance measurements, that each contains one integrated electrode, that a stable voltage source generates a current through the electrodes and the hand, that skin conductance is calculated as the ratio between the measured current and the voltage of the source, that the voltage source is built into and/or attached to bracelet (3), which is connected to rings (21, 22, 23 and 24) via connection wires (41, 42, 43 and 44); that connections (41, 42, 43 and 44) can also be wireless, that the voltage source can also be built into the individual rings; that measurement subsystem (1) includes a belt (5) around the user's body, that an ECG sensor consisting of three or four electrodes is built into the belt (5), that data about respiration are gathered by measuring the circumference of the body using strain gauges built into belt (5), that measurement subsystem (1 ) also includes a microphone for recording the user's voice, that measurement subsytem (1) includes a camera for measuring pupil dilation and eye movement, that the camera can also be connected directly to a computer, that the signals from the sensors in rings (21 , 22, 23, 24), bracelet (3) and belt (5) are converted to digital signals suitable for wireless transmission by encoders (K1 , K2, K3, K4, K5 and K6), that the encoded signals from encoders (K1 , K2, K3, K4, K5 and K6) are sent to a wireless receiver (6) via wireless transmitters (01 , 02, 03, 04, 05 in 06), that the wireless receiver is connected to a computer subsystem (7), that the connection between the receiver (6) and computer subsystem (7) can be realized using USB, BlueTooth, WiFi or other methods, that encoders (K1 , K2, K3, K4, K5 and K6) and wireless transmitters (01 , O2, O3, 04, 05 and 06) are built into the belt (5), rings (21, 22, 23, 24) and/or bracelet (3), that the computer subsystem (7) stores the signals received from wireless receiver (6) in memory or on the hard drive, that the memory contains only signals of the last zero to thirty minutes that are required to estimate the current psychological state, that older signals are saved on the hard drive or regularly deleted, that signals in memory are filtered by the computer subsystem with digital bandpass filters, that a different filter is used for each signal, and that the computer subsystem uses the filtered signals to calculate the psychophysiological parameters needed to estimate psychological state.
2. The system according to claim 1, characterised in that a vest is used instead of belt (5).
3. The system according to claims 1 and 2, characterised in that, instead of the computer subsystem, a different electronic device capable of the required processing techniques, wireless signal receiving and data transmission over various networks is used; that this can be, for example, a mobile phone, portable music player or an electronic device created specifically for this purpose.
4. The system according to claims 1 to 3, characterised in that the user can manually define connections between psychophysiological parameters and psychological states by entering mathematical or statistical equations or rules that describe these connections; that the user defines connections using statistical z-values; that the user defines ranges of z-values of psychophysiological parameters for each psychological state or each dimension of psychological state; that the computer subsystem calculates all psychophysiological parameters and z-values, then determines which range (i.e. which psychological state or dimension of psychological state) they lie in.
5. The system according to claims 1 to 3, characterised in that the user can manually define connections between psychophysiological parameters and psychological states using fuzzy logic; that the user defines ranges of psychophysiological parameters, then defines simple rules in the form of »if x then y« that connect ranges of psychophysiological parameters with psychological states; that the computer subsystem calculates all psychophysiological parameters, determines the range they lie in, and uses the defined rules to estimate the psychological state.
6. The system according to claims 1 to 3, characterised in that the system is provided with a database and learns to estimate psychological states on its own based on the data in the database; that neural networks are used as the learning method, that the database contains typical values of psychophysiological parameters for different psychological states; that the database for the discrete model contains, for example, typical values of psychophysiological parameters for anger, fear, satisfaction, boredom etc.; that the database for the dimensional model contains, for example, typical values of psychophysiological parameters for a low value of each dimension and a high value of each dimension; that these dimensions are low/high arousal and/or low/high emotional valence; that these typical values are measured in advance on multiple people; that the computer subsystem uses the data in the database to develop an algorithm for psychological state estimation; that, after a large amount of iterations, the computer subsystem obtains algorithms that work very well for the data in the database.
7. The system according to claims 1, 2, 3 and 6, characterised in that a genetic algorithm is used as the learning method.
8. The system according to claims 1, 2, 3 and 6, characterised in that support vector machines are used as the learning method.
9. The system according to claims 1 to 8, characterised in that the estimate of psychological state is transmitted to a different device or location over various networks, leads or protocols; that the transmission of the estimate can be done in real time, in regular intervals or when changes in psychological state occur; that the algorithm for transmission of the estimate of psychological state can be independent or built into other applications.
10. The system according to claims 1 to 9, characterised in that the system is meant for psychotherapy using virtual reality; that the user is placed into a virtual environment that can stimulate one or more senses, that the patient is not allowed to choose the algorithm for conversion of psychophysiological responses to psychological state since it is either predetermined or chosen by the therapist during therapy.
11. The system according to claims 1 to 9, characterised in that it is meant for socializing and entertainment, that the two principal examples of use are so- called massively multiplayer online virtual worlds and social networking websites.
12. The system according to claims 1 to 8, characterised in that it is used as a personal psychological state monitor, that the purpose of such a system is not to transmit information about psychological state to a different location, that the information is meant primarily for the person whose psychophysiological responses are being measured.
13. The system according to claims 1 and 2, characterised in that the psychophysiological sensors are attached to or integrated into devices or objects that the user holds in his or her hand; that sensors are integrated into a computer mouse, joystick, mobile phone, door handle, haptic robot and/or other object.
14. The system according to claim 13, characterised in that the sensors for skin conductance, skin temperature and photoplethysmography are built into marked locations on a computer mouse, joystick, telephone, mobile phone, door handle, haptic robot and/or other object; that each of these sensors must consist of electrodes that need to be in contact with the user's skin; that measurements proceed while the hand is in contact with the electrodes, that further transmission, processing and analysis of the measured signals is identical to the implementation with rings and belt/vest.
PCT/SI2009/000050 2009-03-12 2009-10-14 System and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks WO2010104480A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
SIP-200900067 2009-03-12
SI200900067A SI23012A (en) 2009-03-12 2009-03-12 System and procedure for assessing psychological statuses based on psychophysical reponses and transfer of such conditions across various networks

Publications (1)

Publication Number Publication Date
WO2010104480A1 true WO2010104480A1 (en) 2010-09-16

Family

ID=42271981

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SI2009/000050 WO2010104480A1 (en) 2009-03-12 2009-10-14 System and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks

Country Status (2)

Country Link
SI (1) SI23012A (en)
WO (1) WO2010104480A1 (en)

Cited By (20)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8700009B2 (en) 2010-06-02 2014-04-15 Q-Tec Systems Llc Method and apparatus for monitoring emotion in an interactive network
CN103932696A (en) * 2014-04-18 2014-07-23 赵小英 Multifunctional health status detection method and system
CN104042199A (en) * 2014-07-04 2014-09-17 时云医疗科技(上海)有限公司 Characteristic index feedback system and method
ES2507891A1 (en) * 2014-03-04 2014-10-15 Francisco DIEGO GÓMEZ Ring with automatic vital signs measurement and localization (Machine-translation by Google Translate, not legally binding)
CN104107037A (en) * 2014-07-15 2014-10-22 北京博实联创科技有限公司 Physiological information acquiring and processing system
CN104116502A (en) * 2014-07-04 2014-10-29 深圳市凯沃尔电子有限公司 Multi-parameter monitor and implementation method thereof
US20150080756A1 (en) * 2012-01-27 2015-03-19 The University Of Sydney Estimating arousal states
US9138558B1 (en) * 2014-06-19 2015-09-22 Biofeedback Systems Design, LLC Apparatus and method for improving psychophysiological function for performance under stress
CN105615901A (en) * 2014-11-06 2016-06-01 中国移动通信集团公司 Emotion monitoring method and system
WO2016123216A1 (en) * 2015-01-27 2016-08-04 Medicomp, Inc. Finger ring electrocardiogram monitor and associated systems and methods
CN105852830A (en) * 2016-04-27 2016-08-17 安徽理工大学 Mental health monitoring system and method based on cloud platform
CN105997019A (en) * 2016-05-09 2016-10-12 鲍崇智 A body sensor network-based multi-dimensional heartbeat information synchronous collection method and system
CN106236117A (en) * 2016-09-22 2016-12-21 天津大学 Emotion detection method based on electrocardio and breath signal synchronism characteristics
CN106491110A (en) * 2016-10-13 2017-03-15 深圳缪斯智能珠宝有限公司 A kind of method of testing of heart rate data and mobile terminal
CN106491111A (en) * 2016-10-13 2017-03-15 深圳缪斯智能珠宝有限公司 A kind of acquisition method of heart rate data and mobile terminal
CN106491112A (en) * 2016-10-13 2017-03-15 深圳缪斯智能珠宝有限公司 A kind of processing method of heart rate data and mobile terminal
CN106691430A (en) * 2016-12-21 2017-05-24 深圳欧德蒙科技有限公司 Emotion judging method and device
CN108784717A (en) * 2018-04-16 2018-11-13 无锡市人民医院 A kind of sound behavioral data of artificial intelligence acquires the analyzer for the degree that feels depressed
EP3673792A1 (en) 2018-12-27 2020-07-01 Politechnika Slaska A device for carrying out psychological tests as part of professional suitability and qualification for work verification process
CN115641942A (en) * 2022-12-01 2023-01-24 威特瑞特技术有限公司 Examination psychological training method, device and system based on virtual reality

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002001478A1 (en) * 2000-06-28 2002-01-03 Emotra Ab Method and device for wireless transmission and processing of psychophysiological data
US20030120172A1 (en) * 2001-05-25 2003-06-26 Eastman Kodak Company Technique for diagnosing attention deficit hyperactivity disorder
US20080081963A1 (en) * 2006-09-29 2008-04-03 Endothelix, Inc. Methods and Apparatus for Profiling Cardiovascular Vulnerability to Mental Stress
WO2008055078A2 (en) * 2006-10-27 2008-05-08 Vivometrics, Inc. Identification of emotional states using physiological responses

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2002001478A1 (en) * 2000-06-28 2002-01-03 Emotra Ab Method and device for wireless transmission and processing of psychophysiological data
US20030120172A1 (en) * 2001-05-25 2003-06-26 Eastman Kodak Company Technique for diagnosing attention deficit hyperactivity disorder
US20080081963A1 (en) * 2006-09-29 2008-04-03 Endothelix, Inc. Methods and Apparatus for Profiling Cardiovascular Vulnerability to Mental Stress
WO2008055078A2 (en) * 2006-10-27 2008-05-08 Vivometrics, Inc. Identification of emotional states using physiological responses

Non-Patent Citations (4)

* Cited by examiner, † Cited by third party
Title
DOMEN NOVAK ET AL: "Using Psychophysiological Measurements in Physically Demanding Virtual Environments", 24 August 2009, HUMAN-COMPUTER INTERACTION Â INTERACT 2009, SPRINGER BERLIN HEIDELBERG, BERLIN, HEIDELBERG, PAGE(S) 490 - 493, ISBN: 978-3-642-03654-5, XP019126190 *
MANDRYK R L ET AL: "A fuzzy physiological approach for continuously modeling emotion during interaction with play technologies", INTERNATIONAL JOURNAL OF HUMAN COMPUTER STUDIES APRIL 2007 ACADEMIC PRESS GB, vol. 65, no. 4, April 2007 (2007-04-01), pages 329 - 347, XP008123987, DOI: DOI:10.1016/J.IJHCS.2006.11.011 *
MATJAZ MIHELJ ET AL: "Emotion-aware system for upper extremity rehabilitation", VIRTUAL REHABILITATION INTERNATIONAL CONFERENCE, 2009, IEEE, PISCATAWAY, NJ, USA, 29 June 2009 (2009-06-29), pages 160 - 165, XP031496997, ISBN: 978-1-4244-4188-4 *
PETER C ET AL: "Emotion representation and physiology assignments in digital systems", INTERACTING WITH COMPUTERS, BUTTERWORTH-HEINEMANN, GB LNKD- DOI:10.1016/J.INTCOM.2005.10.006, vol. 18, no. 2, 1 March 2006 (2006-03-01), pages 139 - 170, XP025079324, ISSN: 0953-5438, [retrieved on 20060301] *

Cited By (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8700009B2 (en) 2010-06-02 2014-04-15 Q-Tec Systems Llc Method and apparatus for monitoring emotion in an interactive network
US20150080756A1 (en) * 2012-01-27 2015-03-19 The University Of Sydney Estimating arousal states
ES2507891A1 (en) * 2014-03-04 2014-10-15 Francisco DIEGO GÓMEZ Ring with automatic vital signs measurement and localization (Machine-translation by Google Translate, not legally binding)
CN103932696A (en) * 2014-04-18 2014-07-23 赵小英 Multifunctional health status detection method and system
US9668693B2 (en) 2014-06-19 2017-06-06 Biofeedback Systems Design, LLC Method for improving psychophysiological function for performance under stress
US9138558B1 (en) * 2014-06-19 2015-09-22 Biofeedback Systems Design, LLC Apparatus and method for improving psychophysiological function for performance under stress
US9402581B2 (en) 2014-06-19 2016-08-02 Biofeedback Systems Design, LLC Apparatus and method for improving psychophysiological function for performance under stress
US11647948B2 (en) 2014-06-19 2023-05-16 Optivio, Inc. System for training a subject to improve psychophysiological function for performance under stress
US10758180B2 (en) 2014-06-19 2020-09-01 Biofeedback Systems Design, LLC Method for improving psychophysiological function for performance under stress
CN104042199A (en) * 2014-07-04 2014-09-17 时云医疗科技(上海)有限公司 Characteristic index feedback system and method
CN104116502A (en) * 2014-07-04 2014-10-29 深圳市凯沃尔电子有限公司 Multi-parameter monitor and implementation method thereof
CN104107037A (en) * 2014-07-15 2014-10-22 北京博实联创科技有限公司 Physiological information acquiring and processing system
CN105615901A (en) * 2014-11-06 2016-06-01 中国移动通信集团公司 Emotion monitoring method and system
WO2016123216A1 (en) * 2015-01-27 2016-08-04 Medicomp, Inc. Finger ring electrocardiogram monitor and associated systems and methods
CN105852830A (en) * 2016-04-27 2016-08-17 安徽理工大学 Mental health monitoring system and method based on cloud platform
CN105997019B (en) * 2016-05-09 2019-02-19 鲍崇智 Multidimensional heartbeat message synchronous collection method and system based on body-sensing net
CN105997019A (en) * 2016-05-09 2016-10-12 鲍崇智 A body sensor network-based multi-dimensional heartbeat information synchronous collection method and system
CN106236117A (en) * 2016-09-22 2016-12-21 天津大学 Emotion detection method based on electrocardio and breath signal synchronism characteristics
CN106491112A (en) * 2016-10-13 2017-03-15 深圳缪斯智能珠宝有限公司 A kind of processing method of heart rate data and mobile terminal
CN106491111A (en) * 2016-10-13 2017-03-15 深圳缪斯智能珠宝有限公司 A kind of acquisition method of heart rate data and mobile terminal
CN106491110A (en) * 2016-10-13 2017-03-15 深圳缪斯智能珠宝有限公司 A kind of method of testing of heart rate data and mobile terminal
CN106691430A (en) * 2016-12-21 2017-05-24 深圳欧德蒙科技有限公司 Emotion judging method and device
CN106691430B (en) * 2016-12-21 2019-10-11 深圳欧德蒙科技有限公司 A kind of emotion judgment method and apparatus
CN108784717A (en) * 2018-04-16 2018-11-13 无锡市人民医院 A kind of sound behavioral data of artificial intelligence acquires the analyzer for the degree that feels depressed
EP3673792A1 (en) 2018-12-27 2020-07-01 Politechnika Slaska A device for carrying out psychological tests as part of professional suitability and qualification for work verification process
CN115641942A (en) * 2022-12-01 2023-01-24 威特瑞特技术有限公司 Examination psychological training method, device and system based on virtual reality

Also Published As

Publication number Publication date
SI23012A (en) 2010-09-30

Similar Documents

Publication Publication Date Title
WO2010104480A1 (en) System and procedure for estimation of psychological state based on psychophysiological responses and transmission of the estimated state over various networks
CN109804331B (en) Detecting and using body tissue electrical signals
Novak et al. Psychophysiological measurements in a biocooperative feedback loop for upper extremity rehabilitation
US8700009B2 (en) Method and apparatus for monitoring emotion in an interactive network
Novak et al. Workload estimation in physical human–robot interaction using physiological measurements
Novak et al. Predicting targets of human reaching motions using different sensing technologies
AU2022268375B2 (en) Multiple switching electromyography (emg) assistive communications device
CN114007705A (en) Brain-computer interface for computing systems
US11699524B2 (en) System for continuous detection and monitoring of symptoms of Parkinson's disease
Castro-Garcia et al. Towards human stress and activity recognition: A review and a first approach based on low-cost wearables
Kawala-Janik et al. Method for EEG signals pattern recognition in embedded systems
KR102429630B1 (en) A system that creates communication NPC avatars for healthcare
Ponte et al. Monitoring game-based motor rehabilitation of patients at home for better plans of care and quality of life
KR20190080598A (en) System for recognizing emotion using biometric data and method thereof
Jakopin et al. An unobtrusive measurement method for assessing physiological response in physical human–robot interaction
Edmonds et al. On physiological computing with an application in interactive art
Bacchini et al. Emopad: An affective gamepad
Tsiourti Artificial agents as social companions: design guidelines for emotional interactions
Tavares et al. Physiologically attentive user interface for improved robot teleoperation
Bacchini et al. Developing an affective Point-of-Care technology
WO2023026927A1 (en) Psychological state estimation system, psychological state estimation method, and program
Kubacki et al. Application of bio signals in the brain-device interfaces
US20240090827A1 (en) Methods and Systems for Improving Measurement of Sleep Data by Classifying Users Based on Sleeper Type
Choi Minimally-invasive wearable sensors and data processing methods for mental stress detection
Pozaić et al. Closed-loop system for assisted strength exercising

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 09764331

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 09764331

Country of ref document: EP

Kind code of ref document: A1