EP1605338A2 - Apparatus and method for determining a position, in particular for augmented reality applications - Google Patents

Abstract

An improved position determining system, especially for an augmented reality system, has a number of imaging systems (1, 2) and position sensors (3, 4) linked to the system via an interface (9) via which the inputs are integrated into the processing system. The different sensor outputs and their algorithms are processed to provide an optimum position identity, with simple and rapid updates for new data.

Description

The invention relates to a system and a method for position determination a user and / or a mobile one Device by means of tracking methods, in particular for augmented reality Applications.

Augmented reality, augmented reality, is a form of Human-technology interaction, which gives humans z. B. over a Data glasses display information in his field of view and so that the reality perceived by him expands. Around virtual extension of the real field of view accurately positioned tracking methods are used. these can consist of both hardware and software. Come here optical, inertial, acoustic, magnetic etc. procedures for use. The individual methods are based on different Algorithms on, all a more or less provide accurate information about the position of the user.

For the determination of the user position and its viewing direction, is a positionally precise overlay necessary. Separate Tracking methods have advantages in themselves also disadvantages. So comes an optical process without additional Instrumentation, but requires a lot of computing power, Which is currently not an industrial grade mobile system can be provided. inertial Systems have very good real-time capability, whereas such methods capture only relative movements of the user can. Magnetic processes are prone to magnetic fields, so due to industrial environments the existing machines and electric drives such procedures are disturbed. Likewise, due to the Noise background in a production hall. Acoustic process only conditionally used.

The different procedures are so far almost exclusively considered alone. First approaches exist in the combination of these methods (e.g., inertial with acoustically). These approaches are, however, due to the elaborate Instrumentation of the environment for industrial, medical or consumer applications are not suitable.

Today's systems are mainly based on a procedure. Purely video-based methods usually work with instrumentation the environment, which for industrial applications not suitable because the environment is equipped with markers must become. These markers are subject to an industrial Surrounding the environment, so that this pollution and then no longer be recognized by the system. Next is one Instrumentation of the environment with a high temporal Effort, so the effectiveness of such System is no longer acceptable. There are also approaches for optical processes, which without instrumentation get along. But there is no known method which the Real-time capability required for an industrial application on a mobile system.

In the research landscape are also combinations of To find methods that are not eligible for real-time capability lay. But this is for an industrial use absolutely necessary. Next must when using in the Application domains that are considered portable systems are used, the only limited computing power provide. One approach to such a system is currently unknown. It should also be noted that the various Process their results in different ways Provide frequencies. When merging the individual results this must be taken into account.

Object of the present invention is therefore a system and to provide a method by means of which a simple combination and application of different tracking methods to determine a position is possible.

The object is achieved by a system with the specified in claim 1 Characteristics solved. Furthermore, the task is through a method having the features specified in claim 15 solved.

The invention is based on the finding that the combination the results of the procedures to improve the Overall result leads.

About the definition of a universal system, below Framework called, for the positioning method (Tracking framework) will ensure that the individual Procedures are interchangeable or supplements. This one has the Advantage that new or improved procedures quickly over the system or framework are integrated into an application can. By defining interfaces can also foreign-developed procedures are involved. This ensures the greatest possible openness.

Because of the very different applications and the so related requirements (location, conditions of use, Environmental conditions, etc.) will be accompanied by different approaches needed the tracking method. In the application case automotive industry comes as workspace e.g. an engine room of a car for use. This differs considerably from the Application aircraft, where e.g. the chassis no limit of the working space is given by the body. Of the Application Automation can e.g. not on existing ones Falling back on 3D models as these are in this application domain usually are not present. Other application domains have even more different boundary conditions.

To meet these requirements, is according to the invention a system or framework is provided which the Exchange or adaptation of the procedures allowed. Furthermore a well-defined framework quickly allows new, improved To include procedures in this framework and thus the Suitability of the whole system in the application domain too improve.

A universal tracking framework plays a major role in the realization of industrial tracking methods or the related applications because of this framework the possibility is given of different tracking systems / procedures to integrate into the system. Thus, within applications quickly responded to changing conditions and that is suitable and optimized for the application System can be selected.

So that the different types of tracking methods work together can, the different sensors have to System can be connected. The division into fast and Robust tracking shares enable real-time capability to be realized of the overall system under consideration of existing resources (e.g., a portable computer).

The tracking framework builds at runtime based on a configuration description the communication between the needed Tracking methods and algorithms. The data flows This communication will be automatic by the tracking framework with timestamps that synchronize with the Ensure processing of the data. Due to the flexibility With these basic mechanisms, the framework ensures its openness towards future improved or enhanced tracking methods.

It is particularly advantageous that a framework for position determination in industrial environments. This framework will meet the requirements of the application domains Industry, medicine, komsumer etc. just by the procedures to follow an overall system concept. The framework offers the possibility of different sensors in the system neutral to integrate and thus creates an openness for later Adjustments and extensions. The integration of procedures happens with the help of defined interfaces, so that an exchange of information between processing units different suppliers within the framework is possible.

Sensors and visualization units can have a neutral Interface to access the framework or information provide. Sensors can be used via the interface integrated, which are not yet known today. The Configurability through a simple description (for example XML file) is a novel method of data flows build up between the individual procedures. The takeover the synchronization of the data through the framework to facilitate the construction of such methods today's techniques. Thus, this framework meets the requirements fair, but also future applications.

The interface further allows that different visualization units can be used, which e.g. optimized for a particular application. Thus, will with the framework, e.g. a pure visual positioning method build on the basis of markers. For industrial applications, however, must be combined procedures be used. Due to the openness of the framework Ensures that complex combinations of different Procedures can be established.

Further, various approaches of augmented reality can be made possible become. These are for a video sea-through Representation in which the user uses his environment a perceived video. This is for industrial applications but only conditionally usable, so that using the framework also Optical See-Through approaches can be realized in which only the additional virtual Objects in the correct position based on the position determination to be displayed.

In Figur 1 ist das System, im Folgenden Framework, dargestellt. Die Wahrnehmung der Umgebung erfolgt über die Sensoren 1, 2, 3, 4. Zum einen können mehrere Videokameras 1, 2 eingesetzt werden, die den Arbeitsraum oder die Umgebung beobachten. Um die Stabilität des Gesamtsystems zu verbessern können weitere Sensoren 3, 4 integriert werden. Diese können z.B. Intertial-Sensoren aber auch Sensoren sein, welche auf anderen Verfahren beruhen.

In the following the invention with reference to the embodiment shown in the figure is described and explained in more detail:

FIG. 1 shows the system, in the following framework. The perception of the environment via the sensors 1, 2, 3, 4. On the one hand, a plurality of video cameras 1, 2 can be used, which observe the work space or the environment. To improve the stability of the overall system further sensors 3, 4 can be integrated. These can be eg intertial sensors but also sensors based on other methods.

The sensors are via sensor-specific processing units (Sources) 5, 6, 7, 8 connected to the system. These accomplish sensor-specific processing units the process-specific implementation as well as the synchronization necessary extensions of the sensor information. About these sources 5, 6, 7, 8 becomes a sensor-neutral Integration via the framework interface or the interface 9, so that more and possibly still unknown Sensors can also be integrated later.

The video-based data becomes features from the images certainly. The features (e.g., points, lines, color values, etc.) are then tracked from image to image and from it the respective Shifts calculated. The shifts give the Movement of the user again. This takes place in the respective Pre-processing units 10, 11. These units are specially optimized for fast execution, which for mobile systems with limited computing power is needed.

In the direct processing unit (preprocessing light) 12 will be based on the results of preprocessing 10, 11 and provided by the central processing unit 13 exact position calculates a user position. The "Preprocessing light" must also have real-time requirements fulfill. In the indirect processing unit 16 at the "pre-processing robust" algorithms are used the fault accumulations, such as Avoid drifts, and with it a long-term stability can be ensured. These Procedures consume more power than the other methods.

Should the performance of the processing system be insufficient, so this unit can be outsourced. This can e.g. take place via a radio link 21.

All results of the individual processing steps 7, 8, 10, 11, 16 are merged in the central processing unit 13. Here are the calculations by means of the reliabilities the results evaluated by a statement as accurate as possible about the current position of the user. By means of an optional movement model 15 of humans the calculation result can be further improved. The The result of the central processing unit 13 can be any module provided as a basis for the calculations become. The modules "preprocessing light" 12 and the motion model 15 need this data for further, new calculations to carry out. The result of central processing 13 is also delivered to a manager 17.

The processing must be initialized at the beginning. The initialization 14 represents the exact position of the user to disposal. Should the reliability of the position calculation get worse in central processing 13, a further initialization (reinitialization) is initiated. This will again be a precise position of the user made available. When (re) initializing is no user interaction necessary.

The manager 17 manages the result of the position calculation of the framework. The results are the visualization 18, 19 provided on the basis of User position the virtual objects in the field of view of the user Pose user. This is done via a neutral interface 9, so that different visualizations are served can. Only by means of an exact position can a user-friendly Overlay occur.

On the other hand, a configuration 20 is read at startup. The configuration contains the ones for the position determination necessary modules and the exchanged between the modules Information flows. By means of the configuration can Thus, the framework is built and adapted to the application become. There is also the possibility of the visualizations 18, 19 parameter sets via the manager 17 to the Transfer processing units. The parameter sets contain the situation-based configurations of the position determination, which can vary over a use case.

FIG. 1 gives an overview of the framework for position determination of a user. In this picture is a System with several video-based sensors and several shown further sensors. The information of the sensors are processed and used to calculate the position of the user. The position determination takes place, as described above, in various processing steps, so that the necessary real-time capability of the system can be ensured can.

Furthermore, by means of the system or framework in addition to the Position of a human user also the position and the direction of movement of a movable device, such as of a robot. The results processing the various tracking methods the device then in machine-readable form made available.

• Es wird durch z.B. einen Anwendungsentwickler festgestellt, welche Sensoren 1, 2, 3, 4 und welche Verarbeitungseinheiten 10, 11, 12, 13, 14, 15, 16 für einen Anwendungsfall zum Einsatz kommen sollen. Aus diesen Informationen wird eine Konfiguration 20 erstellt, der die Sensoren an das System anschließt sowie die Algorithmen mit einander verschaltet. Diese Konfiguration wird dem System bekannt gemacht.
• Durch das Starten des Frameworks bzw. des Manager 17 wird die Konfiguration eingelesen und das Positionsbestimmungssystem entsprechend der Konfiguration aufgebaut. Die einzelnen Verarbeitungseinheiten 10, 11, 12, 13, 14, 15, 16 werden miteinander verschaltet, sodass eine Verarbeitung der Sensordaten ermöglicht wird.
• Eine Visualisierung 18 oder 19 wird gestartet und die Visualisierung baut über das Interface bzw. die Schnittstelle 9 eine Verbindung zum Manager 17 auf.
• Mittels der Initialisierung 14 wird die Position des Benutzers zu Beginn festgestellt. Hier kann z.B. ein markerbasiertes Verfahren zum Einsatz kommen. Es sind aber auch andere Verfahren denkbar.
• Nachdem die initiale Position des Benutzers bekannt ist, wird diese den Verarbeitungseinheiten bekannt gegeben.
• Die Anbindungen der Sensoren 5, 6, 7, 8 liefern daraufhin ihre Daten inkl. Synchronisationsdaten über das Interface bzw. die Schnittstelle 9 an die Verarbeitungseinheiten.
• Aus den videobasierten Daten 5, 6 werden in der Vorverarbeitung 10, 11 Merkmale (z.B. Punkte, Linien etc.) extrahiert, die weiter verfolgt werden. Bei einer Bewegung des Benutzers und damit einer Bewegung der erkannten Merkmale innerhalb des Bildes kann die Positionsveränderung berechnet werden.
• Diese Positionsveränderungen werden an die "Vorverarbeitung light" 12 weitergegeben. Aus der vorherigen Position sowie der Positionsveränderung wird eine neue Position mit einer Zuverlässigkeit berechnet. Die Eingangsdaten der "Vorverarbeitung light" werden an eine optionale robuste Verarbeitungseinheit 16 weitergeleitet.
• Die robuste Verarbeitungseinheit 16 hat im Gegensatz zu der vorherigen Verarbeitungseinheit 12 einen wesentlich größeren Suchbereich, so dass eine verlässlichere Positionsbestimmung berechnet werden kann.
• Bei schnellen Bewegungen des Benutzers kann es vorkommen, dass die Einheiten unter 7 und 8 keine verlässlichen Werte geliefert können, da die Verschiebungen zu groß sind. Dieses wird durch die weiteren Sensoren 3, 4 kompensiert, da diese ihr Informationen in einer wesentlich größeren Frequenz bereitstellen können.
• Die Auswertung der verschiedenen Positionsberechnungen mit ihren Verlässlichkeiten erfolgt in der zentralen Verarbeitung 13. Sollte ein Sensor temporär keine verlässlichen Positionen liefern können, werden verlässlichere Sensoren höher bewertet.
• Sollten keine Position mehr durch die zentrale Verarbeitungseinheit 13 bereitgestellt werden können, wird automatisch eine Reinitialisierung 14 angestoßen, so dass wieder eine genaue Position des Benutzers dem System bekannt ist. Diese Reinitialisierung erfolgt möglichst ohne Benutzersinteraktion.
• Der Manager 17 bezieht die aktuelle Benutzerposition von der Verarbeitung 13.
• Die Visualisierung 18 oder 19 fordert die aktuelle Position des Benutzers über das Interface 9 vom Manager 17 an. Auf Basis dieser Position wird die Überlagerung des realen Blickfeldes mit virtuellen Informationen erreicht.

A scenario for a possible use of the system / framework is as follows:

• It is determined by, for example, an application developer, which sensors 1, 2, 3, 4 and which processing units 10, 11, 12, 13, 14, 15, 16 are to be used for an application. From this information, a configuration 20 is created which connects the sensors to the system and interconnects the algorithms. This configuration will be made known to the system.
• By starting the framework or the manager 17, the configuration is read in and the positioning system is set up according to the configuration. The individual processing units 10, 11, 12, 13, 14, 15, 16 are interconnected, so that a processing of the sensor data is made possible.
• A visualization 18 or 19 is started and the visualization establishes a connection to the manager 17 via the interface or the interface 9.
• By means of the initialization 14, the position of the user is determined at the beginning. Here, for example, a marker-based method can be used. But there are also other methods conceivable.
• After the initial position of the user is known, this is announced to the processing units.
• The connections of the sensors 5, 6, 7, 8 then deliver their data including synchronization data via the interface or the interface 9 to the processing units.
• From the video-based data 5, 6 are in the pre-processing 10, 11 features (eg points, lines, etc.) extracted, which are further tracked. When the user moves and thus moves the recognized features within the image, the change in position can be calculated.
• These position changes are forwarded to the "preprocessing light" 12. From the previous position and the position change, a new position is calculated with a reliability. The input data of the "pre-processing light" are forwarded to an optional robust processing unit 16.
• The robust processing unit 16, in contrast to the previous processing unit 12, has a much larger search area, so that a more reliable position determination can be calculated.
• In the case of fast movements of the user, the units under 7 and 8 can not deliver reliable values because the shifts are too large. This is compensated by the other sensors 3, 4, since they can provide their information in a much greater frequency.
• The evaluation of the various position calculations with their reliability takes place in the central processing 13. If a sensor temporarily can not deliver reliable positions, more reliable sensors are rated higher.
• If no more position can be provided by the central processing unit 13, reinitialization 14 is automatically triggered, so that again an exact position of the user is known to the system. This reinitialization is possible without user interaction.
• The manager 17 obtains the current user position from the processing 13.
• The visualization 18 or 19 requests the current position of the user via the interface 9 from the manager 17. Based on this position, the superposition of the real field of view is achieved with virtual information.

Claims (28)

System for determining the position of a user and / or a movable device by means of tracking methods, in particular for augmented reality applications an interface (9) for integrating at least one sensor type and / or data transmitter (1, 2, 3, 4) of a tracking method, a configuration unit (20) for describing communication between the tracking methods and / or tracking algorithms and at least one processing unit (5, 6, 7, 8, 10, 11, 12, 13, 16) for calculating the position of the user and / or the mobile device based on the data supplied by the tracking methods and / or tracking algorithms. System according to claim 1,
with means, in particular timestamps, for the synchronization of the data flows of the communication between the tracking methods and / or tracking algorithms. System according to one of the preceding claims,
in which the at least one processing unit (5, 6, 7, 8, 10, 11, 12, 13, 16) for calculating the angle of view of the user and / or the direction of movement of the movable device based on the supplied by the tracking method and / or tracking algorithms Data is provided. System according to one of the preceding claims,
in which sensor-type-specific processing units (5, 6, 7, 8) are provided for sensor-neutral integration of the data supplied by the sensors and / or data transmitters (1, 2, 3, 4) into the interface (9). System according to one of the preceding claims,
in which preprocessing units (10, 11) are provided for calculating the movement of the user and / or the moving device on the basis of video-based data, the calculation taking place on the basis of a tracking of the displacement of features in successive images. System according to one of the preceding claims,
with an initialization unit (14) for providing the position and / or the viewing angle of the user and / or the direction of movement of the movable device at the beginning of the use of the system. System according to claim 6,
in which the initialization unit (14) is provided for re-initializing and / or calibrating the system if the reliability of the position determination and / or the viewing angle and / or the direction of movement deteriorates. System according to one of the preceding claims,
in which a direct processing unit (12) is provided for calculating the position and / or the viewing angle of the user and / or the direction of movement of the movable device, the calculation being based on the results of the preprocessing units (10, 11) and / or of a position provided to a central processing unit (13). System according to one of the preceding claims,
in which an indirect processing unit (16) is provided for calculating the position and / or the angle of view of the user and / or the direction of movement of the movable device, the calculation taking place on the basis of algorithms for avoiding errors. System according to one of the preceding claims,
in which the central processing unit (13) is provided for merging the individual processing steps of the processing units (7, 8, 10, 11, 16), wherein means (22) are provided for calculating the reliability of the results. System according to one of the preceding claims,
with a radio link (21) for transmitting the data between the central processing unit (13) and / or the direct processing unit (12) and the indirect processing unit (16). System according to one of the preceding claims,
with a human movement model (15) and / or the mobile device for improving the calculation result. System according to one of the preceding claims,
with a manager (17) for receiving the calculated results of the central processing unit (13), wherein the manager (17) is provided for managing the results of the position and / or viewing angle and / or movement direction calculation. System according to one of the preceding claims,
with at least one visualization device (18, 19) for displaying virtual objects in a field of view of a user, wherein the manager (17) is provided for providing the results of the position and / or viewing angle calculation for the at least one visualization device. Method for determining the position of a user and / or a movable device by means of tracking methods, in particular for augmented reality applications, in which at least one sensor type and / or data transmitter (1, 2, 3, 4) of a tracking method is integrated via an interface (9), a communication between the tracking methods and / or tracking algorithms is described by means of a configuration unit (20) and a position of the user and / or the mobile device is calculated on the basis of the data supplied by the tracking methods and / or tracking algorithms by at least one processing unit (5, 6, 7, 8, 10, 11, 12, 13, 16). Method according to claim 15,
in which the data flows of the communication between the tracking methods and / or tracking algorithms are synchronized in particular with time stamps. Method according to one of claims 15 or 16,
wherein the angle of view of the user and / or the direction of movement of the mobile device is calculated on the basis of the data supplied by the tracking methods and / or tracking algorithms. Method according to one of claims 15 to 17,
in which sensor-neutral integration of the data supplied by the sensors and / or data transmitters (1, 2, 3, 4) into the interface (9) is effected by sensor-type-specific processing units (5, 6, 7, 8). Method according to one of claims 15 to 18,
wherein the calculation of the motion of the user and / or the moving device is based on video-based data due to a tracking of the displacement of features in successive images. Method according to one of claims 15 to 19,
in which the position and / or the angle of view of the user and / or the direction of movement of the mobile device is provided by an initialization unit (14) at the start of the use of the system. Method according to claim 20,
in which a re-initialization and / or calibration of the system by the initialization unit (14) takes place when the reliability of the position determination and / or the viewing angle and / or the direction of movement becomes worse. Method according to one of claims 15 to 21,
in which the position and / or the angle of view of the user and / or the direction of movement of the movable device are determined by a direct processing unit (12) on the basis of the results of the pre-processing units (10, 11) and / or the position provided by a central processing unit (13) ) is calculated. Method according to one of claims 15 to 22,
in which the position and / or the viewing angle of the user and / or the direction of movement of the movable device is calculated on the basis of algorithms for error prevention by an indirect processing unit (16). Method according to one of claims 15 to 23,
in which the individual processing steps of the processing units (7, 8, 10, 11, 16) are brought together by a central processing unit (13), wherein a calculation of the reliability of the results takes place. Method according to one of claims 15 to 24,
in which the data is transmitted between the central processing unit (13) and / or the direct processing unit (12) and the indirect processing unit (16) with a radio link (21). Method according to one of claims 15 to 25,
in which the calculation result is improved by means of a movement model (15) of the human and or the mobile device. Method according to one of claims 15 to 26,
in which the calculated results of the central processing unit (13) are recorded by a manager (17), wherein the manager (17) manages the results of the position and / or viewing angle and / or movement direction calculation. Method according to one of claims 15 to 27,
in which virtual objects are displayed in a field of view of a user by means of at least one visualization device (18, 19), wherein the results of the position and / or viewing angle calculation for the at least one visualization device are provided by the manager (17).

Images