US7545956B2 - Single camera system and method for range and lateral position measurement of a preceding vehicle - Google Patents

Single camera system and method for range and lateral position measurement of a preceding vehicle Download PDF

Info

Publication number
US7545956B2
US7545956B2 US11/202,917 US20291705A US7545956B2 US 7545956 B2 US7545956 B2 US 7545956B2 US 20291705 A US20291705 A US 20291705A US 7545956 B2 US7545956 B2 US 7545956B2
Authority
US
United States
Prior art keywords
window
camera
vehicle
electrical image
range
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Expired - Fee Related, expires
Application number
US11/202,917
Other versions
US20070035385A1 (en
Inventor
Shunji Miyahara
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Visteon Global Technologies Inc
Original Assignee
Visteon Global Technologies Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Visteon Global Technologies Inc filed Critical Visteon Global Technologies Inc
Priority to US11/202,917 priority Critical patent/US7545956B2/en
Assigned to VISTEON GLOBAL TECHNOLOGIES, INC. reassignment VISTEON GLOBAL TECHNOLOGIES, INC. ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: MIYAHARA, SHUNJI
Publication of US20070035385A1 publication Critical patent/US20070035385A1/en
Assigned to JPMORGAN CHASE BANK reassignment JPMORGAN CHASE BANK SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: VISTEON GLOBAL TECHNOLOGIES, INC.
Assigned to WILMINGTON TRUST FSB, AS ADMINISTRATIVE AGENT reassignment WILMINGTON TRUST FSB, AS ADMINISTRATIVE AGENT GRANT OF SECURITY INTEREST IN PATENT RIGHTS Assignors: VISTEON GLOBAL TECHNOLOGIES, INC.
Application granted granted Critical
Publication of US7545956B2 publication Critical patent/US7545956B2/en
Assigned to THE BANK OF NEW YORK MELLON, AS ADMINISTRATIVE AGENT reassignment THE BANK OF NEW YORK MELLON, AS ADMINISTRATIVE AGENT ASSIGNMENT OF PATENT SECURITY INTEREST Assignors: JPMORGAN CHASE BANK, N.A., A NATIONAL BANKING ASSOCIATION
Assigned to VISTEON GLOBAL TECHNOLOGIES, INC. reassignment VISTEON GLOBAL TECHNOLOGIES, INC. RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS RECORDED AT REEL 022732 FRAME 0263 Assignors: WILMINGTON TRUST FSB
Assigned to VISTEON GLOBAL TECHNOLOGIES, INC. reassignment VISTEON GLOBAL TECHNOLOGIES, INC. RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS RECORDED AT REEL 022974 FRAME 0057 Assignors: THE BANK OF NEW YORK MELLON
Assigned to MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT reassignment MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT SECURITY AGREEMENT Assignors: VC AVIATION SERVICES, LLC, VISTEON CORPORATION, VISTEON ELECTRONICS CORPORATION, VISTEON EUROPEAN HOLDING, INC., VISTEON GLOBAL TECHNOLOGIES, INC., VISTEON GLOBAL TREASURY, INC., VISTEON INTERNATIONAL BUSINESS DEVELOPMENT, INC., VISTEON INTERNATIONAL HOLDINGS, INC., VISTEON SYSTEMS, LLC
Assigned to MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT reassignment MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT SECURITY AGREEMENT (REVOLVER) Assignors: VC AVIATION SERVICES, LLC, VISTEON CORPORATION, VISTEON ELECTRONICS CORPORATION, VISTEON EUROPEAN HOLDINGS, INC., VISTEON GLOBAL TECHNOLOGIES, INC., VISTEON GLOBAL TREASURY, INC., VISTEON INTERNATIONAL BUSINESS DEVELOPMENT, INC., VISTEON INTERNATIONAL HOLDINGS, INC., VISTEON SYSTEMS, LLC
Assigned to VISTEON GLOBAL TECHNOLOGIES, INC., VISTEON INTERNATIONAL HOLDINGS, INC., VISTEON CORPORATION, VISTEON SYSTEMS, LLC, VISTEON ELECTRONICS CORPORATION, VC AVIATION SERVICES, LLC, VISTEON GLOBAL TREASURY, INC., VISTEON EUROPEAN HOLDING, INC., VISTEON INTERNATIONAL BUSINESS DEVELOPMENT, INC. reassignment VISTEON GLOBAL TECHNOLOGIES, INC. RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317 Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Assigned to CITIBANK., N.A., AS ADMINISTRATIVE AGENT reassignment CITIBANK., N.A., AS ADMINISTRATIVE AGENT SECURITY INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: VISTEON CORPORATION, AS GRANTOR, VISTEON GLOBAL TECHNOLOGIES, INC., AS GRANTOR
Assigned to VISTEON GLOBAL TECHNOLOGIES, INC., VISTEON GLOBAL TREASURY, INC., VISTEON EUROPEAN HOLDINGS, INC., VISTEON SYSTEMS, LLC, VC AVIATION SERVICES, LLC, VISTEON ELECTRONICS CORPORATION, VISTEON INTERNATIONAL HOLDINGS, INC., VISTEON CORPORATION, VISTEON INTERNATIONAL BUSINESS DEVELOPMENT, INC. reassignment VISTEON GLOBAL TECHNOLOGIES, INC. RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY Assignors: MORGAN STANLEY SENIOR FUNDING, INC.
Expired - Fee Related legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S11/00—Systems for determining distance or velocity not using reflection or reradiation
    • G01S11/12—Systems for determining distance or velocity not using reflection or reradiation using electromagnetic waves other than radio waves
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/86—Combinations of sonar systems with lidar systems; Combinations of sonar systems with systems not using wave reflection
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S15/00—Systems using the reflection or reradiation of acoustic waves, e.g. sonar systems
    • G01S15/88—Sonar systems specially adapted for specific applications
    • G01S15/93—Sonar systems specially adapted for specific applications for anti-collision purposes
    • G01S15/931—Sonar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems
    • G08G1/166—Anti-collision systems for active traffic, e.g. moving vehicles, pedestrians, bikes
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/16—Anti-collision systems

Definitions

  • the present invention generally relates to a system and method for range and lateral position measurement of a preceding vehicle.
  • Radar and stereo camera systems for adaptive cruise control have been already introduced into the market. Recently, radar has been applied to for pre-crash safety systems and collision avoidance. Typically, the range and lateral position measurement of a preceding vehicle is accomplished utilizing radar and/or stereo camera systems. Radar systems can provide a very accurate range. However, millimeter wave type radar systems such as 77 Ghz systems are typically quite expensive. Laser radar is low cost but has the drawback of requiring moving parts which scan the laser across a field of view. Furthermore, the laser is less effective in adverse weather conditions which may effect the reflectivity of the preceding vehicle. For example, if mud covers the reflectors of the preceding vehicle, the reflectivity of the preceding vehicle will be less which minimizes the effectiveness of the laser. Finally, radar generally is not well suited to identify the object and give an accurate lateral position.
  • Stereo cameras can determine the range and identity of an object.
  • these systems are typically difficult to manufacture due to the accurate alignment required between the two stereo cameras and requires two image processors.
  • the present invention provides a system for determining range and lateral position of a vehicle.
  • the primary components of the system include a single camera, at least one sonar sensor and a processor.
  • the camera is configured to view a first region of interest containing a preceding vehicle and generate an electrical image of the region.
  • the processor is in electrical communication with the camera to receive the electrical image.
  • the processor identifies a series of windows within the image, each window corresponding to a fixed physical size at a different target range. For example, from the perspective of the camera the vehicle will appear larger when it is closer to the camera than if it is further away from the camera.
  • each window is sized proportionally in the image as it would appear to the camera at each target range.
  • the processor evaluates characteristics of the electrical image within each window to identify the vehicle. For example, the size of the vehicle is compared to the size of the window to create a size ratio. A score is determined indicating the likelihood that certain characteristics of the electrical image actually correspond to the vehicle and also that the vehicle is at the target range for that window.
  • the sonar sensor is configured to view a second region of interest.
  • the processor receives data indicating if the preceding vehicle is in the second field of view. If the processor determines that the preceding vehicle is within the second field of view, the processor will calculate the range of the preceding vehicle based on the data generated by the sonar sensor.
  • the characteristics of electrical image evaluated by the processor includes the width and height of edge segments in the image, as well as, the height, width, and location of objects constructed from multiple edge segments.
  • the position of the window in the electrical image is calculated based on the azimuth angle and the elevation angle of the camera.
  • a method for identifying the vehicle within the electrical image and a sonar signal to determine the vehicle range.
  • an edge enhanced algorithm is applied to the image. Only characteristics of the electrical image within a particular window are evaluated. The edge enhanced image is trinalized and segmented. The segments are evaluated and objects are constructed from multiple segments. A score is determined for each object based on criteria, such as, the object width, object height position, object height, and segment width. Based on the width ratio, the range and lateral position of the object are estimated on the basis of the target range of the window. However, if the sonar signal indicates that the object is closer than estimated by evaluating the electrical image, the sonar signal will be used to estimate the range of the object.
  • FIG. 1 is a side view of a system for range and lateral position measurement of a preceding vehicle, embodying the principles of the present invention
  • FIG. 2 is a view of an electronic image from the perspective of the camera in FIG. 1 ;
  • FIG. 3 is a side view of the system illustrating the calculation of the upper and lower edge of the windows in accordance with the present invention
  • FIG. 4 is a top view of the system illustrating the calculation of the left and right edge of the windows, in accordance with the present invention
  • FIG. 5A is a view of the electronic image, with only the image information in the first window extracted;
  • FIG. 5B is a view of the electronic image, with only the image information in the second window extracted;
  • FIG. 5C is a view of the electronic image, with only the image information in the third window extracted;
  • FIG. 6 is a view of an electronic image generated by the camera prior to processing
  • FIG. 7 is a view of the electronic image after a vertical edge enhancement algorithm has been applied to the electronic image shown in FIG. 6 ;
  • FIG. 8 is a view of the electronic image including segments that are extracted from the edge enhanced image
  • FIG. 9 is a view of the electronic image including objects constructed from the segments illustrated in FIG. 8 ;
  • FIG. 10 is a side view of the system for range and lateral position measurement of a large preceding vehicle, embodying the principles of the present invention.
  • FIG. 11 is a view of the electronic image generated by the camera prior to processing.
  • FIG. 12 is a view of the electronic image after the vertical edge enhancement algorithm has been applied the electronic image shown in FIG. 11 .
  • the system 10 includes a camera 12 , a sonar sensor 15 and a processor 14 .
  • the camera 12 is located in the rearview mirror to collect an optical image of a first region of interest 16 including a vehicle 18 .
  • the sonar sensor 15 is located near the front bumper of the vehicle to collect data of a second region of interest 17 .
  • the first and second regions of interest 16 , 17 partially overlap.
  • the optical image received by the camera 12 is converted to an electrical image that is provided to the processor 14 .
  • the processor 14 calculates the position of multiple windows 20 , 22 , 24 within the first region of interest 16 .
  • the windows 20 , 22 , 24 are located at varying target ranges from the camera 12 .
  • the size of the windows 20 , 22 , 24 are a predetermined physical size (about 4 ā‡ 2 m as shown) and may correspond to the size of a typical vehicle. To provide increased resolution the windows 20 , 22 , 24 are spaced closer together and the number of windows is increased.
  • the system 10 is configured to track a vehicle 18 preceding the system 10 , it is fully contemplated that the camera 12 could be directed to the side or rear the system 10 to track a vehicle 18 that may be approaching from other directions.
  • FIG. 2 an electronic image of the first region of interest 16 as viewed by the camera 12 is provided.
  • the windows 20 , 22 , 24 are projected into their corresponding size and location according to the perspective of the camera 12 .
  • the vehicle 18 is located between windows 22 and 24 , accordingly, the size of the vehicle 18 corresponds much more closely to the height and width of windows 22 and 24 than window 20 .
  • the window sizes appear to vary from the perspective of the camera 12 .
  • the height and width of the preceding vehicle 18 will appear to vary at each target range.
  • the perspective of the camera 12 will affect the apparent size and location of the preceding vehicle 18 within the electrical image based on the elevation angle and the azimuth angle of the camera 12 .
  • the processor 14 can use the location and size of each of the windows 20 , 22 , 24 to evaluate characteristics of the electrical image and determine a score indicating the probability the vehicle 18 is at the target range associated with a particular window.
  • a side view of the system 10 is provided illustrating the use of the elevation angle in calculating the height and position of the window 20 within the electrical image.
  • the elevation angle is the angle between the optical axis of the camera 12 and the surface of the road.
  • the lower edge of window 20 is calculated based on Equation (1).
  • ā‡ 1 a tan( ā‡ r 1/ hc ) (1)
  • hc is the height of the camera 12 from the road surface
  • r 1 is the horizontal range of window 20 from the camera 12
  • the module is [0, ā‡ ].
  • Equation (2) the upper edge of the first window is calculated based on Equation (2).
  • ā‡ 1h a tan( rc/ ( hw ā‡ hc )) (2)
  • hw is the height of the window
  • hc is the height of the camera 12 from the road surface
  • r 1 is the range of window 20 from the camera 12
  • the module is [0, ā‡ ].
  • the horizontal position of the window in a picture corresponds to the azimuth angle.
  • the azimuth angle is the angle across the width of the preceding vehicle from the perspective of the camera 12 .
  • the right edge of the range window 20 is calculated according to Equation (3).
  • ā‡ 1 a tan( ā‡ width 13 w/ (2* r 1))+( ā‡ /2) (3)
  • ā‡ 1h a tan(width 13 w/ (2* r 1))+( ā‡ /2) (4)
  • window w is the distance from the center of the window 20 to the horizontal edges
  • r 1 is the horizontal range of the window 20 from the camera 12
  • the module is [ ā‡ /2, ā‡ /2].
  • the window positions for the additional windows 22 , 24 are calculated according to Equations (1)-(4), substituting their respective target ranges for r 1 .
  • the electronic image is shown relative to window 20 .
  • the width of the object 26 is about 30% of the width of the window 20 .
  • the processor 14 evaluates vertical offset and object height criteria. For example, the distance of the object 26 from the bottom of the processing window 20 is used in determining likelihood that the object 26 is at the target range.
  • the extracted object 26 should have a height of 0.5 m or 1.2 m.
  • the processor 14 will detect an object height of 0.5 m if only the bottom portion of the vehicle 18 is detected or 1.2 m if the full height of the vehicle 18 is detected. The closer the height of the object 26 is to the expected height the more probable the object 26 is the vehicle 18 and the more probable it is located at the target range r 1 .
  • the vertical offset may also affect the height of the object 26 , as the top of the object, in FIG. 5A , is chopped off by the edge of the window 20 . Therefore, the object 26 appears shorter than expected, again lowering the likelihood the object is the vehicle 18 at the range r 1 .
  • the electronic image is shown relative to window 22 .
  • the width of the object 27 is about 45% of the window 22 . Therefore, the estimated width of the object 27 at range r 2 is equal to 4 ā‡ 0.45 ā‡ 1.8 m much closer to the expected size of the vehicle 18 .
  • the object 27 is only slightly offset from the bottom of the window 22 , and the entire height of the object 27 is still included in the window 22 .
  • the electronic image is shown relative to window 24 .
  • the width of the object 28 is about 80% of the width of the window 24 .
  • the processor 14 can make a determination that object 27 most probably corresponds to vehicle 18 and r 2 is the most probable range.
  • the range accuracy of the system 10 can be increased by using a finer pitch of target range for each window. Using a finer pitch between windows is especially useful as the vehicle 18 is closer to the camera 12 , due to the increased risk of collision.
  • FIG. 6 a typical electronic image as seen by the camera 12 is provided and will be used to further describe the method implemented by the processor 14 to determine the range and lateral position of the vehicle 18 .
  • the electronic image includes additional features that could be confusing for the processor 14 such as the lane markings 30 , an additional car 32 , and a motorcycle 34 .
  • FIG. 7 shows a vertically edge enhanced image.
  • the electronic image is comprised of horizontal rows and vertical columns of picture elements (pixels). Each pixel contains a value corresponding to the brightness of the image at that row and column location.
  • a typical edge enhancement algorithm includes calculating the derivative of the brightness across the horizontal rows or vertical columns of the image. However, many other edge enhancement techniques are contemplated and may be readily used.
  • the position and size of the window 36 is calculated for a given target range. Edge information located outside the window 36 is ignored. In this instance, much of the edge enhanced information from the car 38 and the motorcycle 40 can be eliminated.
  • the edge enhanced image is then trinarized, meaning each of the pixels are set to a value of ā‡ 1, +1, or 0.
  • a typical method for trinarizing the image includes taking the value of each pixel value and applying an upper and lower threshold value, where if the brightness of the pixel value is above the upper threshold value, the pixel value is set to 1. If the brightness of the pixel value is below the lower threshold value, the pixel value is set to ā‡ 1. Otherwise, the pixel value is set to 0. This effectively separates the pixels into edge pixels with a bright to dark (negative) transition, edge pixels with a dark to bright (positive) transition, and non-edge pixels.
  • the above described method is fast and simple, other more complicated thresholding methods may be used including local area thresholding or other commonly used approaches.
  • the pixels are grouped based on their relative position to other pixels having the same value. Grouping of these pixels is called segmentation and each of the groups is referred to as a line-segment. Height, width and position information is stored for each line-segment.
  • Segment 42 and 43 represent the lane marking on the road.
  • Segment 44 represents the upper portion of the left side of the vehicle.
  • Segment 46 represents the lower left side of the vehicle.
  • Segment 48 represents the left tire of the vehicle.
  • Segment 50 represents the upper right side of the vehicle.
  • Segment 52 represents the lower right side of the vehicle while segment 54 represents the right tire.
  • objects may be constructed from two segments having different polarity. Segment 42 and segment 44 are combined to construct object 56 . Segment 42 and segment 46 are combined to construct object 58 . In segment 46 and segment 52 are combined to construct object 60 .
  • Each of the objects are then scored based on the width of the object, the height of the object, the position of the object relative to the bottom edge of the window, the segment width, and the segment height. The above process is repeated for multiple windows with different target ranges. The object with the best score is compared with a minimum score threshold. If the best score is higher than the minimum score threshold the characteristics of the object are used to determine the object's range and lateral position.
  • FIG. 10 a sideview of the system 10 is provided illustrating the use of the sonar sensor 15 in estimating the range of a large vehicle 19 , such as a semi truck having a trailer. In certain low speed situations, such as a traffic backup, it is preferable to follow the preceding vehicle at a close distance. As shown in FIG. 10 , the larger vehicle 19 is within the first and second regions of interest 16 , 17 .
  • FIG. 11 a typical electronic image as seen by the camera 12 is provided and will be used to further describe the method implemented by the processor 14 to determine the range and lateral position of the larger vehicle 19 .
  • the processor 14 determines the range and lateral position of the larger vehicle 19 .
  • the electronic image as seen in FIG. 11 will be vertically edge enhanced and then trinalized as described previously.
  • a trinalized image of the electronic image is shown. However, because of the size of the larger vehicle 19 and the closeness of the larger vehicle 19 to the camera 12 , only the edge of the larger vehicle, as represented by segment 64 , is shown. The processor 14 will be unable to determine if the edge 64 represents a vehicle, such as the larger vehicle 19 , a lane marker, a tree or other object.
  • the processor 14 will receive data from the sonar sensor 15 .
  • the sonar sensor 15 has a shorter range than the camera 12 as illustrated by the second region of interest 17 .
  • the inherent characteristics of sonar sensor 15 which are well known in the art, allow the sonar sensor 15 to detect large objects, such as the larger vehicle 19 , that would otherwise be undetectable to the processed image captured by camera 12 . If the processor 14 received data from the sonar sensor 15 indicating that a vehicle is present and undetected by the camera 12 in the first region of interest, the processor 14 will use the data generated by the sonar sensor 15 to calculate the range of the larger vehicle 19 .

Abstract

A system for determining range and lateral position of a vehicle is provided. The system includes a camera, a sonar and a processor. The camera is configured to view a long range region of interest and generate an electrical image of the region. The sonar is configured to view a short range region of interest an output a sonar signal. The processor is in electrical communication with the camera and the sonar to receive the electrical image and the sonar signal. The processor analyzes the image and the sonar signal in order to determine the range of an object.

Description

BACKGROUND
1. Field of the Invention
The present invention generally relates to a system and method for range and lateral position measurement of a preceding vehicle.
2. Description of Related Art
Radar and stereo camera systems for adaptive cruise control (ACC), have been already introduced into the market. Recently, radar has been applied to for pre-crash safety systems and collision avoidance. Typically, the range and lateral position measurement of a preceding vehicle is accomplished utilizing radar and/or stereo camera systems. Radar systems can provide a very accurate range. However, millimeter wave type radar systems such as 77 Ghz systems are typically quite expensive. Laser radar is low cost but has the drawback of requiring moving parts which scan the laser across a field of view. Furthermore, the laser is less effective in adverse weather conditions which may effect the reflectivity of the preceding vehicle. For example, if mud covers the reflectors of the preceding vehicle, the reflectivity of the preceding vehicle will be less which minimizes the effectiveness of the laser. Finally, radar generally is not well suited to identify the object and give an accurate lateral position.
Stereo cameras can determine the range and identity of an object. However, these systems are typically difficult to manufacture due to the accurate alignment required between the two stereo cameras and requires two image processors.
In view of the above, it can be seen that conventional ACC systems typically do not have a high cost-performance ratio even though they may perform to the desired functional requirements. Further, it is apparent that there exists a need for an improved system and method for measuring the range and lateral position of the preceding vehicle.
SUMMARY
In satisfying the above need, as well as overcoming the enumerated drawbacks and other limitations of the related art, the present invention provides a system for determining range and lateral position of a vehicle. The primary components of the system include a single camera, at least one sonar sensor and a processor. The camera is configured to view a first region of interest containing a preceding vehicle and generate an electrical image of the region. The processor is in electrical communication with the camera to receive the electrical image. To analyze the electrical image, the processor identifies a series of windows within the image, each window corresponding to a fixed physical size at a different target range. For example, from the perspective of the camera the vehicle will appear larger when it is closer to the camera than if it is further away from the camera. Accordingly, each window is sized proportionally in the image as it would appear to the camera at each target range. The processor evaluates characteristics of the electrical image within each window to identify the vehicle. For example, the size of the vehicle is compared to the size of the window to create a size ratio. A score is determined indicating the likelihood that certain characteristics of the electrical image actually correspond to the vehicle and also that the vehicle is at the target range for that window.
The sonar sensor is configured to view a second region of interest. The processor receives data indicating if the preceding vehicle is in the second field of view. If the processor determines that the preceding vehicle is within the second field of view, the processor will calculate the range of the preceding vehicle based on the data generated by the sonar sensor.
In another aspect of the present invention, the characteristics of electrical image evaluated by the processor includes the width and height of edge segments in the image, as well as, the height, width, and location of objects constructed from multiple edge segments. The position of the window in the electrical image is calculated based on the azimuth angle and the elevation angle of the camera.
In yet another aspect of the present invention, a method is provided for identifying the vehicle within the electrical image and a sonar signal to determine the vehicle range. To simplify the image, an edge enhanced algorithm is applied to the image. Only characteristics of the electrical image within a particular window are evaluated. The edge enhanced image is trinalized and segmented. The segments are evaluated and objects are constructed from multiple segments. A score is determined for each object based on criteria, such as, the object width, object height position, object height, and segment width. Based on the width ratio, the range and lateral position of the object are estimated on the basis of the target range of the window. However, if the sonar signal indicates that the object is closer than estimated by evaluating the electrical image, the sonar signal will be used to estimate the range of the object.
Further objects, features and advantages of this invention will become readily apparent to persons skilled in the art after a review of the following description, with reference to the drawings and claims that are appended to and form a part of this specification.
BRIEF DESCRIPTION OF THE DRAWINGS
FIG. 1 is a side view of a system for range and lateral position measurement of a preceding vehicle, embodying the principles of the present invention;
FIG. 2 is a view of an electronic image from the perspective of the camera in FIG. 1;
FIG. 3 is a side view of the system illustrating the calculation of the upper and lower edge of the windows in accordance with the present invention;
FIG. 4 is a top view of the system illustrating the calculation of the left and right edge of the windows, in accordance with the present invention;
FIG. 5A is a view of the electronic image, with only the image information in the first window extracted;
FIG. 5B is a view of the electronic image, with only the image information in the second window extracted;
FIG. 5C is a view of the electronic image, with only the image information in the third window extracted;
FIG. 6 is a view of an electronic image generated by the camera prior to processing;
FIG. 7 is a view of the electronic image after a vertical edge enhancement algorithm has been applied to the electronic image shown in FIG. 6;
FIG. 8 is a view of the electronic image including segments that are extracted from the edge enhanced image;
FIG. 9 is a view of the electronic image including objects constructed from the segments illustrated in FIG. 8;
FIG. 10 is a side view of the system for range and lateral position measurement of a large preceding vehicle, embodying the principles of the present invention;
FIG. 11 is a view of the electronic image generated by the camera prior to processing; and
FIG. 12 is a view of the electronic image after the vertical edge enhancement algorithm has been applied the electronic image shown in FIG. 11.
 DETAILED DESCRIPTION
Referring now to FIG. 1, a system embodying the principles of the present invention is illustrated therein and designated at 10. As its primary components, the system 10 includes a camera 12, a sonar sensor 15 and a processor 14. The camera 12 is located in the rearview mirror to collect an optical image of a first region of interest 16 including a vehicle 18. The sonar sensor 15 is located near the front bumper of the vehicle to collect data of a second region of interest 17. Typically, the first and second regions of interest 16, 17 partially overlap.
The optical image received by the camera 12, is converted to an electrical image that is provided to the processor 14. To filter out unwanted distractions in the electronic image and aid in determining the range of the vehicle 18, the processor 14 calculates the position of multiple windows 20, 22, 24 within the first region of interest 16. The windows 20, 22, 24 are located at varying target ranges from the camera 12. The size of the windows 20, 22, 24 are a predetermined physical size (about 4Ɨ2 m as shown) and may correspond to the size of a typical vehicle. To provide increased resolution the windows 20, 22, 24 are spaced closer together and the number of windows is increased. Although the system 10, as shown, is configured to track a vehicle 18 preceding the system 10, it is fully contemplated that the camera 12 could be directed to the side or rear the system 10 to track a vehicle 18 that may be approaching from other directions.
Now referring to FIG. 2, an electronic image of the first region of interest 16 as viewed by the camera 12 is provided. The windows 20, 22, 24 are projected into their corresponding size and location according to the perspective of the camera 12. The vehicle 18 is located between windows 22 and 24, accordingly, the size of the vehicle 18 corresponds much more closely to the height and width of windows 22 and 24 than window 20. As can be seen from FIG. 1, although the size and width of the windows are physically constant at each target range, the window sizes appear to vary from the perspective of the camera 12. Similarly, the height and width of the preceding vehicle 18 will appear to vary at each target range. The perspective of the camera 12 will affect the apparent size and location of the preceding vehicle 18 within the electrical image based on the elevation angle and the azimuth angle of the camera 12. The processor 14 can use the location and size of each of the windows 20, 22, 24 to evaluate characteristics of the electrical image and determine a score indicating the probability the vehicle 18 is at the target range associated with a particular window.
Now referring to FIG. 3, a side view of the system 10 is provided illustrating the use of the elevation angle in calculating the height and position of the window 20 within the electrical image. The elevation angle is the angle between the optical axis of the camera 12 and the surface of the road. The lower edge of window 20 is calculated based on Equation (1).
Īø1 =a tan(āˆ’r1/hc)ā€ƒā€ƒ(1)
Where hc is the height of the camera 12 from the road surface, r1 is the horizontal range of window 20 from the camera 12, and the module is [0, Ļ€].
Similarly, the upper edge of the first window is calculated based on Equation (2).
Īø1h =a tan(rc/(hwāˆ’hc))ā€ƒā€ƒ(2)
Where hw is the height of the window, hc is the height of the camera 12 from the road surface, r1 is the range of window 20 from the camera 12, and the module is [0, Ļ€]. The difference, Ī”Īø1=Īø1āˆ’Īø1h, corresponds to the height of the window in the electronic image.
Now referring to FIG. 4, the horizontal position of the window in a picture corresponds to the azimuth angle. The azimuth angle is the angle across the width of the preceding vehicle from the perspective of the camera 12. The right edge of the range window 20 is calculated according to Equation (3).
Ļ†1 =a tan(āˆ’width13 w/(2*r1))+(Ļ€/2)ā€ƒā€ƒ(3)
Similarly, the left edge of the range window 20 is calculated according to Equation (4).
Ļ†1h =a tan(width13 w/(2*r1))+(Ļ€/2)ā€ƒā€ƒ(4)
Where window w is the distance from the center of the window 20 to the horizontal edges, r1 is the horizontal range of the window 20 from the camera 12, and the module is [āˆ’Ļ€/2, Ļ€/2].
The window positions for the additional windows 22, 24 are calculated according to Equations (1)-(4), substituting their respective target ranges for r1.
Now referring to FIG. 5A, the electronic image is shown relative to window 20. Notice the width of the object 26 is about 30% of the width of the window 20. If the window width is set at a width of 4 m, about twice the expected width of the vehicle 18, the estimated width of the object 26 at a distance of r1 would equal 4Ɨ0.3=1.2 m. Therefore, the likelihood that the object 26 is the vehicle 18 at range r1 is low. In addition, the processor 14 evaluates vertical offset and object height criteria. For example, the distance of the object 26 from the bottom of the processing window 20 is used in determining likelihood that the object 26 is at the target range. Assuming a flat road, if the object 26 were at the range r1, the lowest feature of the object 26 would appear at the bottom of the window 20 corresponding to being in contact with the road at the target range. However, the object 26 in FIG. 5A, appears to float above the road, thereby decreasing the likelihood it is located at the target range. Further, the extracted object 26 should have a height of 0.5 m or 1.2 m. The processor 14 will detect an object height of 0.5 m if only the bottom portion of the vehicle 18 is detected or 1.2 m if the full height of the vehicle 18 is detected. The closer the height of the object 26 is to the expected height the more probable the object 26 is the vehicle 18 and the more probable it is located at the target range r1. The vertical offset, described above, may also affect the height of the object 26, as the top of the object, in FIG. 5A, is chopped off by the edge of the window 20. Therefore, the object 26 appears shorter than expected, again lowering the likelihood the object is the vehicle 18 at the range r1.
Now referring to FIG. 5B, the electronic image is shown relative to window 22. The width of the object 27 is about 45% of the window 22. Therefore, the estimated width of the object 27 at range r2 is equal to 4Ɨ0.45āˆ’1.8 m much closer to the expected size of the vehicle 18. In this image, the object 27 is only slightly offset from the bottom of the window 22, and the entire height of the object 27 is still included in the window 22.
Now referring to FIG. 5C, the electronic image is shown relative to window 24. The width of the object 28 is about 80% of the width of the window 24. Accordingly, the estimated width of the object 28 at range r3 is equal to 4Ɨ0.08=3.2 m. Therefore, the object width is significantly larger than the expected width of vehicle 18, usually about 1.75 m. Based on the object width, the processor 14 can make a determination that object 27 most probably corresponds to vehicle 18 and r2 is the most probable range. The range accuracy of the system 10 can be increased by using a finer pitch of target range for each window. Using a finer pitch between windows is especially useful as the vehicle 18 is closer to the camera 12, due to the increased risk of collision.
Now referring to FIG. 6, a typical electronic image as seen by the camera 12 is provided and will be used to further describe the method implemented by the processor 14 to determine the range and lateral position of the vehicle 18. The electronic image includes additional features that could be confusing for the processor 14 such as the lane markings 30, an additional car 32, and a motorcycle 34.
FIG. 7 shows a vertically edge enhanced image. The electronic image is comprised of horizontal rows and vertical columns of picture elements (pixels). Each pixel contains a value corresponding to the brightness of the image at that row and column location. A typical edge enhancement algorithm includes calculating the derivative of the brightness across the horizontal rows or vertical columns of the image. However, many other edge enhancement techniques are contemplated and may be readily used. In addition, the position and size of the window 36 is calculated for a given target range. Edge information located outside the window 36 is ignored. In this instance, much of the edge enhanced information from the car 38 and the motorcycle 40 can be eliminated.
Now referring to FIG. 9, the edge enhanced image is then trinarized, meaning each of the pixels are set to a value of āˆ’1, +1, or 0. A typical method for trinarizing the image includes taking the value of each pixel value and applying an upper and lower threshold value, where if the brightness of the pixel value is above the upper threshold value, the pixel value is set to 1. If the brightness of the pixel value is below the lower threshold value, the pixel value is set to āˆ’1. Otherwise, the pixel value is set to 0. This effectively separates the pixels into edge pixels with a bright to dark (negative) transition, edge pixels with a dark to bright (positive) transition, and non-edge pixels. Although, the above described method is fast and simple, other more complicated thresholding methods may be used including local area thresholding or other commonly used approaches. Next, the pixels are grouped based on their relative position to other pixels having the same value. Grouping of these pixels is called segmentation and each of the groups is referred to as a line-segment. Height, width and position information is stored for each line-segment.
Relating these segments back to the original image, Segment 42 and 43 represent the lane marking on the road. Segment 44 represents the upper portion of the left side of the vehicle. Segment 46 represents the lower left side of the vehicle. Segment 48 represents the left tire of the vehicle. Segment 50 represents the upper right side of the vehicle. Segment 52 represents the lower right side of the vehicle while segment 54 represents the right tire.
Still referring to FIG. 9, objects may be constructed from two segments having different polarity. Segment 42 and segment 44 are combined to construct object 56. Segment 42 and segment 46 are combined to construct object 58. In segment 46 and segment 52 are combined to construct object 60. Each of the objects are then scored based on the width of the object, the height of the object, the position of the object relative to the bottom edge of the window, the segment width, and the segment height. The above process is repeated for multiple windows with different target ranges. The object with the best score is compared with a minimum score threshold. If the best score is higher than the minimum score threshold the characteristics of the object are used to determine the object's range and lateral position.
Now referring to FIG. 10, a sideview of the system 10 is provided illustrating the use of the sonar sensor 15 in estimating the range of a large vehicle 19, such as a semi truck having a trailer. In certain low speed situations, such as a traffic backup, it is preferable to follow the preceding vehicle at a close distance. As shown in FIG. 10, the larger vehicle 19 is within the first and second regions of interest 16, 17.
Now referring to FIG. 11, a typical electronic image as seen by the camera 12 is provided and will be used to further describe the method implemented by the processor 14 to determine the range and lateral position of the larger vehicle 19. When following closely to a larger vehicle 19, only a portion of the larger vehicle 19 will be within view of the camera 12. The electronic image as seen in FIG. 11 will be vertically edge enhanced and then trinalized as described previously.
Referring to FIG. 12, a trinalized image of the electronic image is shown. However, because of the size of the larger vehicle 19 and the closeness of the larger vehicle 19 to the camera 12, only the edge of the larger vehicle, as represented by segment 64, is shown. The processor 14 will be unable to determine if the edge 64 represents a vehicle, such as the larger vehicle 19, a lane marker, a tree or other object.
In order to take into account situations as described above, the processor 14 will receive data from the sonar sensor 15. The sonar sensor 15 has a shorter range than the camera 12 as illustrated by the second region of interest 17. The inherent characteristics of sonar sensor 15, which are well known in the art, allow the sonar sensor 15 to detect large objects, such as the larger vehicle 19, that would otherwise be undetectable to the processed image captured by camera 12. If the processor 14 received data from the sonar sensor 15 indicating that a vehicle is present and undetected by the camera 12 in the first region of interest, the processor 14 will use the data generated by the sonar sensor 15 to calculate the range of the larger vehicle 19.
As a person skilled in the art will readily appreciate, the above description is meant as an illustration of implementation of the principles this invention. This description is not intended to limit the scope or application of this invention in that the invention is susceptible to modification, variation and change, without departing from spirit of this invention, as defined in the following claims.

Claims (44)

1. A system for determining range of a vehicle, the system comprising:
a camera configured to view a first region of interest including the vehicle and generate an electrical image of the region;
a sonar sensor configured to detect objects within a second region of interest;
processor in electrical communication with the camera to receive the electrical image, wherein the processor is configured to identify a plurality of windows within the electrical image, each window of the plurality of windows corresponding to a predetermined physical size and target range from the camera, the processor being further configured to evaluate characteristics of a candidate object in relation to each window to identify the vehicle;
the processor in electrical communication with the sonar sensor, the processor configured to identify objects detected by the sonar sensor and determine the range of the objects detected by the sonar sensor;
wherein the window position is determined based on an elevation angle of the camera; and
wherein the lower edge of each window is calculated based on the relationship Īø1=a tan(āˆ’rc/hc) where hc is the height of the camera from the road surface, rc is the range of window from the camera, and the module is [0, Ļ€].
2. The system according to claim 1, wherein the processor is configured to determine a score indicating likelihood the candidate object is the target.
3. The system according to claim 2, wherein the processor is configured to determine the score based on a size ratio between each window and the characteristics of the candidate object.
4. The system according to claim 1, wherein the predetermined physical size is based on vehicle characteristics.
5. The system according to claim 4, wherein the vehicle characteristics include a width of the vehicle.
6. The system according to claim 4, wherein the vehicle characteristics include a height of the vehicle.
7. The system according to claim 1, wherein the characteristics of the electrical image include edge segments within the electrical image.
8. The system according to claim 7, wherein the edge segments are vertical edge segments.
9. The system according to claim 7, wherein the characteristics of the electrical image include a height of the edge segments.
10. The system according to claim 7, wherein the characteristics of the electrical image include a width of the edge segments.
11. The system according to claim 7, wherein the characteristics of the electrical image include objects constructed from the edge segments.
12. The system according to claim 11, wherein the characteristics of the electrical image include a height of the objects.
13. The system according to claim 11, wherein the characteristics of the electrical image include a width of the objects.
14. The system according to claim 1, wherein a position of each window is determined based on the target range.
15. A system for determining range of a vehicle, the system comprising:
a camera configured to view a first region of interest including the vehicle and generate an electrical image of the region;
a sonar sensor configured to detect objects within a second region of interest;
a processor in electrical communication with the camera to receive the electrical image, wherein the processor is configured to identify a plurality of windows within the electrical image, each window of the plurality of windows corresponding to a predetermined physical size and target range from the camera, the processor being further configured to evaluate characteristics of a candidate object in relation to each window to identify the vehicle;
the processor in electrical communication with the sonar sensor, the processor configured to identify objects detected by the sonar sensor and determine the range of the objects detected by the sonar sensor;
wherein the window position is determined based on an elevation angle of the camera; and
wherein the upper edge of each window is calculated based on the relationship Īø1h=a tan(r1/(hwāˆ’hc)) where hw is a window height, hc is the height of the camera from the road surface, r1 is the range of window from the camera, and the module is [0, Ļ€].
16. A system for determining range of a vehicle, the system comprising:
a camera configured to view a first region of interest including the vehicle and generate an electrical image of the region;
a sonar sensor configured to detect objects within a second region of interest;
a processor in electrical communication with the camera to receive the electrical image, wherein the processor is configured to identify a plurality of windows within the electrical image, each window of the plurality of windows corresponding to a predetermined physical size and target range from the camera, the processor being further configured to evaluate characteristics of a candidate object in relation to each window to identify the vehicle;
the processor in electrical communication with the sonar sensor, the processor configured to identify objects detected by the sonar sensor and determine the range of the objects detected by the sonar sensor;
wherein the position of each window is determined based on an azimuth angle of the camera; and
wherein the right edge of each window is calculated based on the relationship Ļ†1=a tan(āˆ’width_w/(2r1))+Ļ€/2 where window w is the distance from the center of the window to the horizontal edges, r1 is the horizontal range of the window from the camera, and the module is [āˆ’Ļ€/2,Ļ€/2].
17. A system for determining range of a vehicle, the system comprising:
a camera configured to view a first region of interest including the vehicle and generate an electrical image of the region;
a sonar sensor configured to detect objects within a second region of interest;
a processor in electrical communication with the camera to receive the electrical image, wherein the processor is configured to identify a plurality of windows within the electrical image, each window of the plurality of windows corresponding to a predetermined physical size and target range from the camera, the processor being further configured to evaluate characteristics of a candidate object in relation to each window to identify the vehicle;
the processor in electrical communication with the sonar sensor, the processor configured to identify objects detected by the sonar sensor and determine the range of the objects detected by the sonar sensor;
wherein the position of each window is determined based on an azimuth angle of the camera; and
wherein the left edge of each window is calculated based on the relationship Ļ†1h=a tan(width_w/r1)+Ļ€/2 where window w is the distance from the center of the window to the horizontal edges, r1 is the horizontal range of the window from the camera, and the module is [āˆ’Ļ€/2,Ļ€/2].
18. A method for determining range of a vehicle, the method comprising:
receiving an optical image of a first region of interest onto a camera;
receiving a sonar signal of a second region of interest onto a sonar;
generating an electrical image based on the optical image;
identifying a plurality of windows within the electrical image, each window corresponding to a predetermined physical size and a target range from the camera;
evaluating characteristics of a candidate object within each window signal to identify the vehicle;
evaluating the sonar signal to determine if objects are located within the second region of interest;
wherein the window Position is determined based on an elevation angle of the camera; and
wherein the lower edge of each window is calculated based on the relationship Īø1=a tan(āˆ’r1/hc) where hc is the height of the camera from the road surface, r1 is the range of window from the camera, and the module is [0, Ļ€].
19. The method according to claim 18, wherein the processor is configured to determine a score indicating likelihood the candidate object is a target vehicle.
20. The method according to claim 19, wherein the processor is configured to determine the score based on a size ratio between each window and the characteristics of the electrical image.
21. The method according to claim 18, wherein the predetermined physical size is based on vehicle characteristics.
22. The method according to claim 21, wherein the vehicle characteristics include a width of the vehicle.
23. The method according to claim 21, wherein the vehicle characteristics include a height of the vehicle.
24. The method according to claim 18, wherein the characteristics of the electrical image include edge segments within the electrical image.
25. The method according to claim 24, wherein the edge segments are vertical edge segments.
26. The method according to claim 24, wherein the characteristics of the electrical image include a height of the edge segments.
27. The method according to claim 24, wherein the characteristics of the electrical image include a width of the edge segments.
28. The method according to claim 24, wherein the characteristics of the electrical image include objects constructed from the edge segments.
29. The method according to claim 28, wherein the characteristics of the electrical image include a height of the objects.
30. The method according to claim 28, wherein the characteristics of the electrical image include a width of the objects.
31. The method according to claim 18, wherein a position of each window is determined based on the target range.
32. The method according to claim 18, wherein evaluating characteristics of the electrical image includes performing an edge enhancement algorithm to the electrical image.
33. The method according to claim 32, wherein the edge enhancement algorithm is a vertical edge enhancement algorithm.
34. The method according to claim 32, wherein evaluating characteristics of the electrical image includes extracting only the electrical image within the window.
35. The method according to claim 32, wherein evaluating characteristics of the electrical image includes binarizing the electrical image utilizing a threshold.
36. The method according to claim 32, wherein evaluating characteristics of the electrical image includes identifying segments from the binarized image.
37. The method according to claim 36, wherein evaluating characteristics of the electrical image includes constructing objects from the segments.
38. The method according to claim 37, wherein evaluating the electrical image, includes determining a score for each object based on the object width, object height position, object height, and segment width.
39. The method according to claim 38, wherein evaluating characteristics of the electrical image, further includes determining a range of the object that is based on the score of the object.
40. The method according to claim 18, further comprising the step of determining the range of objects located within the second region of interest using the sonar signal.
41. The method of claim 18, further comprising the step of:
determining if the object detected within the second region of interest is detected in the first region of interest;
using the sonar signal to determine the range of the object detected within the second region of interest if the object is not detected within the first region of interest.
42. A method for determining range of a vehicle, the method comprising:
receiving an optical image of a first region of interest onto a camera;
receiving a sonar signal of a second region of interest onto a sonar;
generating an electrical image based on the optical image;
identifying a plurality of windows within the electrical image, each window corresponding to a predetermined physical size and a target range from the camera;
evaluating characteristics of a candidate object within each window signal to identify the vehicle;
evaluating the sonar signal to determine if objects are located within the second region of interest;
wherein the window position is determined based on an elevation angle of the camera; and
wherein the upper edge of each window is calculated based on the relationship Īø1h=a tan(r1/(hwāˆ’hc)) where hw is a window height, hc is the height of the camera from the road surface, r1 is the range of window from the camera, and the module is [0, Ļ€].
43. A method for determining range of a vehicle, the method comprising:
receiving an optical image of a first region of interest onto a camera;
receiving a sonar signal of a second region of interest onto a sonar;
generating an electrical image based on the optical image;
identifying a plurality of windows within the electrical image, reach window corresponding to a predetermined physical size and a target range from the camera;
evaluating characteristics of a candidate object within each window signal to identify the vehicle;
evaluating the sonar signal to determine if objects are located within the second region of interest;
wherein the position of each window is determined based on an azimuth angle of the camera; and
wherein the right edge of each window is calculated based on the relationship Ļ†1=a tan(āˆ’width13 /(2*r1))+Ļ€/2 where window w is the distance from the center of the window to the horizontal edges, r1 is the horizontal range of the window from the camera, and the module is [āˆ’Ļ€/2,Ļ€/2].
44. method for determining range of a vehicle, the method comprising:
receiving an optical image of a first region of interest onto a camera;
receiving a sonar signal of a second region of interest onto a sonar;
generating an electrical image based on the optical image;
identifying a plurality of windows within the electrical image, each window corresponding to a predetermined physical size and a target range from the camera;
evaluating characteristics of a candidate object within each window signal to identify the vehicle;
evaluating the sonar signal to determine if objects are located within the second region of interest;
wherein the position of each window is determined based on an azimuth angle of the camera; and
wherein the left edge of each window is calculated based on the relationship Ļ†1h=a tan(width_w/(2*r1))+Ļ€/2 where window w is the distance from the center of the window to the horizontal edges, r1 is the horizontal range of the window from the camera, and the module is [āˆ’Ļ€/2,Ļ€/2].
US11/202,917 2005-08-12 2005-08-12 Single camera system and method for range and lateral position measurement of a preceding vehicle Expired - Fee Related US7545956B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US11/202,917 US7545956B2 (en) 2005-08-12 2005-08-12 Single camera system and method for range and lateral position measurement of a preceding vehicle

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
US11/202,917 US7545956B2 (en) 2005-08-12 2005-08-12 Single camera system and method for range and lateral position measurement of a preceding vehicle

Publications (2)

Publication Number Publication Date
US20070035385A1 US20070035385A1 (en) 2007-02-15
US7545956B2 true US7545956B2 (en) 2009-06-09

Family

ID=37742027

Family Applications (1)

Application Number Title Priority Date Filing Date
US11/202,917 Expired - Fee Related US7545956B2 (en) 2005-08-12 2005-08-12 Single camera system and method for range and lateral position measurement of a preceding vehicle

Country Status (1)

Country Link
US (1) US7545956B2 (en)

Cited By (17)

* Cited by examiner, ā€  Cited by third party
Publication number Priority date Publication date Assignee Title
US20070060383A1 (en) * 2005-09-14 2007-03-15 Nintendo Co., Ltd. Video game program and video game system
US20070060384A1 (en) * 2005-09-14 2007-03-15 Nintendo Co., Ltd. Storage medium storing video game program
US20070191112A1 (en) * 2006-02-07 2007-08-16 Nintendo Co., Ltd. Storage medium storing subject selecting program and subject selecting apparatus
US20070211027A1 (en) * 2006-03-09 2007-09-13 Nintendo Co., Ltd. Image processing apparatus and storage medium storing image processing program
US20070211050A1 (en) * 2006-03-09 2007-09-13 Nintendo Co., Ltd. Coordinate calculating apparatus and coordinate calculating program
US20070222750A1 (en) * 2006-03-23 2007-09-27 Nintendo Co., Ltd., Position calculation apparatus, storage medium storing position calculation program, game apparatus, and storage medium storing game program
US20070255480A1 (en) * 2006-04-21 2007-11-01 Southall John B Apparatus and method for object detection and tracking and roadway awareness using stereo cameras
US7834848B2 (en) 2006-03-09 2010-11-16 Nintendo Co., Ltd Coordinate calculating apparatus and coordinate calculating program
US8157651B2 (en) 2005-09-12 2012-04-17 Nintendo Co., Ltd. Information processing program
US20130129171A1 (en) * 2011-11-23 2013-05-23 Samsung Electronics Co., Ltd. Computer-aided diagnosis method and computer-aided diagnosis apparatus therefor
US20140002657A1 (en) * 2012-06-29 2014-01-02 Lg Innotek Co., Ltd. Forward collision warning system and forward collision warning method
US20140088928A1 (en) * 2012-09-27 2014-03-27 Futurewei Technologies, Inc. Constructing Three Dimensional Model Using User Equipment
US9720556B2 (en) 2007-10-01 2017-08-01 Nintendo Co., Ltd. Storage medium storing image processing program and image processing apparatus
US20210302573A1 (en) * 2020-03-30 2021-09-30 Aisin Seiki Kabushiki Kaisha Object detection system
US20210318431A1 (en) * 2020-04-08 2021-10-14 Aisin Seiki Kabushiki Kaisha Object detection device
US20220057506A1 (en) * 2020-08-20 2022-02-24 Aisin Seiki Kabushiki Kaisha Object detection device
US20220087208A1 (en) * 2019-01-24 2022-03-24 Lely Patent N.V. Position-determining device

Families Citing this family (10)

* Cited by examiner, ā€  Cited by third party
Publication number Priority date Publication date Assignee Title
JP4858574B2 (en) * 2009-05-19 2012-01-18 惈ćƒØć‚æč‡Ŗå‹•č»Šę Ŗ式会ē¤¾ Object detection device
DE102012001554A1 (en) * 2012-01-26 2013-08-01 Connaught Electronics Ltd. Method for operating a driver assistance device of a motor vehicle, driver assistance device and motor vehicle
JP5809653B2 (en) * 2013-02-28 2015-11-11 åÆŒå£«ćƒ•ć‚¤ćƒ«ćƒ ę Ŗ式会ē¤¾ Inter-vehicle distance calculation device and operation control method thereof
FR3023921B1 (en) * 2014-06-12 2017-11-03 Valeo Schalter & Sensoren Gmbh METHOD AND SYSTEM FOR ESTIMATING A PARAMETER IN A MOTOR VEHICLE AND A MOTOR VEHICLE EQUIPPED WITH SUCH A SYSTEM
US10138672B2 (en) 2014-12-12 2018-11-27 Serge B. HOYDA Mounting system for a camera
US11518309B2 (en) * 2014-12-12 2022-12-06 Serge Hoyda LLC System and process for viewing in blind spots
US10125918B2 (en) 2014-12-12 2018-11-13 Serge B. HOYDA Mounting system for a camera
US11124116B2 (en) * 2014-12-12 2021-09-21 Serge B. HOYDA System and process for viewing in blind spots
WO2016094801A1 (en) * 2014-12-12 2016-06-16 Hoyda Serge B System and process for viewing in blind spots
KR102286174B1 (en) * 2015-09-01 2021-08-06 ģ£¼ģ‹ķšŒģ‚¬ ė§Œė„ Driving assistance apparatus and driving assistance method

Citations (28)

* Cited by examiner, ā€  Cited by third party
Publication number Priority date Publication date Assignee Title
US5790403A (en) 1994-07-12 1998-08-04 Honda Giken Kogyo Kabushiki Kaisha Lane image processing system for vehicle
US5901806A (en) 1996-12-16 1999-05-11 Nissan Motor Co., Ltd. Vehicle speed control system
JPH11142168A (en) 1997-11-07 1999-05-28 Nissan Motor Co Ltd Environment-recognizing apparatus
US20020057195A1 (en) 2000-09-22 2002-05-16 Nissan Motor Co., Ltd. Method and apparatus for estimating inter-vehicle distance using radar and camera
US6459982B1 (en) 1999-07-19 2002-10-01 Nissan Motor Co., Ltd. Preceding vehicle following control system
US6470257B1 (en) 2001-04-16 2002-10-22 Nissan Motor Co., Ltd. Adaptive cruise control system for automotive vehicles
US6484086B2 (en) 2000-12-28 2002-11-19 Hyundai Motor Company Method for detecting road slope and system for controlling vehicle speed using the method
US20030045990A1 (en) 2001-08-30 2003-03-06 Hitachi, Ltd. Adaptive cruise control apparatus and system
US6556913B2 (en) 2000-08-29 2003-04-29 Hitachi, Ltd. Cruise control system and vehicle loaded with the same
JP2003143597A (en) 2001-11-07 2003-05-16 Denso Corp Vehicle surrounding picture processor and recording medium
US6580385B1 (en) 1999-05-26 2003-06-17 Robert Bosch Gmbh Object detection system
US20030204299A1 (en) 2002-04-30 2003-10-30 Ford Global Technologies, Inc. Ramp identification in adaptive cruise control
US6708099B2 (en) 2002-01-17 2004-03-16 Ford Global Technologies, Llc Stop and go adaptive cruise control system
US20040056950A1 (en) * 2002-09-25 2004-03-25 Kabushiki Kaisha Toshiba Obstacle detection apparatus and method
US20040061626A1 (en) 2002-09-30 2004-04-01 Nissan Motor Co., Ltd. Preceding-vehicle following control system
US6771208B2 (en) 2002-04-24 2004-08-03 Medius, Inc. Multi-sensor system
US20040149504A1 (en) 2002-02-22 2004-08-05 Hans-Christian Swoboda Adaptive cruise control device for a motor vehicle
US20040176900A1 (en) 2003-03-03 2004-09-09 Koji Yajima Vehicle drive assist system
US6792344B2 (en) 1997-04-25 2004-09-14 Hitachi, Ltd. Automotive control apparatus and method
US6795014B2 (en) 2002-10-28 2004-09-21 Hyundai Motor Company Method and apparatus for detecting vehicle distance
US6825778B2 (en) 2002-10-21 2004-11-30 International Road Dynamics Inc. Variable speed limit system
US6834232B1 (en) 2003-07-30 2004-12-21 Ford Global Technologies, Llc Dual disimilar sensing object detection and targeting system
US20050010351A1 (en) 2001-10-05 2005-01-13 Klaus-Peter Wagner Speed regulator with distance regulating function
US6856887B2 (en) 2001-01-19 2005-02-15 Nissan Motor Co., Ltd. Adaptive cruise control system and method for automotive vehicle with inter-vehicle distance control function
US6859716B2 (en) 2002-08-06 2005-02-22 Honda Giken Kogyo Kabushiki Kaisha Travel following system for vehicle
US6873911B2 (en) 2002-02-01 2005-03-29 Nissan Motor Co., Ltd. Method and system for vehicle operator assistance improvement
US6873899B2 (en) 2003-04-02 2005-03-29 Honda Motor Co., Ltd. Drive control apparatus for vehicle
US20050102070A1 (en) 2003-11-11 2005-05-12 Nissan Motor Co., Ltd. Vehicle image processing device

Patent Citations (28)

* Cited by examiner, ā€  Cited by third party
Publication number Priority date Publication date Assignee Title
US5790403A (en) 1994-07-12 1998-08-04 Honda Giken Kogyo Kabushiki Kaisha Lane image processing system for vehicle
US5901806A (en) 1996-12-16 1999-05-11 Nissan Motor Co., Ltd. Vehicle speed control system
US6792344B2 (en) 1997-04-25 2004-09-14 Hitachi, Ltd. Automotive control apparatus and method
JPH11142168A (en) 1997-11-07 1999-05-28 Nissan Motor Co Ltd Environment-recognizing apparatus
US6580385B1 (en) 1999-05-26 2003-06-17 Robert Bosch Gmbh Object detection system
US6459982B1 (en) 1999-07-19 2002-10-01 Nissan Motor Co., Ltd. Preceding vehicle following control system
US6556913B2 (en) 2000-08-29 2003-04-29 Hitachi, Ltd. Cruise control system and vehicle loaded with the same
US20020057195A1 (en) 2000-09-22 2002-05-16 Nissan Motor Co., Ltd. Method and apparatus for estimating inter-vehicle distance using radar and camera
US6484086B2 (en) 2000-12-28 2002-11-19 Hyundai Motor Company Method for detecting road slope and system for controlling vehicle speed using the method
US6856887B2 (en) 2001-01-19 2005-02-15 Nissan Motor Co., Ltd. Adaptive cruise control system and method for automotive vehicle with inter-vehicle distance control function
US6470257B1 (en) 2001-04-16 2002-10-22 Nissan Motor Co., Ltd. Adaptive cruise control system for automotive vehicles
US20030045990A1 (en) 2001-08-30 2003-03-06 Hitachi, Ltd. Adaptive cruise control apparatus and system
US20050010351A1 (en) 2001-10-05 2005-01-13 Klaus-Peter Wagner Speed regulator with distance regulating function
JP2003143597A (en) 2001-11-07 2003-05-16 Denso Corp Vehicle surrounding picture processor and recording medium
US6708099B2 (en) 2002-01-17 2004-03-16 Ford Global Technologies, Llc Stop and go adaptive cruise control system
US6873911B2 (en) 2002-02-01 2005-03-29 Nissan Motor Co., Ltd. Method and system for vehicle operator assistance improvement
US20040149504A1 (en) 2002-02-22 2004-08-05 Hans-Christian Swoboda Adaptive cruise control device for a motor vehicle
US6771208B2 (en) 2002-04-24 2004-08-03 Medius, Inc. Multi-sensor system
US20030204299A1 (en) 2002-04-30 2003-10-30 Ford Global Technologies, Inc. Ramp identification in adaptive cruise control
US6859716B2 (en) 2002-08-06 2005-02-22 Honda Giken Kogyo Kabushiki Kaisha Travel following system for vehicle
US20040056950A1 (en) * 2002-09-25 2004-03-25 Kabushiki Kaisha Toshiba Obstacle detection apparatus and method
US20040061626A1 (en) 2002-09-30 2004-04-01 Nissan Motor Co., Ltd. Preceding-vehicle following control system
US6825778B2 (en) 2002-10-21 2004-11-30 International Road Dynamics Inc. Variable speed limit system
US6795014B2 (en) 2002-10-28 2004-09-21 Hyundai Motor Company Method and apparatus for detecting vehicle distance
US20040176900A1 (en) 2003-03-03 2004-09-09 Koji Yajima Vehicle drive assist system
US6873899B2 (en) 2003-04-02 2005-03-29 Honda Motor Co., Ltd. Drive control apparatus for vehicle
US6834232B1 (en) 2003-07-30 2004-12-21 Ford Global Technologies, Llc Dual disimilar sensing object detection and targeting system
US20050102070A1 (en) 2003-11-11 2005-05-12 Nissan Motor Co., Ltd. Vehicle image processing device

Non-Patent Citations (10)

* Cited by examiner, ā€  Cited by third party
Title
"Beyond cruise control", source(s): www.economist.com/science/tq/PrinterFriendly.cfm?Story-662334, Jun. 21, 2001.
"Environment Recognition Technologies for Supporting Safe Driving", Author(s): Kazuaki Takano, Tatsuhiko Monji, Hiroshi Kondo, Dr. Eng, Yuji Otsuka, 2004.
"Game-like interfaces for Toyota concept car", source(s): http://www.sae.org/automag/electronics/06-2004/1-112-6-337.pdf.
"Giving Vision to Vehicles", Author: Eric J. Lerner, Aug. 2002.
"Mobile Vision", source(s): http://www.aglaia-gmbh.de/videos/doc/AMAA-2004-03-25-Aglaia-Visual-Sensors.pdf.
"Pedestrian Detection Review", source(s): http://path.berkeley.edu/~cychan/Research-and-Presentation/Pedestrian-Detection-TO5200/Literature-Review-v4.pdf, Apr. 2005.
Cheok et al., "A Multisensor-Based Collision Avoidance System with Application to a Military HMMWV", Oct. 2000, IEEE Intelligent Transportation Systems,pp. 288-292. *
Kim et al., "Front and Rear Vehicle Detection and Tracking in the Day an Night Times Using Vision and Sonar Fusion", Aug. 2-6, 2005, IEEE, pp. 2173-2178. *
Srinivasa et al, "A Fusion for Real-Time Forward Collision Warning in Automobiles", 2003, IEEE, pp. 457-462. *
Stein et al, "Vision-based ACC with a Single Camera: Bounds on Range and Range Rate Accuracy", 2003, IEEE, pp. 120-125. *

Cited By (32)

* Cited by examiner, ā€  Cited by third party
Publication number Priority date Publication date Assignee Title
US8708824B2 (en) 2005-09-12 2014-04-29 Nintendo Co., Ltd. Information processing program
US8157651B2 (en) 2005-09-12 2012-04-17 Nintendo Co., Ltd. Information processing program
US7833100B2 (en) 2005-09-14 2010-11-16 Nintendo Co., Ltd. Video game program and video game system
US20070060384A1 (en) * 2005-09-14 2007-03-15 Nintendo Co., Ltd. Storage medium storing video game program
US9561441B2 (en) 2005-09-14 2017-02-07 Nintendo Co., Ltd. Storage medium storing video game program for calculating a distance between a game controller and a reference
US20070060383A1 (en) * 2005-09-14 2007-03-15 Nintendo Co., Ltd. Video game program and video game system
US20080318692A1 (en) * 2005-09-14 2008-12-25 Nintendo Co., Ltd. Storage medium storing video game program for calculating a distance between a game controller and a reference
US8228293B2 (en) 2005-09-14 2012-07-24 Nintendo Co., Ltd. Remote control and system and method using the remote control
US20070191112A1 (en) * 2006-02-07 2007-08-16 Nintendo Co., Ltd. Storage medium storing subject selecting program and subject selecting apparatus
US9389751B2 (en) 2006-02-07 2016-07-12 Nintendo Co., Ltd. Storage medium storing subject selecting program and subject selecting apparatus
US7786976B2 (en) 2006-03-09 2010-08-31 Nintendo Co., Ltd. Coordinate calculating apparatus and coordinate calculating program
US7834848B2 (en) 2006-03-09 2010-11-16 Nintendo Co., Ltd Coordinate calculating apparatus and coordinate calculating program
US8797264B2 (en) 2006-03-09 2014-08-05 Nintendo Co., Ltd. Image processing apparatus and storage medium storing image processing program
US20070211050A1 (en) * 2006-03-09 2007-09-13 Nintendo Co., Ltd. Coordinate calculating apparatus and coordinate calculating program
US20070211027A1 (en) * 2006-03-09 2007-09-13 Nintendo Co., Ltd. Image processing apparatus and storage medium storing image processing program
US7831064B2 (en) * 2006-03-23 2010-11-09 Nintendo Co., Ltd. Position calculation apparatus, storage medium storing position calculation program, game apparatus, and storage medium storing game program
US20070222750A1 (en) * 2006-03-23 2007-09-27 Nintendo Co., Ltd., Position calculation apparatus, storage medium storing position calculation program, game apparatus, and storage medium storing game program
US8108119B2 (en) * 2006-04-21 2012-01-31 Sri International Apparatus and method for object detection and tracking and roadway awareness using stereo cameras
US20070255480A1 (en) * 2006-04-21 2007-11-01 Southall John B Apparatus and method for object detection and tracking and roadway awareness using stereo cameras
US9720556B2 (en) 2007-10-01 2017-08-01 Nintendo Co., Ltd. Storage medium storing image processing program and image processing apparatus
US9087373B2 (en) * 2011-11-23 2015-07-21 Samsung Electronics Co., Ltd. Computer-aided diagnosis method and computer-aided diagnosis apparatus therefor
US20130129171A1 (en) * 2011-11-23 2013-05-23 Samsung Electronics Co., Ltd. Computer-aided diagnosis method and computer-aided diagnosis apparatus therefor
US20140002657A1 (en) * 2012-06-29 2014-01-02 Lg Innotek Co., Ltd. Forward collision warning system and forward collision warning method
US20140088928A1 (en) * 2012-09-27 2014-03-27 Futurewei Technologies, Inc. Constructing Three Dimensional Model Using User Equipment
US9589078B2 (en) * 2012-09-27 2017-03-07 Futurewei Technologies, Inc. Constructing three dimensional model using user equipment
US20220087208A1 (en) * 2019-01-24 2022-03-24 Lely Patent N.V. Position-determining device
US20210302573A1 (en) * 2020-03-30 2021-09-30 Aisin Seiki Kabushiki Kaisha Object detection system
US11698456B2 (en) * 2020-03-30 2023-07-11 Aisin Corporation Object detection system
US20210318431A1 (en) * 2020-04-08 2021-10-14 Aisin Seiki Kabushiki Kaisha Object detection device
US11719815B2 (en) * 2020-04-08 2023-08-08 Aisin Corporation Object detection device
US20220057506A1 (en) * 2020-08-20 2022-02-24 Aisin Seiki Kabushiki Kaisha Object detection device
US11619736B2 (en) * 2020-08-20 2023-04-04 Aisin Corporation Object detection device

Also Published As

Publication number Publication date
US20070035385A1 (en) 2007-02-15

Similar Documents

Publication Publication Date Title
US7545956B2 (en) Single camera system and method for range and lateral position measurement of a preceding vehicle
US7561720B2 (en) Single camera system and method for range and lateral position measurement of a preceding vehicle
US7561721B2 (en) System and method for range measurement of a preceding vehicle
US11348266B2 (en) Estimating distance to an object using a sequence of images recorded by a monocular camera
Rezaei et al. Robust vehicle detection and distance estimation under challenging lighting conditions
US10521676B2 (en) Lane detection device, lane departure determination device, lane detection method and lane departure determination method
US20070031008A1 (en) System and method for range measurement of a preceding vehicle
US8040227B2 (en) Method for detecting moving objects in a blind spot region of a vehicle and blind spot detection device
CN111289980B (en) Roadside stationary object detection method and system based on vehicle-mounted millimeter wave radar
US7623681B2 (en) System and method for range measurement of a preceding vehicle
US7046822B1 (en) Method of detecting objects within a wide range of a road vehicle
EP2993654B1 (en) Method and system for forward collision warning
Cualain et al. Automotive standards-grade lane departure warning system
Rezaei et al. Vehicle detection based on multi-feature clues and Dempster-Shafer fusion theory
JP4246766B2 (en) Method and apparatus for locating and tracking an object from a vehicle
US20030002713A1 (en) Vision-based highway overhead structure detection system
Arenado et al. Monovisionā€based vehicle detection, distance and relative speed measurement in urban traffic
JP3674400B2 (en) Ambient environment recognition device
Ponsa et al. On-board image-based vehicle detection and tracking
CN114118252A (en) Vehicle detection method and detection device based on sensor multivariate information fusion
Cualain et al. Multiple-camera lane departure warning system for the automotive environment
Barua et al. An Efficient Method of Lane Detection and Tracking for Highway Safety
Zarbakht et al. Lane detection under adverse conditions based on dual color space
Oniga et al. A fast ransac based approach for computing the orientation of obstacles in traffic scenes
CN115327572A (en) Method for detecting obstacle in front of vehicle

Legal Events

Date Code Title Description
AS Assignment

Owner name: VISTEON GLOBAL TECHNOLOGIES, INC., MICHIGAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:MIYAHARA, SHUNJI;REEL/FRAME:016545/0412

Effective date: 20050915

AS Assignment

Owner name: JPMORGAN CHASE BANK, TEXAS

Free format text: SECURITY INTEREST;ASSIGNOR:VISTEON GLOBAL TECHNOLOGIES, INC.;REEL/FRAME:022368/0001

Effective date: 20060814

Owner name: JPMORGAN CHASE BANK,TEXAS

Free format text: SECURITY INTEREST;ASSIGNOR:VISTEON GLOBAL TECHNOLOGIES, INC.;REEL/FRAME:022368/0001

Effective date: 20060814

STCF Information on status: patent grant

Free format text: PATENTED CASE

AS Assignment

Owner name: WILMINGTON TRUST FSB, AS ADMINISTRATIVE AGENT, MIN

Free format text: GRANT OF SECURITY INTEREST IN PATENT RIGHTS;ASSIGNOR:VISTEON GLOBAL TECHNOLOGIES, INC.;REEL/FRAME:022732/0263

Effective date: 20090430

Owner name: WILMINGTON TRUST FSB, AS ADMINISTRATIVE AGENT,MINN

Free format text: GRANT OF SECURITY INTEREST IN PATENT RIGHTS;ASSIGNOR:VISTEON GLOBAL TECHNOLOGIES, INC.;REEL/FRAME:022732/0263

Effective date: 20090430

AS Assignment

Owner name: THE BANK OF NEW YORK MELLON, AS ADMINISTRATIVE AGE

Free format text: ASSIGNMENT OF PATENT SECURITY INTEREST;ASSIGNOR:JPMORGAN CHASE BANK, N.A., A NATIONAL BANKING ASSOCIATION;REEL/FRAME:022974/0057

Effective date: 20090715

CC Certificate of correction
AS Assignment

Owner name: VISTEON GLOBAL TECHNOLOGIES, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS RECORDED AT REEL 022974 FRAME 0057;ASSIGNOR:THE BANK OF NEW YORK MELLON;REEL/FRAME:025095/0711

Effective date: 20101001

Owner name: VISTEON GLOBAL TECHNOLOGIES, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS RECORDED AT REEL 022732 FRAME 0263;ASSIGNOR:WILMINGTON TRUST FSB;REEL/FRAME:025095/0451

Effective date: 20101001

AS Assignment

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT, NEW

Free format text: SECURITY AGREEMENT;ASSIGNORS:VISTEON CORPORATION;VC AVIATION SERVICES, LLC;VISTEON ELECTRONICS CORPORATION;AND OTHERS;REEL/FRAME:025241/0317

Effective date: 20101007

Owner name: MORGAN STANLEY SENIOR FUNDING, INC., AS AGENT, NEW

Free format text: SECURITY AGREEMENT (REVOLVER);ASSIGNORS:VISTEON CORPORATION;VC AVIATION SERVICES, LLC;VISTEON ELECTRONICS CORPORATION;AND OTHERS;REEL/FRAME:025238/0298

Effective date: 20101001

AS Assignment

Owner name: VISTEON GLOBAL TREASURY, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VISTEON EUROPEAN HOLDING, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VC AVIATION SERVICES, LLC, MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VISTEON SYSTEMS, LLC, MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VISTEON GLOBAL TECHNOLOGIES, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VISTEON INTERNATIONAL HOLDINGS, INC., MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VISTEON INTERNATIONAL BUSINESS DEVELOPMENT, INC.,

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VISTEON CORPORATION, MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

Owner name: VISTEON ELECTRONICS CORPORATION, MICHIGAN

Free format text: RELEASE BY SECURED PARTY AGAINST SECURITY INTEREST IN PATENTS ON REEL 025241 FRAME 0317;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:026178/0412

Effective date: 20110406

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: CITIBANK., N.A., AS ADMINISTRATIVE AGENT, NEW YORK

Free format text: SECURITY INTEREST;ASSIGNORS:VISTEON CORPORATION, AS GRANTOR;VISTEON GLOBAL TECHNOLOGIES, INC., AS GRANTOR;REEL/FRAME:032713/0065

Effective date: 20140409

AS Assignment

Owner name: VISTEON GLOBAL TREASURY, INC., MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VC AVIATION SERVICES, LLC, MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VISTEON INTERNATIONAL HOLDINGS, INC., MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VISTEON GLOBAL TECHNOLOGIES, INC., MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VISTEON INTERNATIONAL BUSINESS DEVELOPMENT, INC.,

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VISTEON ELECTRONICS CORPORATION, MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VISTEON EUROPEAN HOLDINGS, INC., MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VISTEON CORPORATION, MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

Owner name: VISTEON SYSTEMS, LLC, MICHIGAN

Free format text: RELEASE OF SECURITY INTEREST IN INTELLECTUAL PROPERTY;ASSIGNOR:MORGAN STANLEY SENIOR FUNDING, INC.;REEL/FRAME:033107/0717

Effective date: 20140409

FPAY Fee payment

Year of fee payment: 8

FEPP Fee payment procedure

Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

LAPS Lapse for failure to pay maintenance fees

Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STCH Information on status: patent discontinuation

Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362

FP Lapsed due to failure to pay maintenance fee

Effective date: 20210609