CN103700110A - Full-automatic image matching method - Google Patents
Full-automatic image matching method Download PDFInfo
- Publication number
- CN103700110A CN103700110A CN201310734262.6A CN201310734262A CN103700110A CN 103700110 A CN103700110 A CN 103700110A CN 201310734262 A CN201310734262 A CN 201310734262A CN 103700110 A CN103700110 A CN 103700110A
- Authority
- CN
- China
- Prior art keywords
- image
- matched
- point
- matching
- full
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Image Processing (AREA)
- Image Analysis (AREA)
Abstract
The invention provides a full-automatic image matching method to solve the problems of lower degree of automation, narrow application range and the like of the prior art of image matching. The method comprises the following steps: building image pyramids for a reference image and an image to be matched, and acquiring the homonymy point, on the reference image, of the central point of the image to be matched through Fourier transformation, log-polar transformation and a phase correlation method; respectively matching image points in eight or four regions through a partitioning phase correlation method by using a matching seed point pair; optionally selecting another successfully matched point pair as a seed point pair unless matching of all image points in the whole image is finished. The full-automatic image matching method has the beneficial technical effects that automatic matching of satellite images and aerial images, which have affine transformation, local tiny geometric distortion or a certain degree of radiometric distortion, can be achieved, any auxiliary information and manual intervention are not needed, and the total matching accuracy is remarkably superior to that of the accepted optimum correlation coefficient method among basic matching methods.
Description
Invention field
The present invention relates to Image Matching technology, specially refer to a kind of full-automatic image matching process.
Background technology
Increase day by day and the continuous growth to three-dimensional spatial information demand along with remote sensing image data, require also more and more urgent to the robotization of Remote Sensing Image Processing Technology and real time implementation.Image Matching is that three-dimensional spatial information obtains and the gordian technique of rebuilding, and is the basis that many remote sensing images such as visual fusion, variation detection, target identification are processed application.In a sense, the solution degree of image matching problem has determined the automaticity of Photogrammetry and Remote Sensing.At present, although the research of Image Matching technology makes great progress, prior art image matching method is all existing certain limitation aspect robotization and the scope of application.For example, conventional based on gray scale (region) matching process, utilize normalized correlation coefficient as similarity measurement, can solve the partial radiation distortion problem of Image Matching, but responsive to the Affine distortion of image, cannot solve the matching problem that has Rotation and Zoom distorted image.For another example, the matching process based on feature mates by building various feature invariant amounts, has good anti-Affine distortion performance, but the precision of characteristic matching result depends on precision and the feature descriptive model of feature extraction and location.Now widely used SIFT characteristic matching method is the local invariant feature point in multiple dimensioned lower detection, but within the partial structurtes in image are often present in certain range scale, rather than certain fixing yardstick, after multiple dimensioned lower detected characteristics point, can cause the very approaching point of many positions and yardstick, they all represent same partial structurtes, thereby cause more erroneous matching, and cannot carry out dense Stereo Matching, can only obtain sparse matching characteristic point.In addition, aspect the robotization of Image Matching, current matching process also needs manual intervention or some supplementarys, does not realize robotization completely.Obviously, prior art image matching method exists the problems such as the lower and scope of application of automaticity is narrower.
Summary of the invention
For solving the problems such as the automaticity of Image Matching prior art existence is lower and the scope of application is narrower, the present invention proposes a kind of full-automatic image matching process.Full-automatic image matching process of the present invention, builds image pyramid to reference images and image to be matched, and tries to achieve the same place of image center point to be matched in reference images by Fourier transform, log-polar transform and phase correlation method; Using central point and same place thereof as coupling Seed Points pair, to the picture point in its eight field or four fields, adopt respectively blocking phase correlation method to mate; An optional point that the match is successful, to as Seed Points pair, carries out blocking phase correlation method coupling to the not coupling picture point in its eight neighborhood or neighbours territory again, until all picture points in view picture image complete coupling.
Further, full-automatic image matching process of the present invention, comprises the following steps:
S1, structure image pyramid, set up respectively image pyramid to reference images and image to be matched;
S2, Fourier transform, the amplitude spectrum of two width images after the pyramid top layer image of two width images is carried out Fourier transform and gets conversion;
S3, log-polar transform, carry out log-polar transform to the amplitude spectrum of two width images after converting respectively;
S4, ask rotation parameter and zooming parameter, adopt phase correlation method to try to achieve reference images after step S2 processes and rotation parameter and the zooming parameter between the pyramid top layer image of image to be matched;
S5, carry out image rectification, utilize rotation parameter and zooming parameter to correct the pyramid top layer image of image to be matched;
S6, ask translation parameters, adopt phase correlation method to try to achieve the translation parameters between the pyramid top layer image of the image to be matched after reference images and rectification;
S7, obtain the initial coordinate values of central point and same place thereof, utilize translation parameters to determine the coordinate of image pyramid top layer image center point to be matched and the same place in reference images thereof, then, expanded to bottom, i.e. raw video layer image;
S8, determine the accurate coordinates value of central point and same place thereof, respectively to correct centered by rear image pyramid bottom image center point to be matched and the same place in bottom reference images thereof, respectively get a local window image, to these two window images, adopt phase correlation method to mate, utilize the coordinate initial value after matching result is processed step S7 to proofread and correct, try to achieve the accurate coordinates of image pyramid bottom image center point to be matched same place in reference images;
S9, determine initial seed point pair, choose image pyramid bottom image center point to be matched and the same place in reference images thereof as the initial seed point pair that the match is successful;
S10, neighborhood picture point is mated, utilize the correlativity of contiguous picture point, to initial seed point, to the picture point in eight neighborhoods or neighbours territory, adopt respectively blocking phase correlation method to mate, until this Seed Points is complete to neighborhood Pixel matching;
S11, an optional point that the match is successful, to as Seed Points pair, carries out blocking phase correlation method coupling to this Seed Points to the not coupling picture point in eight neighborhoods or neighbours territory, until this Seed Points is complete to neighborhood Pixel matching again;
S12, repeated execution of steps 11, until all picture points in view picture image complete coupling, complete Image Matching.
Further, remote sensing image or aviation image that the reference images of full-automatic image matching process of the present invention input and image data to be matched are arbitrary format, the line number of single image is identical with columns and be 2 power side.
Further, full-automatic image matching process of the present invention is when step S1 builds image pyramid, as different in the ranks number of reference images and image to be matched, needs to resample, and makes the ranks number of two width images identical.
Further, full-automatic image matching process of the present invention is when step S1 builds image pyramid, for the reference images and the image to be matched that are of a size of 1024 * 1024 pixels, adopt the method fall 2 samplings to build respectively 3 layer image pyramids, top layer image size is 256 * 256 pixels.
Further, full-automatic image matching process of the present invention, carries out log-polar transform to amplitude spectrum respectively at step S2, comprises, by amplitude spectrum by Cartesian coordinate spatial mappings to log-polar space, the log-polar space size after mapping is 128 * 360 pixels.
Further, full-automatic image matching process of the present invention, the size of the local window image that step S8 chooses is 131 * 131 pixels.
Further, full-automatic image matching process of the present invention, adopts respectively blocking phase correlation method to mate to the picture point in its eight neighborhood or neighbours territory in step S10 or S11, and the size of its match window is 31 * 31 pixels.
The useful technique effect of full-automatic image matching process of the present invention can to have the distortion of affined transformation and local small geometrical and to a certain degree the satellite image of radiometric distortion all can realize robotization with aviation image and mate, without any supplementary and manual intervention, and overall matching accuracy is obviously better than generally acknowledged optimum correlation coefficient process in basic matching process.
Accompanying drawing explanation
Accompanying drawing 1 is the step schematic diagram of full-automatic image matching process of the present invention.
Below in conjunction with drawings and the specific embodiments, full-automatic image matching process of the present invention is further described.
Embodiment
Accompanying drawing 1 is the step schematic diagram of full-automatic image matching process of the present invention, as seen from the figure, full-automatic image matching process of the present invention, builds image pyramid and tries to achieve image center point to be matched and the same place in reference images thereof by Fourier transform, log-polar transform and phase correlation method reference images and image to be matched; Using central point and same place as coupling Seed Points pair, to the picture point in its eight field or four fields, adopt respectively phase correlation method to mate; An optional point that the match is successful, to as Seed Points pair, carries out blocking phase correlation method coupling to the not coupling picture point in its eight neighborhood or neighbours territory again, until all picture points in view picture image complete coupling.Remote sensing image or aviation image that the reference images of full-automatic image matching process of the present invention input and image data to be matched are arbitrary format, the line number of single image is identical with columns and be 2 power side, comprises the following steps:
S1, structure image pyramid, set up respectively image pyramid to reference images and image to be matched; If the ranks number of reference images and image to be matched is different, need to resample, make the ranks number of two width images identical; The reference images of embodiment and image to be matched are of a size of 1024 * 1024 pixels, adopt the method fall 2 samplings to build respectively 3 layer image pyramids, and top layer image size is 256 * 256 pixels;
S2, Fourier transform, the amplitude spectrum of two width images after the pyramid top layer image of two width images is carried out Fourier transform and gets conversion;
S3, log-polar transform, carry out log-polar transform to the amplitude spectrum of two width images after converting respectively; Embodiment be by amplitude spectrum by Cartesian coordinate spatial mappings to log-polar space, the log-polar space size after mapping is 128 * 360 pixels.
S4, ask rotation parameter and zooming parameter, adopt phase correlation method to try to achieve reference images after step S2 processes and rotation parameter and the zooming parameter between the pyramid top layer image of image to be matched;
S5, carry out image rectification, utilize rotation parameter and zooming parameter to correct the pyramid top layer image of image to be matched;
S6, ask translation parameters, adopt phase correlation method to try to achieve the translation parameters between the pyramid top layer image of the image to be matched after reference images and rectification;
S7, obtain the initial coordinate values of central point and same place thereof, utilize translation parameters to determine the coordinate of image pyramid top layer image center point to be matched and the same place in reference images thereof, then, expanded to bottom, i.e. raw video layer image;
S8, determine the accurate coordinates value of central point and same place thereof, respectively to correct centered by rear image pyramid bottom image center point to be matched and the same place in bottom reference images thereof, respectively get a local window image, to these two window images, adopt phase correlation method to mate, utilize the coordinate initial value after matching result is processed step S7 to proofread and correct, try to achieve the accurate coordinates of image pyramid bottom image center point to be matched same place in reference images; The size of the local window image of embodiment is 131 * 131 pixels;
S9, determine initial seed point pair, choose image pyramid bottom image center point to be matched and the same place in reference images thereof as the initial seed point pair that the match is successful;
S10, point, to neighborhood Pixel matching, utilize the correlativity of contiguous picture point, adopt respectively blocking phase correlation method to mate, until this Seed Points is complete to neighborhood Pixel matching to initial seed point to the picture point in eight neighborhoods or neighbours territory; The size of embodiment match window is 31 * 31 pixels;
S11, an optional point that the match is successful, to as Seed Points pair, carries out blocking phase correlation method coupling to this Seed Points to the not coupling picture point in eight neighborhoods or neighbours territory, until this Seed Points is complete to neighborhood Pixel matching again; The size of embodiment match window is 31 * 31 pixels;
S12, repeated execution of steps 11, until all picture points in view picture image complete coupling, complete Image Matching.
For the actual effect of checking full-automatic image matching process of the present invention, selected four kinds of typical landforms stereopsis to carrying out full-automatic image coupling, obtained good effect.Wherein, more smooth to physical features, the area image of the not obvious and texture-rich of eclipse phenomena, coupling accuracy is very high, is 100%; More smooth to ground, there are many skyscrapers, exist certain eclipse phenomena and left and right image to have the urban area image of luminance difference, coupling accuracy is lower, is 57.81%; The also larger mountain area image of large and local deformation to topographic relief, coupling accuracy is higher, is 71.88%; Comparatively broken and there is no the image of the wood land of obvious characteristic to texture, coupling accuracy is higher, is 97.92%.However, the overall matching accuracy of full-automatic image matching process of the present invention is obviously better than generally acknowledged optimum correlation coefficient process in basic matching process.
Obviously, the useful technique effect of full-automatic image matching process of the present invention can to have the distortion of affined transformation and local small geometrical and to a certain degree the satellite image of radiometric distortion all can realize robotization with aviation image and mate, without any supplementary and manual intervention, and overall matching accuracy is obviously better than generally acknowledged optimum correlation coefficient process in basic matching process.
Claims (8)
1. a full-automatic image matching process, is characterized in that, reference images and image to be matched are built image pyramid and try to achieve the same place of image center point to be matched in reference images by Fourier transform, log-polar transform and phase correlation method; Using central point and same place thereof as coupling Seed Points pair, to the picture point in its eight field or four fields, adopt respectively phase correlation method to mate; An optional point that the match is successful, to as Seed Points pair, carries out blocking phase correlation method coupling to the not coupling picture point in its eight neighborhood or neighbours territory again, until all picture points in view picture image complete coupling.
2. full-automatic image matching process according to claim 1, is characterized in that, the method comprises the following steps:
S1, structure image pyramid, set up respectively image pyramid to reference images and image to be matched;
S2, Fourier transform, the amplitude spectrum of two width images after the pyramid top layer image of two width images is carried out Fourier transform and gets conversion;
S3, log-polar transform, carry out log-polar transform to the amplitude spectrum of two width images after converting respectively;
S4, ask rotation parameter and zooming parameter, adopt phase correlation method to try to achieve reference images after step S2 processes and rotation parameter and the zooming parameter between the pyramid top layer image of image to be matched;
S5, carry out image rectification, utilize rotation parameter and zooming parameter to correct the pyramid top layer image of image to be matched;
S6, ask translation parameters, adopt phase correlation method to try to achieve the translation parameters between the pyramid top layer image of the image to be matched after reference images and rectification;
S7, obtain the initial coordinate values of central point and same place thereof, utilize translation parameters to determine the coordinate of image pyramid top layer image center point to be matched and the same place in reference images thereof, then, expanded to bottom, i.e. raw video layer image;
S8, determine the accurate coordinates value of central point and same place thereof, respectively to correct centered by rear image pyramid bottom image center point to be matched and the same place in bottom reference images thereof, respectively get a local window image, to these two window images, adopt phase correlation method to mate, utilize the coordinate initial value after matching result is processed step S7 to proofread and correct, try to achieve the accurate coordinates of image pyramid bottom image center point to be matched same place in reference images;
S9, determine initial seed point pair, choose image pyramid bottom image center point to be matched and the same place in reference images thereof as the initial seed point pair that the match is successful;
S10, neighborhood picture point is mated, utilize the correlativity of contiguous picture point, to initial seed point, to the picture point in eight neighborhoods or neighbours territory, adopt respectively blocking phase correlation method to mate, until this Seed Points is complete to neighborhood Pixel matching;
S11, an optional point that the match is successful, to as Seed Points pair, carries out blocking phase correlation method coupling to this Seed Points to the not coupling picture point in eight neighborhoods or neighbours territory, until this Seed Points is complete to neighborhood Pixel matching again;
S12, repeated execution of steps 11, until all picture points in view picture image complete coupling, complete Image Matching.
3. full-automatic image matching process according to claim 2, is characterized in that, remote sensing image or aviation image that the reference images of input and image data to be matched are arbitrary format, and the line number of single image is identical with columns and be 2 power side.
4. full-automatic image matching process according to claim 2, is characterized in that, when step S1 builds image pyramid, as different in the ranks number of reference images and image to be matched, needs resampling, makes the ranks number of two width images identical.
5. full-automatic image matching process according to claim 2, it is characterized in that, when step S1 builds image pyramid, for the reference images and the image to be matched that are of a size of 1024 * 1024 pixels, the method that 2 samplings fall in employing builds respectively 3 layer image pyramids, and top layer image size is 256 * 256 pixels.
6. full-automatic image matching process according to claim 2, it is characterized in that, at step S2, respectively amplitude spectrum is carried out to log-polar transform, comprise, by amplitude spectrum by Cartesian coordinate spatial mappings to log-polar space, the log-polar space size after mapping is 128 * 360 pixels.
7. full-automatic image matching process according to claim 2, is characterized in that, the size of the local window image that step S8 chooses is 131 * 131 pixels.
8. full-automatic image matching process according to claim 2, is characterized in that, in step S10 or S11, to the picture point in its eight neighborhood or neighbours territory, adopt respectively blocking phase correlation method to mate, the size of its match window is 31 * 31 pixels.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310734262.6A CN103700110B (en) | 2013-12-27 | 2013-12-27 | Full-automatic image matching method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310734262.6A CN103700110B (en) | 2013-12-27 | 2013-12-27 | Full-automatic image matching method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103700110A true CN103700110A (en) | 2014-04-02 |
| CN103700110B CN103700110B (en) | 2017-03-22 |
Family
ID=50361628
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310734262.6A Expired - Fee Related CN103700110B (en) | 2013-12-27 | 2013-12-27 | Full-automatic image matching method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103700110B (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104156969B (en) * | 2014-08-21 | 2017-02-01 | 重庆数字城市科技有限公司 | Plane exploration method based on panoramic image depth map |
| CN106940181A (en) * | 2017-03-10 | 2017-07-11 | 中国电建集团昆明勘测设计研究院有限公司 | A kind of unmanned plane image picture control distribution of net is built and the optional commensurate in scope method of aerophotograph |
| CN108876826A (en) * | 2017-05-10 | 2018-11-23 | 深圳先进技术研究院 | A kind of image matching method and system |
| CN111241935A (en) * | 2019-12-31 | 2020-06-05 | 飞燕航空遥感技术有限公司 | Automatic matching and comparing method for multi-period aerial images |
| CN111936946A (en) * | 2019-09-10 | 2020-11-13 | 北京航迹科技有限公司 | Positioning system and method |
| CN114037913A (en) * | 2022-01-10 | 2022-02-11 | 成都国星宇航科技有限公司 | Automatic deviation rectifying method and device for remote sensing image, electronic equipment and storage medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0651337A1 (en) * | 1992-07-03 | 1995-05-03 | Omron Corporation | Object recognizing method, its apparatus, and image processing method and its apparatus |
| US20020154820A1 (en) * | 2001-03-06 | 2002-10-24 | Toshimitsu Kaneko | Template matching method and image processing device |
-
2013
- 2013-12-27 CN CN201310734262.6A patent/CN103700110B/en not_active Expired - Fee Related
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP0651337A1 (en) * | 1992-07-03 | 1995-05-03 | Omron Corporation | Object recognizing method, its apparatus, and image processing method and its apparatus |
| US20020154820A1 (en) * | 2001-03-06 | 2002-10-24 | Toshimitsu Kaneko | Template matching method and image processing device |
Non-Patent Citations (3)
| Title |
|---|
| SIAVASH ZOKAI等: "Image Registration Using Log-Ploar Mappings for Recovery of Large-Scale Similarity and Projective Transformations", 《IEEE TRANSACTIONS ON IMAGE PROCESSING》 * |
| 徐俊泽: "对数极坐标域下的图像配准技术", 《中国优秀硕士论文全文数据库》 * |
| 林卉等: "利用Fourier-Mellin变换的遥感图像自动配准", 《武汉大学学报信息科学版》 * |
Cited By (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104156969B (en) * | 2014-08-21 | 2017-02-01 | 重庆数字城市科技有限公司 | Plane exploration method based on panoramic image depth map |
| CN106940181A (en) * | 2017-03-10 | 2017-07-11 | 中国电建集团昆明勘测设计研究院有限公司 | A kind of unmanned plane image picture control distribution of net is built and the optional commensurate in scope method of aerophotograph |
| CN106940181B (en) * | 2017-03-10 | 2019-04-26 | 中国电建集团昆明勘测设计研究院有限公司 | A kind of building of unmanned plane image picture control distribution of net and the optional commensurate in scope method of aerophotograph |
| CN108876826A (en) * | 2017-05-10 | 2018-11-23 | 深圳先进技术研究院 | A kind of image matching method and system |
| CN108876826B (en) * | 2017-05-10 | 2021-09-21 | 深圳先进技术研究院 | Image matching method and system |
| CN111936946A (en) * | 2019-09-10 | 2020-11-13 | 北京航迹科技有限公司 | Positioning system and method |
| US11940279B2 (en) | 2019-09-10 | 2024-03-26 | Beijing Voyager Technology Co., Ltd. | Systems and methods for positioning |
| CN111241935A (en) * | 2019-12-31 | 2020-06-05 | 飞燕航空遥感技术有限公司 | Automatic matching and comparing method for multi-period aerial images |
| CN111241935B (en) * | 2019-12-31 | 2020-10-13 | 飞燕航空遥感技术有限公司 | Automatic matching and comparing method for multi-period aerial images |
| CN114037913A (en) * | 2022-01-10 | 2022-02-11 | 成都国星宇航科技有限公司 | Automatic deviation rectifying method and device for remote sensing image, electronic equipment and storage medium |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103700110B (en) | 2017-03-22 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103700110A (en) | Full-automatic image matching method | |
| CN104574347A (en) | On-orbit satellite image geometric positioning accuracy evaluation method on basis of multi-source remote sensing data | |
| CN104156957B (en) | Stable and high-efficiency high-resolution stereo matching method | |
| JP5470607B2 (en) | Computer system and method for aligning images / charts | |
| CN109523585B (en) | Multisource remote sensing image feature matching method based on direction phase consistency | |
| CN105389774A (en) | Method and device for aligning images | |
| CN109255808B (en) | Building texture extraction method and device based on oblique images | |
| Peña-Villasenín et al. | 3-D modeling of historic façades using SFM photogrammetry metric documentation of different building types of a historic center | |
| EP2923333A1 (en) | Method for the automatic creation of two- or three-dimensional building models | |
| CN112819903A (en) | Camera and laser radar combined calibration method based on L-shaped calibration plate | |
| US7778808B2 (en) | Geospatial modeling system providing data thinning of geospatial data points and related methods | |
| CN104392457A (en) | Automatic matching method and device for connection points of slanted images | |
| CN102446356A (en) | Parallel and adaptive matching method for acquiring remote sensing images with homogeneously-distributed matched points | |
| Kwak et al. | Registration of aerial imagery and aerial LiDAR data using centroids of plane roof surfaces as control information | |
| CN111003214B (en) | Attitude and orbit refinement method for domestic land observation satellite based on cloud control | |
| US20120321199A1 (en) | Object identification system and method of identifying an object using the same | |
| CN105678720A (en) | Image matching judging method and image matching judging device for panoramic stitching | |
| CN111709876A (en) | Image splicing method, device, equipment and storage medium | |
| CN106204507A (en) | A kind of unmanned plane image split-joint method | |
| Hussnain et al. | Automatic feature detection, description and matching from mobile laser scanning data and aerial imagery | |
| Aktar et al. | Robust mosaicking of maize fields from aerial imagery | |
| CN105184736B (en) | A kind of method of the image registration of narrow overlapping double-view field hyperspectral imager | |
| KR101580594B1 (en) | Method for line extraction of high resolution satellite images using regular grid-based hough transform | |
| Li et al. | Automatic Keyline Recognition and 3D Reconstruction For Quasi‐Planar Façades in Close‐range Images | |
| Skuratovskyi et al. | Outdoor mapping framework: from images to 3d model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20170322 Termination date: 20171227 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |