WO2015165222A1

WO2015165222A1 - Method and device for acquiring panoramic image

Info

Publication number: WO2015165222A1
Application number: PCT/CN2014/088726
Authority: WO
Inventors: 申东祚; 尼尔•道奇森; 鲁亚东
Original assignee: 华为技术有限公司; 剑桥实业有限公司
Priority date: 2014-04-29
Filing date: 2014-10-16
Publication date: 2015-11-05
Also published as: CN105096283B; CN105096283A

Abstract

Provided are a method and device for acquiring a panoramic image, the method comprising: acquiring at least two first images; extracting the background images and foreground images of the first images; determining a first overlapping image and a first non-overlapping image formed by the background images of the at least two first images, and determining a second overlapping image and a second non-overlapping image formed by the foreground images of the at least two first images; dividing the first overlapping image to obtain a plurality of first divided sub-images, and dividing the second overlapping image to obtain a plurality of second divided sub-images; selecting first splicing sub-images for each first divided sub-image, and selecting second splicing sub-images for each second divided sub-image; splicing the first splicing sub-images to obtain a target background image, and splicing the second splicing sub-images to obtain a target foreground image; and synthesizing the target background image, the target foreground image, the first non-overlapping image and the second non-overlapping image to obtain a target panoramic image having high resolution.

Description

Method and device for acquiring panoramic image

Technical field

Embodiments of the present invention relate to image processing technologies, and in particular, to a method and an apparatus for acquiring a panoramic image.

Background technique

With the development of technology, people's pursuit of resolution has reached a new height, especially the display device's screen is getting bigger and bigger, which also objectively requires higher resolution of the acquisition device.

At present, ultra-high definition video (image) acquisition devices are very expensive, so it is a natural alternative to form a camera array with relatively inexpensive lower resolution cameras. The existing panoramic stitching technique superimposes (or approximately coincides) the optical centers of a plurality of cameras, thereby obtaining a panoramic image of a large field of view.

However, these technologies require sophisticated optical devices to achieve optical center coincidence of multiple cameras, which is complicated and costly.

Summary of the invention

The invention provides a method and a device for acquiring a panoramic image, so that a high-resolution panoramic image with a large field of view can be obtained only by obtaining the internal and external parameters of the camera without calibration.

In a first aspect, an embodiment of the present invention provides a method for acquiring a panoramic image, including: acquiring at least two first images; respectively extracting a background image and a foreground image of the first image; and determining at least two of the first images. a first overlapping image and a first non-overlapping image formed by the background image, and a second overlapping image and a second non-overlapping image formed by determining at least two foreground images of the first image; performing the first overlapping image Dividing to obtain a plurality of first divided sub-images, and dividing the second overlapping image to obtain a plurality of second divided sub-images; for each of the first divided sub-images, at least two background images of the first image Select corresponding a first splicing sub-image, selecting, for each of the second divided sub-images, a corresponding second splicing sub-image in at least two foreground images of the first image; corresponding to the plurality of the first divided sub-images The first spliced sub-image is spliced to obtain a target background image, and the second spliced sub-image corresponding to the plurality of the second divided sub-images is spliced to obtain a target foreground image; and the target background image, the target foreground image, and the target image The first non-overlapping image and the second non-overlapping image are combined to obtain a target panoramic image.

With reference to the first aspect, in a first possible implementation manner of the first aspect, the foreground template of the first image is determined according to the depth value of the first image; the foreground template of the first image is 0, 1 Forming a matrix, at the same pixel point, extracting pixel points of the first image corresponding to the foreground template 1 to form a foreground image of the first image; and extracting one of the foreground templates at the same pixel point Corresponding pixel points of the first image constitute a background image of the first image.

In conjunction with the first aspect or the first possible implementation of the first aspect, in a second possible implementation of the first aspect, the determining the first overlap of the at least two background images formed by the first image And the image and the first non-overlapping image, and determining the second overlapping image and the second non-overlapping image of the foreground image formed by the at least two of the first images, specifically comprising: converting each of the background images into a background virtual image, each of the foreground images being converted into a foreground virtual image by a homography transformation; calculating the first overlapping image and the first non-overlapping image of each of the background images according to the background virtual image, according to The foreground virtual image calculates the second overlapping image and the second non-overlapping image of each of the foreground images.

With reference to the first aspect, or the first possible implementation manner of the first aspect, or the second possible implementation manner of the first aspect, in a third possible implementation manner of the first aspect, The superimposing the image to obtain the plurality of first divided sub-images comprises: calculating a first difference value of the first overlapping image on each color channel, and performing Laplacian filtering transformation and smoothing on the first difference Processing, obtaining a first Morse function value; determining the first divided sub-image on the first overlapping image according to the first Morse function value; and dividing the second overlapping image to obtain a plurality of second The dividing the sub-image includes: calculating a second difference image of the second overlapping image on each color channel, performing Laplacian filtering transformation and smoothing processing on the second difference image to obtain a second mole Function value; The second Morse function value determines the second divided sub-image on the second overlay image.

In conjunction with the third possible implementation of the first aspect, in a fourth possible implementation of the first aspect, the determining, by the first Morse function value, the first on the first overlapping image The method of determining the first partial minimum point, the first local maximum point, and the first overlapping point on the first overlapping image according to the first Morse function value of the pixel point on the first overlapping image a saddle point; determining a first segmentation sub-image according to each of the first local minimum point, a first local maximum point, and two of the first saddle points; the determining according to the second Morse function value The second divided sub-image on the second overlapping image specifically includes: determining a second local minimum on the second overlapping image according to a second Morse function value of a pixel point on the second overlapping image a value point, a second local maximum point and a second saddle point; a second segmentation sub-image is determined according to each of the second local minimum points, a second local maximum point, and two of the second saddle points.

In conjunction with the fourth possible implementation of the first aspect, in a fifth possible implementation manner of the first aspect, the determining, by the first Morse function value, the first on the first overlapping image After dividing the sub-image, the method further includes: determining a first maximum point of each of the first divided sub-images, and determining, by using a region algorithm, a first duality corresponding to the first overlapping image according to the first maximum point Dividing the sub-image; after determining the second divided sub-image on the second overlapping image according to the second Morse function value, further comprising: determining a second maximum of each of the second divided sub-images a value point, using a region algorithm, determining a second dual segmentation sub-image corresponding to the second overlapping image according to the second maximum point.

In conjunction with the fifth possible implementation of the first aspect, in a sixth possible implementation manner of the first aspect, the background of each of the first divided sub-images in at least two of the first images Selecting a corresponding first spliced sub-image in the image, the method comprising: calculating a smoothing overhead according to an average value of the maximum color difference of the connected first dual-divided sub-images, according to the image center of the first overlapping image to the first Calculating a data overhead for the distance of the dual-divided sub-image; selecting the corresponding first splice sub-image for each of the first dual-divided sub-images according to the smoothing overhead and the data overhead; The second divided sub-image selects the corresponding second splice sub-image in the foreground image of the at least two of the first images, and specifically includes: calculating light according to an average value of maximum color differences of the connected second dual-divided sub-images a sliding overhead, calculating a data overhead according to a distance from an image center of the second overlapping image to the second dual-divided sub-image; selecting each of the second dual-divided sub-images according to the smoothing overhead and the data overhead Corresponding to the second splice sub-image.

With reference to the sixth possible implementation manner of the first aspect, in a seventh possible implementation manner of the first aspect, the performing the first dual-divided sub-image according to the smoothing overhead and the data overhead Selecting the corresponding first spliced sub-image includes: constructing a first map according to the first dual-divided sub-image; determining a weight of each side of the first graph according to the smoothing overhead and the data overhead; And the first dual-divided sub-image is classified according to the weight of each edge of the first figure by using a graph cutting method, and the first stitching corresponding to each of the first dual-divided sub-images is selected according to the classification result. a sub-image; the selecting the corresponding second splice sub-image for each of the second dual-divided sub-images according to the smoothing overhead and the data overhead, comprising: constructing according to the second dual-divided sub-image structure a second graph; determining a weight of each edge of the second graph according to the smoothing overhead and the data overhead; and using the graph cutting method to the second dual segment according to a weight of each edge of the second graph Figure Classified according to the classification result of each of said second dual segmented sub-images corresponding to the selected second sub-image mosaic.

In a second aspect, an embodiment of the present invention provides a device for acquiring a panoramic image, including: an acquiring module, configured to acquire at least two first images; and an extracting module, configured to extract a background image and a foreground image of the first image; a determining module, configured to determine a first overlapping image and a first non-overlapping image of the background image of the at least two first image first images, and to determine a foreground image of the at least two first image first images a second overlapping image and a second non-overlapping image; a segmentation module, configured to segment the first overlapping image to obtain a plurality of first divided sub-images, and divide the second overlapping image to obtain a plurality of second divided sub-images a selection module, configured to select, for each of the first divided sub-images, a corresponding first splice sub-image in a background image of at least two of the first images, and at least two for each of the second divided sub-images Corresponding second splicing sub-images are selected from the foreground image of the first image; a splicing module is configured to perform the first splicing sub-image corresponding to the plurality of the first divided sub-images And obtaining a target background image, and splicing the second spliced sub-image corresponding to the plurality of the second divided sub-images to obtain a target foreground image; and a synthesizing module, configured to: the target background image, the target foreground image, and the Combining the first non-overlapping image and the second non-overlapping image to obtain a mesh Panoramic image.

With reference to the second aspect, in a first possible implementation manner of the second aspect, the foreground template of the first image is determined according to the depth value of the first image; the first foreground template is composed of 0, 1. a matrix, at a same pixel, extracting pixel points of the first image corresponding to the foreground template 1 to form a foreground image of the first image; and extracting a corresponding one of the foreground templates at the same pixel point The pixel of the first image constitutes a background image of the first image.

With reference to the second aspect, or the first possible implementation manner of the second aspect, in a second possible implementation manner of the second aspect, the determining module is specifically configured to: Converting the background image into a first background virtual image, converting each of the first foreground images into a first foreground virtual image by homography conversion; calculating the respective of the first background images according to the first background virtual image a first overlapping image and the first non-overlapping image, and calculating the second overlapping image and the second non-overlapping image of each of the first foreground images according to the first foreground virtual image.

With reference to the second aspect or the first possible implementation manner of the second aspect, or the second possible implementation manner of the second aspect, in a third possible implementation manner of the second aspect, the segmentation module is specifically used to Calculating a first difference value of the first overlapping image on each color channel, performing a Laplace filter transformation and smoothing processing on the first difference value to obtain a first Morse function value; a Morse function value determining the first divided sub-image on the first overlapping image; calculating a second difference image of the second overlapping image on each color channel, sequentially for the second difference image Performing a Laplacian transform transform and smoothing to obtain a second Morse function value; and determining the second divided sub-image on the second overlapping image according to the second Morse function value.

In conjunction with the third possible implementation of the second aspect, in a fourth possible implementation of the second aspect, the segmentation module is further configured to: determine, according to a first mole of pixels on the first overlapping image The sigma function value determines a first local minimum point, a first local maximum point, and a first saddle point on the first overlapping image; according to each of the first local minimum point, a first local maximum point, and Determining, by the two first saddle points, a first divided sub-image; determining a second local minimum point on the second overlapping image according to a second Morse function value of the pixel on the second overlapping image, Two local maximum points and a second saddle point; according to each The second partial minimum point, a second local maximum point, and the two second saddle points determine a second segmentation sub-image.

In conjunction with the fourth possible implementation of the second aspect, in a fifth possible implementation of the second aspect, the segmentation module is further configured to: determine a first maximum value of each of the first segmentation sub-images Point, using a region algorithm, determining a first dual-divided sub-image corresponding to the first overlapping image according to the first maximum point; determining a second maximum point of each of the second divided sub-images, using a region algorithm And determining, according to the second maximum point, a second dual-divided sub-image corresponding to the second overlapping image.

With reference to the fifth possible implementation of the second aspect, in a sixth possible implementation manner of the second aspect, the selecting module is specifically configured to: determine, according to a maximum color difference of the connected first dual-divided sub-images Calculating a smoothing overhead, calculating a data overhead according to a distance from an image center of the first overlapping image to the first dual-divided sub-image; and each of the first duals according to the smoothing overhead and the data overhead Dividing the sub-image to select the corresponding first splice sub-image; calculating a smoothing overhead according to the average of the maximum color differences of the connected second dual-divided sub-images, according to the image center of the second overlapping image to the Calculating a data overhead of the distance of the two dual-divided sub-images; selecting the corresponding second splice sub-image for each of the second dual-divided sub-images according to the smoothing overhead and the data overhead.

In conjunction with the sixth possible implementation of the second aspect, in a seventh possible implementation manner of the second aspect, the selecting module is further configured to: construct a first image according to the first dual-divided sub-image, The first map includes: a plurality of the first dual-divided sub-images; determining a weight of each side of the first graph according to the smoothing overhead and the data overhead; and each edge according to the first graph The weighting is performed by using a graph cutting method to classify the first dual-divided sub-image, and selecting the corresponding first splice sub-image for each of the first dual-divided sub-images according to the classification result; And selecting, by the data overhead, the second splice sub-image corresponding to each of the second dual-divided sub-images, comprising: constructing a second map according to the second dual-divided sub-image, where the second map includes: a plurality of the second dual-divided sub-images; determining a weight of each side of the second graph according to the smoothing overhead and the data overhead; and using a graph cutting method according to the weight of each edge of the second graph For the second pair Classifying the divided sub-images, each of said second dual segmented image of the second sub-splicing corresponding selected sub-image based on the classification result.

The method and device for acquiring a panoramic image according to an embodiment of the present invention divides a first overlapping image of a background image of a first image captured by at least two cameras and a second overlapping image of a foreground image, for each first segmentation The sub-image selects a corresponding first splice sub-image in the background image of the at least two first images, and selects a corresponding second splice sub-image in the foreground image of the at least two first images for each second divided sub-image, The first spliced sub-image is spliced to obtain a target background image, the second spliced sub-image is spliced to obtain a target foreground image, and finally the target background image, the target foreground image, the first non-overlapping image and the second non-overlapping image are synthesized, thereby Get a high resolution target panoramic image with a larger field of view.

DRAWINGS

FIG. 1 is a flowchart of a method for acquiring a panoramic image according to an embodiment of the present invention;

2 is a flowchart of a method for acquiring a panoramic image according to another embodiment of the present invention;

3A is a schematic diagram of a first divided sub-image according to an embodiment of the present invention;

FIG. 3B is a schematic diagram of a first segmentation sub-image according to another embodiment of the present invention; FIG.

4A is a schematic diagram of a target panoramic image according to an embodiment of the present invention;

FIG. 4B is a schematic diagram of a target panoramic image according to another embodiment of the present invention; FIG.

FIG. 5 is a flowchart of a method for acquiring a panoramic image according to still another embodiment of the present invention;

FIG. 6A is a schematic diagram of cutting according to still another embodiment of the present invention; FIG.

6B is a schematic diagram showing a classification result of a graph cut according to still another embodiment of the present invention;

6C is a schematic diagram of a target panoramic image according to another embodiment of the present invention;

FIG. 7 is a flowchart of a method for acquiring a panoramic image according to another embodiment of the present invention;

FIG. 8 is a schematic diagram of a camera array according to still another embodiment of the present invention; FIG.

FIG. 9 is a schematic diagram of a background image mosaic according to another embodiment of the present invention; FIG.

FIG. 10 is a schematic structural diagram of a device for acquiring a panoramic image according to an embodiment of the present invention.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the drawings in the embodiments of the present invention. Is a part of the embodiment of the invention, not all Example. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative efforts are within the scope of the present invention.

FIG. 1 is a flowchart of a method for acquiring a panoramic image according to an embodiment of the present invention. The method is applicable to the field of image processing, and the execution body of the method may be a device for acquiring a panoramic image, where the device may be a computer or the like. The specific steps of the device and the method for acquiring the panoramic image include:

S101: Acquire at least two first images.

Specifically, the computer obtains at least two first images, wherein each of the first images is a captured image at different angles to the same object, and the field of view of the first image is different due to different shooting angles, and the first image is obtained. The calibration data of the camera of the image, wherein the calibration data includes an internal parameter matrix of the camera, an external parameter matrix, the internal reference matrix includes parameters such as resolution and focal length of the camera, and the external parameter matrix includes positional parameters such as translation and rotation of the camera.

S102: Extract a background image and a foreground image of the first image, respectively.

Optionally, determining a foreground template of the first image according to the depth value of the first image, where the foreground template of the first image is a matrix consisting of 0, 1 , and at the same pixel point, extracting the first image corresponding to the foreground template 1 The pixel points constitute a foreground image of the first image. Further, at the same pixel point, the pixel points of the first image corresponding to the complement 1 of the foreground template may be extracted to form a background image of the first image.

Specifically, the depth value of the first image is divided to define a threshold range, and the depth value is within the threshold range and the numbers “1” and “0” are respectively outside the threshold range, and the first image may be according to the depth value. Converted into a matrix consisting of "0" and "1", the formed matrix is a foreground template, and at the same pixel point, the pixel of the first image corresponding to the foreground template 1 is extracted to form a foreground image of the first image. Further, at the same pixel point, the pixel points of the first image corresponding to the complement 1 of the foreground template may be extracted to form a background image of the first image. The method of acquiring the foreground image and the background image of the first image may also be other foreground and background image detecting methods.

S103: Determine a first overlapping image and a first non-overlapping image formed by the background images of the at least two first images, and determine a second overlapping image and a second non-overlapping image formed by the foreground images of the at least two first images.

Specifically, each background image is converted into a background virtual image by a homography transformation, each foreground image is converted into a foreground virtual image by a homography transformation, and a first overlapping image and a first non-image of each background image are calculated according to the background virtual image. The images are superimposed, and then the second superimposed image and the second non-overlapping image of the respective foreground images are calculated from the foreground virtual images. The calculation method of the homography transformation may be determined according to an internal parameter matrix, an outer parameter matrix, and a depth value of each camera, and the depth value may be a depth value of a background image of a certain first image, or may be a customized depth value. Assuming that the normal vector n of the plane in which the background virtual image is located is parallel to the z-axis of the coordinate system in which the i-th camera is located, and the distance between it and the camera is z _bg , then the background image of the first image captured by the camera is The correspondence between the corresponding background virtual images is converted to

Where d denotes [001], K _i denotes the 3x3 internal reference matrix of the i-th camera, where K _v can take the average of all camera internal parameters, and R _v and t _v can be obtained by interpolating all camera external parameters. The method of homography transformation is not limited to the above method. Finally, the first overlapping image and the first non-overlapping image are calculated according to the first background virtual image. In the same manner, the second overlapping image and the second non-overlapping image are calculated according to the first foreground virtual image.

S104: Segmenting the first superimposed image to obtain a plurality of first divided sub-images, and dividing the second superimposed image to obtain a plurality of second divided sub-images.

Specifically, first, a first difference value of the first overlapping image on each color channel is calculated, and a Laplace filter transform and a smoothing process are sequentially performed on the first difference value to obtain a first Morse function value, and then according to the first mole. The function value determines the first divided sub-image on the first overlapping image, and the second overlapping image is also divided to obtain a plurality of second divided sub-images, specifically comprising: calculating a second difference of the second overlapping image on each color channel The value image is subjected to Laplacian transform transform and smoothing processing on the second difference image to obtain a second Morse function value, and then the second divided sub-image on the second overlapping image is determined according to the second Morse function value. Determining, according to the first Morse function value, the first divided sub-image on the first overlapping image comprises: determining a first local minimum on the first overlapping image according to a first Morse function value of the pixel point on the first overlapping image a value point, a first local maximum point, and a first saddle point, and determining a first segmentation sub-image according to each of the first local minimum point, a first local maximum point, and two first saddle points, that is, each The first divided sub-images each include the above four points. Similarly, determining the second divided sub-image on the second overlapping image according to the second Morse function value, specifically: determining, according to the second Morse function value of the pixel point on the second overlapping image, the second on the second overlapping image Local minimum point, second local maximum point and second saddle point, then root A second segmentation sub-image is determined according to each of the second local minimum points, a second local maximum point, and two second saddle points.

Further, after the boundary of the first divided sub-image may be incapable of being aligned with the first overlapping image, after determining the first divided sub-image on the first overlapping image according to the first Morse function value, the method further includes: determining a first maximum point of each first divided sub-image, using a region algorithm, determining a first dual-divided sub-image corresponding to the first overlapping image according to the first maximum point, and determining a second according to the second Morse function value After the second divided sub-image on the superimposed image, the method further includes: determining a second maximum point of each second divided sub-image, and determining, by using a region algorithm, a second corresponding to the second overlapping image according to the second maximum point Dual split sub-images.

S105: Select, for each first segmentation sub-image, a corresponding first tiled sub-image in the background images of the at least two first images, and select, in each of the foreground images of the at least two first images, for each second segmentation sub-image. The second splice image.

Specifically, first, calculating a smoothing overhead according to an average value of maximum color differences of the connected first dual-divided sub-images, calculating a data overhead according to a distance from an image center of the first overlapping image to the first dual-divided sub-image, and then according to the smoothing The overhead and the data overhead are selected for each of the first dual-divided sub-images, and the corresponding first splice sub-images are selected for each of the second divided sub-images in the foreground image of the at least two first images. The second splicing sub-image includes: calculating a smoothing overhead according to an average value of maximum color differences of the connected second dual-divided sub-images, and calculating a data overhead according to a distance from the image center of the second overlapping image to the second dual-divided sub-image, Corresponding second splice sub-images are selected for each second dual-divided sub-image according to smoothing overhead and data overhead.

Further, the corresponding first splice sub-image is selected for each first dual-divided sub-image according to the smoothing overhead and the data overhead, including: first, constructing the first map according to the first dual-divided sub-image, the first graph includes: multiple The first dual-divided sub-image, and secondly, the weight of each edge of the first graph is determined according to the smoothing overhead and the data overhead. Finally, the first dual-divided sub-image is performed by using a graph cutting method according to the weight of each edge of the first graph. Sorting, according to the classification result, selecting a corresponding first splice sub-image for each first dual-divided sub-image, and similarly selecting a corresponding second splice sub-image for each second dual-divided sub-image according to smoothing overhead and data overhead, The method includes: constructing a second map according to the second dual segmentation sub-image, and the second image includes: a plurality of second pairs Evenly dividing the sub-image, determining the weight of each edge of the second graph according to the smoothing overhead and the data overhead, and classifying the second dual-divided sub-image according to the weight of each edge of the second graph, according to the classification result Each second dual-divided sub-image selects a corresponding second splice sub-image.

S106: splicing the first spliced sub-images corresponding to the plurality of first divided sub-images to obtain a target background image, and splicing the second spliced sub-images corresponding to the plurality of second divided sub-images to obtain a target foreground image.

Specifically, after the first spliced sub-image is selected, all the first spliced sub-images are spliced to obtain a target background image, and the second spliced sub-image corresponding to the plurality of second divided sub-images is spliced to obtain a target foreground. image.

S107: Synthesize the target background image, the target foreground image, the first non-overlapping image, and the second non-overlapping image to obtain a target panoramic image.

Specifically, the target image is formed by the synthesis algorithm on the target background image and the target foreground image, and then the target image and the first non-overlapping image and the second non-overlapping image are combined to obtain a target panoramic image.

The present invention provides a method for acquiring a panoramic image, wherein at least a first overlapping image of a background image of at least two first images and a second overlapping image of a foreground image are segmented, at least for each of the first divided sub-images Selecting a corresponding first splice sub-image in the background image of the two first images, and selecting a corresponding second splice sub-image in the foreground image of the at least two first images for each second divided sub-image, the first splice The image is spliced to obtain a target background image, the second spliced sub-image is spliced to obtain a target foreground image, and finally the target background image, the target foreground image, the first non-overlapping image and the second non-overlapping image are combined to obtain a larger view. High resolution target panoramic image of the field range.

FIG. 2 is a flowchart of a method for acquiring a panoramic image according to another embodiment of the present invention. The method may be applied to the field of image processing technology, and the execution subject may be a smart device such as a computer, where the previous embodiment is based on the previous embodiment. The method is mainly the refinement of step S104, and specifically includes the following steps:

S1041: Calculate a first difference value of the first overlapping image on each color channel, perform a Laplace filter transformation and a smoothing process on the first difference, and obtain a first Morse function value.

Specifically, calculating a first difference of the first overlapping image on the red, yellow, and blue color channels, wherein the difference formula is

I ₁ and I ₂ respectively represent the color functions corresponding to the two first images, (x, y) represents the position coordinates of the pixel points, i represents the i-th color channel, and the first overlapping image is measured by the difference formula described above. The misalignment condition is then subjected to Laplacian filter transformation and smoothing processing on the first difference to obtain a first Morse function value.

S1042: Determine a first divided sub-image on the first overlapping image according to the first Morse function value.

Specifically, determining, according to a first Morse function value of a pixel point on the first overlapping image, a first local minimum point, a first local maximum point, and a first saddle point on the first overlapping image, according to each of the A partial minimum point, a first local maximum point, and two first saddle points determine a first segmentation sub-image. FIG. 3A is a schematic diagram of a first segmentation sub-image according to an embodiment of the present invention, and FIG. A first divided sub-image diagram provided by an embodiment, as shown in FIG. 3A and FIG. 3B, each of the four points defines a first divided sub-image, wherein the four points are: a first local minimum point, and a first Local maximum point and two first saddle points. In FIG. 3A and FIG. 3B, the smoothing parameters selected in the smoothing process are different, so the size of the obtained first divided sub-images is not the same. For example, the smoothing parameter used in FIG. 3A is 10, and the smoothing parameters used in FIG. 3B are used. A larger smoothing parameter of 20, a larger standard deviation of Gaussian smoothing, results in a larger first segmented sub-image.

Further, FIG. 4A is a schematic diagram of a target panoramic image according to an embodiment of the present invention. As shown in FIG. 4A, after obtaining the first divided sub-image, the first dual-divided sub-image is obtained by using the region algorithm, so there is a first segmentation. The boundary of the sub-image is not aligned with the first overlapping image, so there may be a splicing slot. FIG. 4B is a schematic diagram of a target panoramic image according to another embodiment of the present invention, since the first overlapping image is determined according to the first Morse function value. After the first divided sub-image, the method further includes: determining a first maximum point of each first divided sub-image, and determining, by using a region algorithm, a first dual-divided sub-image corresponding to the first overlapping image according to the first maximum point, The method can solve the problem that the boundary of the first divided sub-image cannot be aligned with the first overlapping image, and thus the obtained target panoramic image does not have a stitching gap.

Further, a second difference image of the second overlapping image on each color channel may be calculated, and the second difference image is sequentially subjected to Laplacian transform transform and smoothing processing to obtain a second Morse function value.

Specifically, calculating a second difference of the second overlapping image on the red, yellow, and green color channels, where the difference formula is

I ₁ and I ₂ respectively represent the color functions corresponding to the two cameras providing the second image, (x, y) represents the coordinate position of the pixel point, and i represents the i-th color channel, which is measured by the difference formula described above. The two overlapping images are misaligned, and then the second difference is sequentially subjected to Laplacian transform and smoothing to obtain a second Morse function value. Then, the second divided sub-image on the second overlapping image is determined according to the second Morse function value. Determining, according to a second Morse function value of the pixel point on the second overlapping image, a second local minimum point on the second overlapping image, a second local maximum point and a second saddle point, according to each second local minimum point A second partial maximum point and two second saddle points determine a second divided sub-image. After determining the second divided sub-image on the second overlapping image according to the second Morse function value, further comprising: determining a second maximum point of each second divided sub-image, using a region algorithm, determining according to the second maximum point The second dual segmentation sub-image corresponding to the second overlapping image.

The embodiment provides a method for acquiring a panoramic image, wherein the first and second overlapping images on the first overlapping image are determined by calculating a first Morse function value and a second Morse function value. The second divided sub-image is further divided into a first overlapping image and a second overlapping image by using a region algorithm, so that the first overlapping image and the second overlapping image are aligned and segmented, and then the stitching method is further obtained. High resolution target panoramic image of large field of view.

FIG. 5 is a flowchart of a method for acquiring a panoramic image according to another embodiment of the present invention. The method may be applied to the field of image processing technology, and the execution subject may be a smart device such as a computer, where the first embodiment is based on the first embodiment. The method is mainly for the refinement of step S105, and specifically includes the following steps:

S1051: Calculate a smoothing overhead according to an average value of maximum color differences of the connected first dual-divided sub-images, and calculate a data overhead according to a distance from an image center of the first overlapping image to the first dual-divided sub-image.

Specifically, for each pixel point in the connected first dual-divided sub-image, a color difference is obtained on the red, yellow and blue color channels, and the color difference indicates that the first background virtual image captured by the two cameras is in each pixel. The color difference on the point, the maximum color difference is obtained, and finally the maximum color difference on the three color channels is averaged, the smoothing overhead is determined according to the average value, and, according to the image center of the first overlapping image to the first dual segmentation sub-image The distance calculates the data overhead.

S1052: Select a corresponding first splice sub-image for each first dual-divided sub-image according to the smoothing overhead and the data overhead.

Optionally, constructing the first map according to the first dual-divided sub-image, determining the weight of each edge of the first graph according to the smoothing overhead and the data overhead, and using the graph cutting method to the first according to the weight of each edge of the first graph The dual-divided sub-images are classified, and corresponding first splice sub-images are selected for each first dual-divided sub-image according to the classification result. For example, if there are two cameras TL and TR, the cutting line of the first dual-divided sub-image is determined by the graph cutting method. Since the first graph includes the first dual-cut slice, the terminal node is further included, and the terminal node is used for FIG. 6A is a schematic diagram of a graph cut according to still another embodiment of the present invention. The node s and the node t respectively represent two cameras. If a first dual split sub-image and the node s are on the same side of the cutting line, The corresponding area image in the first background image captured by the camera corresponding to the first dual-divided sub-image selection node s is the first splicing sub-image, and FIG. 6B is a schematic diagram of the classification result of the dicing cut according to another embodiment of the present invention. The sub-image 601 is composed of a first divided sub-image formed by a first image captured by the camera TL, and the sub-image 602 is composed of a first divided sub-image formed by the first image captured by the camera TR, according to the segmentation As a result, the first splice sub-image may be selected. For example, the first splice sub-image corresponding to one of the first divided sub-images in the sub-image 601 is taken from the first image captured by the TL. Part of the image of the position, the first stitching sub-image is stitched to form a target background image, and the second stitching sub-image can be selected in the same manner to finally form a target background image. FIG. 6C is a target panoramic image according to another embodiment of the present invention. The schematic diagram, as shown in FIG. 6C, is to obtain a target panoramic image by synthesizing the target background image, the target foreground image, the first non-overlapping image, and the second non-overlapping image.

Further, the smoothing overhead may also be calculated according to the average of the maximum color differences of the connected second dual-divided sub-images, and the data overhead is calculated according to the distance from the image center of the second overlapping image to the second dual-divided sub-image.

Specifically, for each pixel point in the connected second dual-divided sub-image, a color difference is obtained on the red, yellow and green color channels, and the color difference indicates that the first foreground virtual image captured by the two cameras is in each pixel. The color difference at the point, the maximum color difference is obtained, and finally the maximum color difference on the three color channels is averaged, the smoothing overhead is determined according to the average value, and, according to the image center of the second overlapping image to the second dual segmentation sub-image The distance calculates the data overhead. Then, according to the smoothing overhead and the data overhead, the corresponding first is selected for each second dual-divided sub-image. Two stitched sub-images. Optionally, constructing a second map according to the second dual-divided sub-image, determining a weight of each edge of the second graph according to the smoothing overhead and the data overhead, and using the graph cutting method to the second according to the weight of each edge of the second graph The dual-divided sub-images are classified, and corresponding second splice sub-images are selected for each second dual-divided sub-image according to the classification result.

The embodiment provides a method for acquiring a panoramic image, which mainly includes: selecting a corresponding first splice sub-image and each second dual split for each first dual-divided sub-image by calculating a smooth overhead and a data overhead. The sub-image selects the corresponding second splice sub-image, and achieves effective splicing on the basis of the above-described graph cutting method, thereby obtaining a high-resolution target panoramic image with a larger field of view range.

FIG. 7 is a flowchart of a method for acquiring a panoramic image according to another embodiment of the present invention. The method may be applied to the technical field of image processing, and the execution subject may be a smart device such as a computer. On the basis of the foregoing embodiment, The embodiment exemplifies a method for acquiring a panoramic image. FIG. 8 is a schematic diagram of a camera array according to another embodiment of the present invention. It is assumed that there are four cameras TL, TR, BL, and BR, and the camera array of 2*2 is the same. The object is photographed, and the shooting angles thereof are different. The depth value may be a depth value of the background image of a certain first image, or may be a customized depth value. The method for acquiring the panoramic image specifically includes the following steps:

S701: Acquire four first images and corresponding four foreground templates.

Specifically, the computer obtains four first images, wherein the four first images are respectively composed of four cameras TL, TR, BL, and BR, and the camera array of 2*2 is taken by the same object, and the shooting angle thereof is not The same, so the field of view of the first image is different, and the calibration data of the camera for capturing the first image is obtained, wherein the calibration data includes an internal parameter matrix of the camera, an external parameter matrix, and the internal reference matrix includes parameters such as resolution and focal length of the camera. The outer parameter matrix includes positional parameters such as translation, rotation, and the like of the camera. Dividing the depth value of the first image to define a threshold range, where the depth value is within the threshold range and outside the threshold range respectively correspond to binary "1" and "0", then the first image may be converted into one according to the depth value a matrix consisting of "0" and "1"

S702: Extract a foreground image and a background image of each first image by using a foreground template.

Specifically, the depth value of the first image is divided to define a threshold range, where the depth value is within the threshold range and outside the threshold range respectively correspond to binary “1” and “0”, then The first image may be converted into a matrix consisting of “0” and “1” according to the depth value, and the formed matrix is a foreground template, and at the same pixel point, the pixel of the first image corresponding to the foreground template 1 is extracted, and the pixel is formed. a foreground image of the first image, and further, at the same pixel point, extracting pixel points of the first image corresponding to the complement 1 of the foreground template to form a background image of the first image, wherein the foreground of the first image is acquired The image and background image methods can also be other foreground, background image detection methods.

S703: Determine a depth value of the background image, and a homography transformation of the background image to the virtual background image.

The method for determining the depth value may be a depth value of the background image of a certain first image as a depth value of the background image, or may be a customized depth value. In addition, the homography transformation is determined by: assuming that the normal vector n of the plane in which the background virtual image is located is parallel to the z-axis of the coordinate system of the i-th camera, and the distance between the camera and the camera is z _bg , then the camera The homography between the captured background image of the first image and the corresponding first background virtual image is converted to

Where d denotes [001], K _i denotes the 3x3 internal reference matrix of the i-th camera, where K _v can take the average of all camera internal parameters, and R _v and t _v can be obtained by interpolating all camera external parameters. The method of homography transformation is not limited to the above method.

S704: Perform segmentation based on the first overlapping image of the first image captured by the cameras TL and TR respectively, and perform splicing of the background image based on the segmentation result, and select the first spliced sub-image to form a target background image.

Specifically, first, a first difference value of the first overlapping image on each color channel is calculated, and a Laplace filter transform and a smoothing process are sequentially performed on the first difference value to obtain a first Morse function value, and then according to the first mole. The sigma function value determines a first segmentation sub-image on the first superimposed image, wherein determining the first segmentation sub-image on the first superimposed image according to the first Morse function value specifically includes: according to the pixel point on the first superimposed image a Morse function value determining a first local minimum point, a first local maximum point, and a first saddle point on the first overlapping image, according to each of the first local minimum point, a first local maximum point, and two The first saddle point determines a first segmentation sub-image, that is to say each of the first segmentation sub-images includes the above four points. Further, after the boundary of the first divided sub-image may be incapable of being aligned with the first overlapping image, after determining the first divided sub-image on the first overlapping image according to the first Morse function value, the method further includes: determining First maximum of each first segmented sub-image Point, using a region algorithm, determining a first dual segmentation sub-image corresponding to the first overlapping image according to the first maximum point. Then, selecting a corresponding first splice sub-image in the background images of the at least two first images for each of the first divided sub-images, forming a target background image, including: according to a maximum color difference of the connected first dual-divided sub-images Calculating a smoothing overhead by an average value, calculating a data overhead according to a distance from the image center of the first overlapping image to the first dual-divided sub-image, and then selecting, for each of the first dual-divided sub-images, according to the smoothing overhead and the data overhead Corresponding to the first splice sub-image,

Further, the corresponding first splice sub-image is selected for each first dual-divided sub-image according to the smoothing overhead and the data overhead, including: first, constructing the first map according to the first dual-divided sub-image, the first graph includes: multiple The first dual-divided sub-image, and secondly, the weight of each edge of the first graph is determined according to the smoothing overhead and the data overhead. Finally, the first dual-divided sub-image is performed by using a graph cutting method according to the weight of each edge of the first graph. Classification, selecting a corresponding first splice sub-image for each first dual-divided sub-image according to the classification result. Finally, the first splice sub-images described above are spliced to form a target background image.

S705: Perform segmentation based on the first overlapping image of the first image captured by the cameras BL and BR respectively, and perform splicing of the background image based on the segmentation result, and select the first spliced sub-image to form a target background image.

This step is similar to S704 and will not be described here.

S706: segment the superimposed image of the target background image formed by the first image captured by the TL and the TR and the target background image formed by the first image captured by the BL and the BR, and form a final target background image based on the segmentation result.

Specifically, FIG. 9 is a schematic diagram of background image mosaic according to another embodiment of the present invention. As shown in FIG. 9, the first line of images is the first image captured by cameras TL, TR, BL, and BR, and the second line is left. The image is the target background image formed by the first image captured by TL and TR, the image on the right is the target background image formed by the first image captured by BL and BR, and the image on the third line is taken on TL and TR. The target background image formed by the first image and the overlapping image of the target background image formed by the first image captured by the BL and the BR are segmented, and a final target background image is formed based on the segmentation result.

S707: Determine a depth value of the foreground image, and a homography transformation of the foreground image to the virtual foreground image.

S708: Perform segmentation based on the second overlapping image of the first image captured by the cameras TL and TR respectively, and perform splicing of the foreground image based on the segmentation result to form a target foreground image.

Since the splicing of the foreground image does not significantly increase the size of the field of view, it is possible to splicing only the foreground image of the first image taken by any two of the above cameras.

S709: Determine a target panoramic image according to the first splice sub image and the second splice sub image.

The target non-overlapping image determined by the target background image, the target foreground image, and the background image of the first image, and the second non-overlapping image determined by the foreground image of the first image are synthesized, and finally the target panoramic image is obtained. The target background image obtained by the background image stitching process shown in FIG. 9 is similar to the foreground stitching process of the background image stitching process shown in FIG. 9, the obtained target foreground image, and simultaneously determined for the background image of the first image. The first non-overlapping image, the second non-overlapping image determined by the foreground image of the first image are synthesized, and finally the target panoramic image is obtained.

The present invention provides a method for acquiring a panoramic image, which includes dividing a first overlapping image to form a first divided sub-image, and equally dividing the second overlapping image to form a second divided sub-image for each first The segmented sub-image is selected corresponding to the first spliced sub-image, and the corresponding second spliced sub-image is selected for each second sub-sub-image, the first spliced sub-image is spliced to obtain the target background image, and the second spliced sub-image is spliced The target foreground image is finally combined with the target background image, the target foreground image, the first non-overlapping image, and the second non-overlapping image to obtain a high-resolution target panoramic image of a larger field of view.

FIG. 10 is a schematic structural diagram of a device for acquiring a panoramic image according to an embodiment of the present invention. The device may be a smart device such as a computer. The device for acquiring a panoramic image includes: an acquiring module 111, configured to acquire at least two first An image extraction module 112, configured to extract a background image and a foreground image of the first image; a determining module 113, configured to determine a first overlapping image and a first non-overlapping image formed by the background images of the at least two first images, and determine a second overlapping image and a second non-overlapping image formed by the foreground image of the at least two first images; wherein the extracting module 112 is configured to: determine a foreground template of the first image according to the depth value of the first image; the first foreground template is a matrix consisting of 0,1, at the same pixel point, extracting pixel points of the first image corresponding to the foreground template 1 to form a foreground image of the first image; at the same pixel point, extracting the complement 1 of the foreground template The pixel of the first image, which constitutes the background of the first image image. The determining module 113 is specifically configured to: convert each first background image into a first background virtual image by a homography conversion, convert each first foreground image into a first foreground virtual image by a homography conversion; The image calculates a first overlap image and a first non-overlapping image of each of the first background images, and calculates a second overlap image and a second non-overlapping image of each of the first foreground images according to the first foreground virtual image. The segmentation module 114 is configured to segment the first overlapping image to obtain a plurality of first divided sub-images, and divide the second overlapping image to obtain a plurality of second divided sub-images; and the selecting module 115 is configured to The sub-image selects a corresponding first splice sub-image in the background images of the at least two first images, and selects a corresponding second splice image in the foreground image of the at least two first images for each second divided sub-image; The module 116 is configured to splicing a first spliced sub-image corresponding to the plurality of first divided sub-images to obtain a target background image, and splicing the second spliced sub-image corresponding to the plurality of second divided sub-images to obtain a target foreground image; The module 117 is configured to synthesize the target background image, the target foreground image, the first non-overlapping image, and the second non-overlapping image to obtain a target panoramic image.

The device for acquiring the panoramic image of the embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 1 , and the implementation principle and technical effects thereof are similar, and details are not described herein again.

On the basis of the previous embodiment, the segmentation module 114 is specifically configured to: calculate a first difference value of the first overlapping image on each color channel, and perform a Laplace filter transformation on the first difference sequentially. Smoothing to obtain a first Morse function value; determining a first segmentation sub-image on the first overlay image according to the first Morse function value; and further for calculating a second difference of the second overlay image on each color channel The value image is subjected to Laplacian transform transform and smoothing processing on the second difference image to obtain a second Morse function value; and the second divided sub-image on the second superimposed image is determined according to the second Morse function value. Further, the segmentation module 114 is further configured to: determine, according to a first Morse function value of the pixel point on the first overlapping image, a first local minimum point, a first local maximum point, and a first saddle point on the first overlapping image Determining a first segmentation sub-image according to each of the first local minimum point, a first local maximum point, and the two first saddle points; and further for using a second Morse function of the pixel points on the second overlapping image The value determines a second local minimum point on the second overlapping image, a second local maximum point and a second saddle point; and a second local maximum point and two second saddle points are determined according to each of the second local minimum points A second segmented sub-image. The segmentation module 114 is further configured to: determine a first maximum point of each first segmentation sub-image, and determine, according to the first maximum point, using a region algorithm Determining a first dual-divided sub-image corresponding to the first overlapping image; determining a second maximum point of each second divided sub-image, determining a second dual-divided sub-correlation corresponding to the second overlapping image according to the second maximum point using a region algorithm image.

The device for acquiring the panoramic image in this embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 2, and the implementation principle and technical effects are similar, and details are not described herein again.

Further, based on the apparatus embodiment shown in FIG. 10, the selecting module 115 is specifically configured to: calculate a smoothing overhead according to an average value of maximum color differences of the connected first dual-divided sub-images, according to the image of the first overlapping image Calculating a data overhead from a distance from the center to the first dual-divided sub-image; selecting a corresponding first splice sub-image for each first dual-divided sub-image according to smoothing overhead and data overhead; according to a maximum color of the connected second dual-divided sub-image The average value of the difference is used to calculate a smoothing overhead, and the data overhead is calculated according to the distance from the image center of the second overlapping image to the second dual-divided sub-image; and the second stitching is selected for each second dual-divided sub-image according to the smoothing overhead and the data overhead. Sub image. In addition, the selecting module 115 is further configured to: construct a first map according to the first dual-divided sub-image, where the first image includes: a plurality of first dual-divided sub-images; and determining, according to the smoothing overhead and the data overhead, the right of each side of the first graph The first dual-divided sub-image is classified according to the weight of each edge of the first graph by using a graph cutting method, and the corresponding first splice sub-image is selected for each first dual-divided sub-image according to the classification result; according to the smoothing overhead And the data overhead is: selecting a second second sub-segment image for each second dual-divided sub-image, comprising: constructing a second image according to the second dual-divided sub-image, the second image comprising: a plurality of second dual-divided sub-images; Smoothing overhead and data overhead determine the weight of each edge of the second graph; according to the weight of each edge of the second graph, the second dual-divided sub-image is classified by a graph cutting method, and each second dual partition is segmented according to the classification result. The sub image selects a corresponding second splice sub image.

The apparatus for acquiring the panoramic image of the embodiment may be used to implement the technical solution of the method embodiment shown in FIG. 5, and the implementation principle and technical effects are similar, and details are not described herein again.

It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and are not limited thereto; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that The technical solutions described in the foregoing embodiments are modified, or the equivalents of the technical features are replaced. The modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

A method for acquiring a panoramic image, comprising:

Obtaining at least two first images;

Extracting a background image and a foreground image of the first image, respectively;

Determining a first overlapping image and a first non-overlapping image formed by the background images of the at least two first images;

Determining a second overlapping image and a second non-overlapping image formed by the foreground image of the at least two first images;

Dividing the first overlapping image to obtain a plurality of first divided sub-images, and dividing the second overlapping image to obtain a plurality of second divided sub-images;

Selecting, for each of the first divided sub-images, a corresponding first splice sub-image in the background images of the at least two first images, and the at least two first images for each of the second divided sub-images Selecting a corresponding second splice image in the foreground image;

And splicing the first spliced sub-images corresponding to the plurality of the first sub-sub-images to obtain a target background image, and splicing the second spliced sub-images corresponding to the plurality of the second sub-sub-images to obtain a target foreground image;

The target background image, the target foreground image, the first non-overlapping image, and the second non-overlapping image are combined to obtain a target panoramic image.
The method according to claim 1, wherein the extracting the background image and the foreground image of the first image respectively comprises:

Determining a foreground template of the first image according to a depth value of the first image;

The foreground template of the first image is a matrix composed of 0,1, and at the same pixel point, the pixel points of the first image corresponding to the foreground template 1 are extracted to form a foreground image of the first image;

At the same pixel point, the pixel points of the first image corresponding to the complement 1 of the foreground template are extracted to form a background image of the first image.
The method according to claim 1 or 2, wherein the determining the first overlapping image and the first non-overlapping image of the background image formed by the at least two first images; determining the at least two first The second overlapping image and the second non-overlapping image formed by the foreground image of the image specifically include:

Converting each of the background images into a background virtual image by a homography transformation, converting each of the foreground images into a foreground virtual image by the homography transformation;

Calculating the first overlapping image and the first non-overlapping image of each of the background images according to the background virtual image, calculating the second overlapping image of each of the foreground images according to the foreground virtual image, and the The second non-overlapping image.
The method according to any one of claims 1-3, wherein the segmenting the first overlapping image to obtain a plurality of first divided sub-images comprises:

Calculating a first difference value of the first overlapping image on each color channel, performing Laplacian filtering transformation and smoothing processing on the first difference, to obtain a first Morse function value;

Determining the first divided sub-image on the first overlapping image according to the first Morse function value.
The method according to claim 4, wherein the determining the first divided sub-image on the first overlapping image according to the first Morse function value comprises:

Determining, according to a first Morse function value of the pixel point on the first overlapping image, a first local minimum point, a first local maximum point, and a first saddle point on the first overlapping image;

A first segmentation sub-image is determined according to each of the first local minimum point, a first local maximum point, and two of the first saddle points.
The method according to claim 5, wherein after determining the first divided sub-image on the first overlapping image according to the first Morse function value, the method further comprises:

Determining a first maximum point of each of the first divided sub-images, and determining, by using a region algorithm, a first dual-divided sub-image corresponding to the first overlapping image according to the first maximum point.
The method according to claim 6, wherein the selecting the first splice sub-image in the background image of the at least two of the first images for each of the first divided sub-images comprises:

Calculating a smoothing overhead according to an average value of the maximum color differences of the connected first dual-divided sub-images, and calculating a data overhead according to a distance from an image center of the first overlapping image to the first dual-divided sub-image;

And corresponding to the first splice sub-image for each of the first dual-divided sub-images according to the smoothing overhead and the data overhead.
The method according to claim 7, wherein the selecting the corresponding first splice sub-image for each of the first dual-divided sub-images according to the smoothing overhead and the data overhead comprises:

Constructing a first map according to the first dual segmentation sub-image;

Determining weights of each side of the first graph according to the smoothing overhead and the data overhead;

And the first dual-divided sub-image is classified according to the weight of each edge of the first figure by using a graph cutting method, and the first stitching corresponding to each of the first dual-divided sub-images is selected according to the classification result. Sub image.
A device for acquiring a panoramic image, comprising:

Obtaining a module, configured to acquire at least two first images;

An extraction module, configured to extract a background image and a foreground image of the first image;

a determining module, configured to determine a first overlapping image and a first non-overlapping image formed by the background image of the at least two first images;

The determining module is further configured to determine a second overlapping image and a second non-overlapping image of the foreground image of the at least two first images;

a segmentation module, configured to divide the first overlapping image to obtain a plurality of first divided sub-images, and divide the second overlapping image to obtain a plurality of second divided sub-images;

a selection module, configured to select, for each of the first divided sub-images, a corresponding first splice sub-image in at least two background images of the first image, and at least two for each of the second divided sub-images Selecting a corresponding second splice sub-image in the foreground image of the first image;

a splicing module, configured to splicing a first splicing sub-image corresponding to the plurality of the first sub-sub-images to obtain a target background image, and splicing the second splicing sub-image corresponding to the second sub-sub-images to obtain a target Foreground image

And a synthesizing module, configured to synthesize the target background image, the target foreground image, the first non-overlapping image, and the second non-overlapping image to obtain a target panoramic image.
The device according to claim 9, wherein the extraction module is specifically configured to:

Determining a foreground template of the first image according to a depth value of the first image;

The first foreground template is a matrix composed of 0, 1. At the same pixel point, the pixel points of the first image corresponding to the foreground template 1 are extracted to form a foreground image of the first image;

At the same pixel point, the pixel points of the first image corresponding to the complement 1 of the foreground template are extracted to form a background image of the first image.
The device according to claim 9 or 10, wherein the determining module is specifically configured to:

Converting each of the first background images into a first background virtual image by a homography conversion, converting each of the first foreground images into a first foreground virtual image by a homography conversion;

Calculating the first overlapping image and the first non-overlapping image of each of the first background images according to the first background virtual image, and calculating, by the first foreground virtual image, each of the first foreground images The second overlapping image and the second non-overlapping image are described.
The device according to any one of claims 9-11, wherein the segmentation module is specifically configured to:

Calculating a first difference value of the first overlapping image on each color channel, performing Laplacian filtering transformation and smoothing processing on the first difference, to obtain a first Morse function value;

Determining the first divided sub-image on the first overlapping image according to the first Morse function value.
The device according to claim 12, wherein the segmentation module is further configured to:

Determining, according to a first Morse function value of the pixel point on the first overlapping image, a first local minimum point, a first local maximum point, and a first saddle point on the first overlapping image;

A first segmentation sub-image is determined according to each of the first local minimum point, a first local maximum point, and two of the first saddle points.
The device according to claim 13, wherein the segmentation module is further configured to:

Determining a first maximum point of each of the first divided sub-images, and determining, by using a region algorithm, a first dual-divided sub-image corresponding to the first overlapping image according to the first maximum point.
The device according to claim 14, wherein the selection module is specifically configured to:

Calculating a smoothing overhead according to an average value of the maximum color differences of the connected first dual-divided sub-images, and calculating a data overhead according to a distance from an image center of the first overlapping image to the first dual-divided sub-image;

And corresponding to the first splice sub-image for each of the first dual-divided sub-images according to the smoothing overhead and the data overhead.
The device according to claim 15, wherein the selection module is further configured to:

Constructing a first map according to the first dual-divided sub-image, the first map comprising: a plurality of the first dual-divided sub-images;

Determining weights of each side of the first graph according to the smoothing overhead and the data overhead;

And the first dual-divided sub-image is classified according to the weight of each edge of the first figure by using a graph cutting method, and the first stitching corresponding to each of the first dual-divided sub-images is selected according to the classification result. Sub image.