US20050262067A1 - Method of searching multimedia data - Google Patents
Method of searching multimedia data Download PDFInfo
- Publication number
- US20050262067A1 US20050262067A1 US11/179,511 US17951105A US2005262067A1 US 20050262067 A1 US20050262067 A1 US 20050262067A1 US 17951105 A US17951105 A US 17951105A US 2005262067 A1 US2005262067 A1 US 2005262067A1
- Authority
- US
- United States
- Prior art keywords
- image
- feature
- information
- color
- texture
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Abandoned
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5838—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/40—Information retrieval; Database structures therefor; File system structures therefor of multimedia data, e.g. slideshows comprising image and additional audio data
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5854—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using shape and object relationship
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
- G06F16/5862—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content using texture
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/40—Extraction of image or video features
- G06V10/56—Extraction of image or video features relating to colour
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
- G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
- G06V10/758—Involving statistics of pixels or of feature values, e.g. histogram matching
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99942—Manipulating data structure, e.g. compression, compaction, compilation
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99943—Generating database or data structure, e.g. via user interface
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99944—Object-oriented database structure
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10—TECHNICAL SUBJECTS COVERED BY FORMER USPC
- Y10S—TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y10S707/00—Data processing: database and file management or data structures
- Y10S707/99941—Database schema or data structure
- Y10S707/99944—Object-oriented database structure
- Y10S707/99945—Object-oriented database structure processing
Definitions
- the present invention relates to a method of searching multimedia data, and more particularly to a method of searching multimedia data more accurately by utilizing user feedback.
- the digital image signal processing technology may be used in a search system for automatically editing only the face of a specific character in a moving picture file of a movie or drama, in a security system for permitting access to only persons registered in the system, or in a search system for searching a particular data from an image or video detected by a detection system.
- the performance of such systems basically depend on the accuracy and speed of detecting or searching the desired object. Accordingly, various image searching methods have been proposed in the related art.
- a user In another searching method entitled “Virage image search engine” (www.virage.com), a user directly inputs the level of importance for features such as the color, texture and shape by assigning weight values. Although an image may be searched according to an importance or a feature using this method, it may be difficult for a user to determine the weights of features.
- Yong Rui in “Relevance feedback techniques in interactive” SPIE Vol. 3312 discloses a method in which images similar to a reference image are found and the importance of features or weight for features are automatically obtained by calculating the similarities among the found images.
- the weight importance information is not maintained after a search for a specific image is finished and must be calculated for each image search, even for a same image.
- an image may be divided into a plurality of regions and a representative color of each region may be utilized as a feature information of the image, or a whole color histogram of the image may be utilized as a feature information. Thereafter, two images are compared to calculate a similarity based upon the feature information and a determination is made whether the two images are similar.
- the image search methods in the related art may utilize weights of features such as color, texture, or shape.
- weights of feature elements are not taken into consideration. Accordingly, the image search methods in related art has the limitations in intellectually training and developing weights for searching, resulting in relatively longer searching period to obtain search results and a deterioration of the reliability of the search results.
- an object of the present invention is to solve at least the problems and disadvantages of the related art.
- An object of the present invention is to provide a method of searching multimedia data by automatically updating weights of features included in a specified image and/or weights of feature elements, and by applying the updated weights to search for the specified object.
- Another object of the present invention is to provide a method or searching multimedia data which constructs image characteristics corresponding to the types of features included in a specified image by analyzing and classifying the judgement standards applied when the user searches the image, and adjusts the feature information set by taking into consideration weights of the features and weights of feature elements during a following image search.
- Still another object of the present invention is to provide a feature structure to be included in a multimedia data to effectively search an image.
- a method of searching multimedia data in a multimedia data search system comprises searching for a reference multimedia data selected by a user; receiving user feedback of relevance information for the searched multimedia data; determining importance of respective feature elements of features included in the multimedia data according to the relevance information; re-performing the search for the reference multimedia data by updating the importance of said respective elements if the user requests an additional search; and updating previous importance to new importance obtained and maintaining the updated importance degrees.
- a method of searching multimedia data in a multimedia data search system comprises receiving an inquiry into previously searched multimedia data; analyzing a judgement standard for the multimedia data searched during the inquiry; constructing image characteristics using at least one feature included in the multimedia data using an analysis result of the judgement standard; and adjusting importance of the image characteristics and re-performing a search of the multimedia data if a user requests an additional search.
- the present invention also provides a feature structure of multimedia data comprising a first information representing a feature of the multimedia data; a second information representing a regional feature of the multimedia data; and a third information representing importance of the first and second information.
- FIG. 1 shows features in an image by a histogram
- FIG. 2 shows an image represented by a local grids
- FIG. 3 is a flowchart of a multimedia data search process according to a first embodiment of the present invention
- FIGS. 4 and 5 are flowcharts of a multimedia data search process of FIG. 2 , where an initial search does not result in a desired image;
- FIG. 6 is a basic structure of image characteristics for an image search according to a second embodiment of the present invention.
- FIG. 7 is image characteristics of FIG. 6 wherein the feature information is constructed using a color and texture
- FIGS. 8 to 11 are different embodiments of the image characteristics
- FIG. 1 is a color histogram of an image by which the colors in the image is grouped into 24 color elements.
- FIG. 2 is an image represented by local grids.
- an image is divided into n*m grid regions and each grid region may be represented either by a regional color histogram or by a color representing the and region.
- the degree of importance of each grid region or the extent that a grid region affects a search may be represented by assigning weights to each of n*m grid regions.
- a predetermined threshold value determined by the system may be utilized as a cap such that certain grid regions has no influence in the search. Namely, if a grid region has a weight which exceeds the predetermined threshold value, the grid region is utilized in the search. Otherwise, the and region is processed as a “Don't care” region which does not affect the image search.
- FIG. 3 is a flowchart of a multimedia data search process when a plurality of reference images are selected and weights of the respective types or elements of features included in a specified image are assigned or updated according to the first embodiment of the present invention.
- a user selects a plurality of reference images (step S 301 ) representing the specified image to be searched. Thereafter, the system judges and determines the importance of the feature elements included in the reference images.
- the system measures the similarities of the features included in the reference images (step S 302 ), and determines the weights of each feature according to the measured similarities of the features (step S 303 ). Also, the system measures the similarities of the feature elements in each feature included in the reference images (step S 304 ), and determines the weights of feature elements in each respective feature according to the measured similarities of the feature elements (step S 305 ).
- the system re-performs the image search by using the importance, i.e. the weights of the features and the weights of the feature elements in the respective features (step S 306 ).
- the system may use either the weights of the features or the weights of the feature elements, or both.
- the weights of the features and feature elements of respective features are determined as follows.
- the search system determines weights of the features and the weights of the feature elements in the respective features by measuring the similarities of features of the selected reference image list and the similarities or the feature elements in the respective features.
- the weights of the features are calculated by Equations 1a, 1b, and 1c below.
- n denotes the reference number
- m denotes the number of features used for measuring the similarity
- Weight_k denotes the weight of the k-th feature
- Simi(i,j,k) denotes the similarity between the I-th reference image and the j-th reference image when the k-th feature is used
- Cont(k) denotes how much the k-th feature contributes to raise the similarity.
- Weights of the respective features Weight_k increases as the similarities rises since the similarities among the reference images are calculated based upon the respective features and the feature having the highest similarity acts as the most important factor.
- weights of the feature elements are determined by Equations 2a and 2b using the similarities of the feature elements in the respective features or the reference images.
- Weight of element i: w i af I (i) [2a]
- the values of a, p, and q denote constants
- m i denotes an average of an element i in the reference image list
- v denotes the distribution of an element i in the reference image list.
- the weight w i of a feature element is inversely proportional to the distribution of the corresponding feature element and is proportional to an average value of the corresponding feature element component. Therefore, a feature element having a large average value acts as an important factor even if the distribution of the corresponding feature element is wide.
- FIG. 4 is a flowchart showing a multimedia data search process for when a plurality of reference images are and are not selected.
- the image search is performed using features with equivalent weights.
- other reference images similar to the specified image to be searched are selected and added to a reference image list. Then, the weights of the respective features and/or feature elements are updated using the reference image list.
- step S 408 when a user selects a plurality of reference images (steps S 401 and S 402 ) the process (steps S 404 -S 408 ) is the same as described with reference to steps S 302 -S 306 of FIG. 3 . Thus, an explanation will be omitted. Thereafter, if the user is satisfied with the result of the image search (step S 408 ), the search operation terminates. However, if the user is not satisfied with the search result, the user selects other reference image(s) similar to the specified image to be searched among the resultant images of the search (step S 411 ). On the other hand, when a user selects only one reference image, the image search is performed using features with weights as assigned (step S 403 ).
- the search is performed using features with equal weights assigned. If the user is not satisfied with the search result, the user selects other reference image(s) similar to the specified image to be searched among the resultant images of the search (step S 411 ).
- the system adds the selected other reference image(s) to an initial reference image list managed by the system (step S 412 ).
- the initial reference image list includes the reference image(s) selected in step S 401 .
- the system measures the similarities of the respective features and/or feature elements in the selected reference images (step S 413 ), and determines the weights of the respective features and/or feature elements in the selected reference image by Equations 1a 1c, 2a and 2b (step S 414 ).
- the system updates the weights of the respective features and/or feature elements, and re-performs the image search utilizing the updated weights (step S 415 ).
- FIG. 5 is a flowchart showing another multimedia data search process for when a plurality of reference images are and are not selected.
- the image search is performed using features with equivalent weights.
- other reference images both similar and dissimilar to the specified image to be searched are selected and respectively added to a reference image list or a dissimilar image list. Then, the weights of the respective features and/or feature elements are updated using the reference image list and the dissimilar image list.
- step S 501 and S 502 when a user selects a plurality or reference images (steps S 501 and S 502 ), the process (steps S 504 -S 50 B) is same as described with reference to steps S 302 -S 306 of FIG. 3 . Thus, an explanation will be omitted.
- step S 509 if the user is satisfied with the result of the image search (step S 509 ), the search operation terminates. However, if the user is not satisfied with the search result, the user selects other reference image(s) similar and dissimilar to the specified image to be searched among the resultant images of the search.
- the image search is performed using features with weights as assigned (step S 503 ).
- the search is performed using features with equal weights assigned. If the user is not satisfied with the search result, the user then selects other reference image(s) similar and dissimilar to the specified image to be searched among the resultant images of the search.
- the system adds the selected similar image(s) to an initial reference image list managed by the system (step S 511 ) and adds the dissimilar image(s) to an initial dissimilar image list (step S 312 ).
- the initial reference image list includes the reference image(s) selected in step S 501 .
- the system measures the similarities of the respective features and/or feature elements in the images included in the reference image list (step S 513 ), and measures the similarities of the respective features and/or feature elements in the images included in the dissimilar image list (step S 514 ).
- the system determines the weights of the respective features and/or feature elements using the images included in the reference image list and the dissimilar image list (step S 515 ).
- the system updates the weights of the respective features and/or feature elements, and re-performs the image search utilizing the updated weights (step S 516 ).
- the weights of the features by the similarly measurement of the features and/or feature elements in the images included in the reference/dissimilar image lists are calculated by Equations 3a ⁇ 3d.
- Weight_k ( a ⁇ Weight l_k ) ⁇ ( b Weight R_k ) [ 3 ⁇ a ]
- n denotes the reference number in the reference image list or the dissimilar image list
- m denotes the number of features used or the similarity measurement
- Weight_k denotes the final weight of the k-th feature
- Simi(i,j,k) denotes the similarity between the I-th reference image and the j-th reference image when the k-th feature is used
- Cont(k) denotes how much the k-th feature contribute to raise the similarity
- Weight I — k denotes the weight of the k-th feature in the reference image list
- Weight R — k denotes the weight of the k-th feature in the dissimilar image list.
- the similarities of the images included in the reference image list and the dissimilar image list are calculated respectively.
- the weights of the respective features Weight_k increase as the similarities of the images included in the reference image list rises, while the weights decrease as the similarities in the images included in the dissimilar image list rises.
- the values of a, b, p, and q denote constants
- m i denotes an average of the element i in the images included in the corresponding (reference and dissimilar) image lists
- v i denotes the distribution of the element i in the images included in the corresponding (reference and dissimilar) image lists.
- the similarities of the images included in the reference image list and the dissimilar image list are calculated respectively.
- the weights of the respective feature elements Weight_k increase as the similarities of the images included in the reference image list rises, while the weights decrease as the similarities in the images included in the dissimilar image list rises.
- A is s constant
- Diff(F k — l p,q) denotes the difference between the I-th elements of the k-th feature of the image p and image q
- w k — i denotes the weight of the 1-th feature element of the k-th feature
- w k denotes the weight of the k-th feature
- n denotes the number of features
- km denotes the number of feature elements of the k-th feature.
- the difference is obtained by multiplying the feature difference value of the respective image, the feature element weight of the respective feature, and the weight of the respective feature.
- the similarity is obtained by subtracting the difference from the constant.
- the system automatically determines and updates both the feature element weights of respective features and the weights of the features of the image to be searched when the user searches an image, Therefore, a rapid and effective search can be performed.
- Table 1 shows examples of queries by users
- Table 2 shows the feature information required according to the type of query when colors and textures are used as the basic features of an image.
- TABLE 1 Query Type 1 What color does image have as a whole? 2 Does any portion of the image have a certain color feature? 3 About what degree does the portion of the image have a certain color feature? 4 What texture does the image as a whole? 5 Does any portion of the image have a certain texture feature? 6 About what degree does the portion of the image have a certain texture feature? 7 Does the image have a certain color and texture feature or have any portion having such features?
- the search system should have at least the following 8 image feature information.
- the first image feature information is a global color information which represents the color feature of the whole image.
- a color histogram may be an example of the global color information.
- the second image feature information is a global texture information which represents the texture feature of the whole image.
- a texture histogram may be an example of the global texture information.
- the feature information of the color and the texture of the whole image may be represented by a combination of the global color information and the global texture information.
- the third image feature information is a local color information which represents the color feature of a region, i.e. grid region, in the image.
- a representative color for each local grid may be an example of the local color information.
- the weights of color elements obtained from the global color information may be utilized as the local color in formation.
- the fourth image feature element is a local texture information representing the texture feature of a grid region in the image.
- a representative texture for each grid may be an example of the local texture information.
- the weights of texture elements obtained from the global texture information may be utilized as the local texture information.
- the fifth image feature element is a local color and texture information which represents the color and texture features or a grid region in the image.
- a representative color and texture for each grid may be an example of the local color and texture information.
- the weights of color and texture elements respectively obtained from the global color information and the global texture information may be utilized as the local color and texture information.
- the sixth image feature element is a local color position information which represents a color feature in a region at a particular position of the image.
- a color local grid feature may be an example of the local color position information.
- the seventh image feature element is a local texture position information which represents a texture feature in a region at a particular position of the image.
- a texture local grid feature may be an example of the local texture position information.
- the specified color and texture feature in a region at the particular position of the image can be represented as a combination of the sixth and seventh information.
- the eighth image feature information is a local color and texture information which represents a specified color and texture feature in a region at a particular position of the image.
- a color and texture local grid feature may be an example of the local color and texture position information.
- the system can perform an effective search by constructing a set of feature information, i.e. image characteristics, as described above using the analyzed results based upon the contents of the queries and add element weights to the constructed features.
- feature information i.e. image characteristics
- the system adjusts the importance of the image characteristics, i.e. the weights of the features and/or feature element, and performs the image search.
- the search method using a reference multimedia data determines a multimedia data having the highest similarity to the reference multimedia data by adjusting the weights of the features and/or feature elements of the respective features included in the multimedia data.
- the weight adjustment of the feature and/or feature elements of the respective features can be performed using one of a direct adjusting method by the user, an automatic adjusting method by the system, or an adjusting method using the relevance information (i.e., positive and negative information) fed back to the system by the user.
- a color histogram represents the color distribution in an image.
- the texture histogram represents the texture distribution in an image.
- the color image grid represents the color information of a grid region generated by dividing an image into n*m grid regions.
- the texture image grid represents the texture information of a grid region generated by dividing an image into n*m grid regions.
- the color-texture joint local grid represents the color texture information of a grid region generated by dividing an image into n*m grid regions.
- FIG. 6 shows the structure of texture description which can be constructed in consideration of the query types and relevance feedbacks of the user 601 .
- the structure comprises comprises a global information 602 which represents a feature of a whole image, a spatial information 603 which represents a feature of an image region, and weight information 604 which represents the importance of the constructed features 602 and 603 .
- the global information includes a global feature descriptor 605 of the whole image, and an element weight descriptor 606 of the feature elements of the global feature descriptor of the whole image.
- the spatial information 603 includes a spatial feature descriptor 607 of an image region, and a position weight descriptor 608 of the image region.
- the global information 602 of the whole image and the spatial information 603 of the image region can be constructed by a selective combination of features included in the image such as the color, texture, and shape.
- the possible combinations of the basic features can be obtained using Equation 6 below, where n denotes the number of the basic features.
- ⁇ k 1 n ⁇ ⁇ nCk [ 6 ]
- Equation 6 the number of feature types obtained by Equation 6 applies to local positions and for global information, since there n number of basic features, the total number of feature types can be obtained by Equation 7.
- ⁇ k 1 n ⁇ ⁇ nCk + 2 ⁇ n . [ 7 ]
- the present invention will be explained utilizing two basic features of color and texture.
- FIG. 7 shows image characteristics constructed using the features of color and texture.
- the relevance feedback image(s) 701 used for adjusting the weights of the image is features according to the user feedback includes global color information 702 a of the whole image, global texture information 702 b of the whole image, spatial information 703 a of image regions, spatial color information 703 b of image regions, and weight descriptor 704 of the global informations 702 a and 702 b , and of the spatial informations 703 a and 703 b.
- the global color information 702 a includes a global color histogram 705 representing the color feature information of the whole image, and an element weight descriptor 706 of the respective bins of the global color histogram.
- the global texture information 702 b includes a global texture histogram 707 representing the texture information of the whole image, and an element weight descriptor 708 of the respective bins of the global texture histogram.
- the spatial color information 703 a includes a color image grid 709 , and a position weight descriptor 710 of the color image grid.
- the spatial texture information 703 b includes a texture image grid 711 , and a position weight descriptor 712 or the texture image grid.
- the color histogram 705 is used as a feature information of the whole image and the weight of each color element in the color histogram 705 are represented by the element weight descriptor 706 .
- the global texture histogram 707 is used as another feature information of the whole image and the weight of each texture element in the global texture histogram 707 are represented by the element weight descriptor 708 .
- the color image grid 709 is used as a feature information of the image regions and the weight or each grid position in the color image grid 709 is represented by the position weight descriptor 710 .
- the texture image grid 711 is used as another feature information of the image regions, and the weight of each grid position in the texture image grid 711 is represented by the position weight descriptor 712 .
- the feature weights are represented the same level as the feature information, and the feature element weights are represented in a level below the respective feature information.
- image characteristics may be constructed alternatively is with the feature weights in a level below the respective feature information as shown in FIG. 8 .
- a global color information 801 is the feature information
- the global color information 801 includes 2 global color feature 802 , and weights 803 .
- the weights 803 is composed of feature weights 804 corresponding to the global color feature and feature element weights 805 .
- FIG. 9 shows another embodiment of the image characteristics used for adjusting the weights of the image features according to the user feedback.
- this image characteristic all information related to weight characteristics are grouped into a set and represented separately.
- the image feature structure 902 i.e. the reference feedback, for adjusting the weights of the image features when searching the image 901 includes image characteristics 903 and the weight characteristics 904 .
- the image characteristics 903 include global information 905 , local information 906 , and local position information 907 .
- the weight characteristics 904 include feature weights 908 and feature element weights 909 .
- the global information 905 includes n feature units 910
- the local information 906 includes a number of feature units 911 equivalent to a sum of the number or features and possible combinations of the features
- the local position information 907 also includes n feature units.
- FIG. 10 shows another example of the image data structure of FIG. 9 when the image information includes two basic features or color and texture.
- the image characteristics 1001 includes global information 1002 , local information 1003 and local positional information 1004 .
- the global information 1002 includes a global color feature unit 1005 and a global texture feature unit 1007 .
- the local information 1003 includes a local color feature unit 1009 , a local texture feature unit 1010 and a local color and texture feature unit 1011 .
- the local position information 1004 includes a local position color feature unit 1013 and a local position texture feature unit 1014 .
- the global color feature unit 1005 is represented by a global color histogram 1006
- the global texture feature unit 1007 is represented by a global texture histogram 1008
- the local color feature unit 1009 and the local position color feature unit 1013 are represented by a color image grid 1012
- the local texture feature unit 1010 and the local position texture feature unit 1014 are represented by a texture image grid 1015
- the local color and texture feature unit 1011 is represented by both the color image grid 1012 and the texture image grid 1015 .
- the query types in Table 2 can be satisfied by constructing image characteristics of the seven features as described above, and the weights are updated by adjusting the weights in the feature weights and the feature element weights as shown in FIG. 11 .
- the image feature structure 1102 i.e. the reference feedback, for adjusting the weights of the image features when searching the image 1101 includes image characteristics 1103 and weight characteristics 1104 .
- the image characteristics 1103 includes global information 1105 , local information 1106 and local positional information 1107 .
- the weight characteristics 1104 includes feature weights 1108 and feature element weights 1109 .
- the global information 1105 includes a global color feature unit 1110 and a global texture feature unit 1111 .
- the local information 1106 includes a local color feature unit 1112 , a local texture feature unit 1113 and a local color and texture feature unit 1114 .
- the local position information 1109 includes a local position color feature unit 1115 and a local position texture feature unit 1116 .
- the system analyzes all possible queries of the user, and provides minimum image characteristics which satisfy all judgement standards during the image search. Accordingly, a rapid and effective image search can be performed by adjusting the weights of the features and feature elements to reflect the user feedbacks.
Abstract
A method of searching multimedia data is disclosed in which a search for an image can re-performed by automatically updating weights of features and/or weights of feature elements in the respective feature in an image.
Description
- 1. Field of the Invention
- The present invention relates to a method of searching multimedia data, and more particularly to a method of searching multimedia data more accurately by utilizing user feedback.
- 2. Background of the Related Art
- Recently, technology for digital image signal processing has been greatly developed and has been applied in various fields. For example, the digital image signal processing technology may be used in a search system for automatically editing only the face of a specific character in a moving picture file of a movie or drama, in a security system for permitting access to only persons registered in the system, or in a search system for searching a particular data from an image or video detected by a detection system. The performance of such systems basically depend on the accuracy and speed of detecting or searching the desired object. Accordingly, various image searching methods have been proposed in the related art.
- An image search system which detects a degree of similarity with an image to be searched utilizing features such as color, texture or shape is disclosed in U.S. Pat. No. 5,579,471 entitled “An image query system and method,” Depending upon the image to be searched, the importance or a feature may vary and within one particular feature such as the color, the importance of a feature element such as the red or green color may also vary. However, the above searching system does not take into consideration the different importance of features or feature elements for each image to be searched.
- In another searching method entitled “Virage image search engine” (www.virage.com), a user directly inputs the level of importance for features such as the color, texture and shape by assigning weight values. Although an image may be searched according to an importance or a feature using this method, it may be difficult for a user to determine the weights of features.
- Therefore, Yong Rui in “Relevance feedback techniques in interactive” SPIE Vol. 3312, discloses a method in which images similar to a reference image are found and the importance of features or weight for features are automatically obtained by calculating the similarities among the found images. However, the weight importance information is not maintained after a search for a specific image is finished and must be calculated for each image search, even for a same image.
- In the image search and browsing system or the video (moving image) search and browsing system of the related art, information which describes a particular feature of an image or video data is utilized to perform a more effective search or browsing of the multimedia data or example, in the image query system, an image may be divided into a plurality of regions and a representative color of each region may be utilized as a feature information of the image, or a whole color histogram of the image may be utilized as a feature information. Thereafter, two images are compared to calculate a similarity based upon the feature information and a determination is made whether the two images are similar.
- Therefore, the image search methods in the related art may utilize weights of features such as color, texture, or shape. However, weights of feature elements are not taken into consideration. Accordingly, the image search methods in related art has the limitations in intellectually training and developing weights for searching, resulting in relatively longer searching period to obtain search results and a deterioration of the reliability of the search results.
- Accordingly, an object of the present invention is to solve at least the problems and disadvantages of the related art.
- An object of the present invention is to provide a method of searching multimedia data by automatically updating weights of features included in a specified image and/or weights of feature elements, and by applying the updated weights to search for the specified object.
- Another object of the present invention is to provide a method or searching multimedia data which constructs image characteristics corresponding to the types of features included in a specified image by analyzing and classifying the judgement standards applied when the user searches the image, and adjusts the feature information set by taking into consideration weights of the features and weights of feature elements during a following image search.
- Still another object of the present invention is to provide a feature structure to be included in a multimedia data to effectively search an image.
- Additional advantages, objects, and features of the invention will be set forth in part in the description which follows and in part will become apparent to those having ordinary skill in the art upon examination of the following or may be learned from practice of L the invention. The objects and is advantages of the invention may be realized and attained as particularly pointed out in the appended claims.
- To achieve the objects and in accordance with the purposes of the invention, as embodied and broadly described herein, a method of searching multimedia data in a multimedia data search system comprises searching for a reference multimedia data selected by a user; receiving user feedback of relevance information for the searched multimedia data; determining importance of respective feature elements of features included in the multimedia data according to the relevance information; re-performing the search for the reference multimedia data by updating the importance of said respective elements if the user requests an additional search; and updating previous importance to new importance obtained and maintaining the updated importance degrees.
- In another embodiment of the present invention, a method of searching multimedia data in a multimedia data search system comprises receiving an inquiry into previously searched multimedia data; analyzing a judgement standard for the multimedia data searched during the inquiry; constructing image characteristics using at least one feature included in the multimedia data using an analysis result of the judgement standard; and adjusting importance of the image characteristics and re-performing a search of the multimedia data if a user requests an additional search.
- The present invention also provides a feature structure of multimedia data comprising a first information representing a feature of the multimedia data; a second information representing a regional feature of the multimedia data; and a third information representing importance of the first and second information.
- The invention will be described in detail with reference to the following drawings in which like reference numerals refer to like elements wherein:
-
FIG. 1 shows features in an image by a histogram; -
FIG. 2 shows an image represented by a local grids; -
FIG. 3 is a flowchart of a multimedia data search process according to a first embodiment of the present invention; -
FIGS. 4 and 5 are flowcharts of a multimedia data search process ofFIG. 2 , where an initial search does not result in a desired image; -
FIG. 6 is a basic structure of image characteristics for an image search according to a second embodiment of the present invention; -
FIG. 7 is image characteristics ofFIG. 6 wherein the feature information is constructed using a color and texture; and - FIGS. 8 to 11 are different embodiments of the image characteristics,
- Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings.
- In a general image search system, features of an image such as color, texture, and shape may be represented by a histogram as shown in
FIG. 1 . Particularly,FIG. 1 is a color histogram of an image by which the colors in the image is grouped into 24 color elements. By adjusting or altering the weights of each color element, the degree of importance of each color element or the extent that a color element affects a search may be represented. -
FIG. 2 is an image represented by local grids. Particularly, an image is divided into n*m grid regions and each grid region may be represented either by a regional color histogram or by a color representing the and region. Here, the degree of importance of each grid region or the extent that a grid region affects a search may be represented by assigning weights to each of n*m grid regions. Also, a predetermined threshold value determined by the system may be utilized as a cap such that certain grid regions has no influence in the search. Namely, if a grid region has a weight which exceeds the predetermined threshold value, the grid region is utilized in the search. Otherwise, the and region is processed as a “Don't care” region which does not affect the image search. - The image search process will next be explained with reference to FIGS. 3 to 5.
-
FIG. 3 is a flowchart of a multimedia data search process when a plurality of reference images are selected and weights of the respective types or elements of features included in a specified image are assigned or updated according to the first embodiment of the present invention. Referring toFIG. 3 , a user selects a plurality of reference images (step S301) representing the specified image to be searched. Thereafter, the system judges and determines the importance of the feature elements included in the reference images. - For example, the system measures the similarities of the features included in the reference images (step S302), and determines the weights of each feature according to the measured similarities of the features (step S303). Also, the system measures the similarities of the feature elements in each feature included in the reference images (step S304), and determines the weights of feature elements in each respective feature according to the measured similarities of the feature elements (step S305).
- Accordingly, if a user requests an additional search, the system re-performs the image search by using the importance, i.e. the weights of the features and the weights of the feature elements in the respective features (step S306). At this time, the system may use either the weights of the features or the weights of the feature elements, or both.
- Particularly, the weights of the features and feature elements of respective features are determined as follows. When the user selects a plurality of the reference images, the search system determines weights of the features and the weights of the feature elements in the respective features by measuring the similarities of features of the selected reference image list and the similarities or the feature elements in the respective features. The weights of the features are calculated by Equations 1a, 1b, and 1c below.
- In the above equations, n denotes the reference number, m denotes the number of features used for measuring the similarity, Weight_k denotes the weight of the k-th feature, Simi(i,j,k) denotes the similarity between the I-th reference image and the j-th reference image when the k-th feature is used, and Cont(k) denotes how much the k-th feature contributes to raise the similarity. Generally, the weights of the respective features Weight_k increases as the similarities rises since the similarities among the reference images are calculated based upon the respective features and the feature having the highest similarity acts as the most important factor.
- The weights of the feature elements are determined by Equations 2a and 2b using the similarities of the feature elements in the respective features or the reference images.
Weight of element i: wi=afI(i) [2a] -
- [2b] Similarlity of element i in reference images
- [2b] Similarlity of element i in reference images
- In the above equations, the values of a, p, and q denote constants, mi denotes an average of an element i in the reference image list, and v, denotes the distribution of an element i in the reference image list. According to Equations 2a and 2b, the weight wi of a feature element is inversely proportional to the distribution of the corresponding feature element and is proportional to an average value of the corresponding feature element component. Therefore, a feature element having a large average value acts as an important factor even if the distribution of the corresponding feature element is wide.
-
FIG. 4 is a flowchart showing a multimedia data search process for when a plurality of reference images are and are not selected. Generally, if the number of reference images is not plural, the image search is performed using features with equivalent weights. However, if the search does not result in a desired image, other reference images similar to the specified image to be searched are selected and added to a reference image list. Then, the weights of the respective features and/or feature elements are updated using the reference image list. - Referring to
FIG. 4 , when a user selects a plurality of reference images (steps S401 and S402) the process (steps S404-S408) is the same as described with reference to steps S302-S306 ofFIG. 3 . Thus, an explanation will be omitted. Thereafter, if the user is satisfied with the result of the image search (step S408), the search operation terminates. However, if the user is not satisfied with the search result, the user selects other reference image(s) similar to the specified image to be searched among the resultant images of the search (step S411). On the other hand, when a user selects only one reference image, the image search is performed using features with weights as assigned (step S403). Generally, if the reference image is selected for the first time, the search is performed using features with equal weights assigned. If the user is not satisfied with the search result, the user selects other reference image(s) similar to the specified image to be searched among the resultant images of the search (step S411). - Accordingly, the system adds the selected other reference image(s) to an initial reference image list managed by the system (step S412). Here, the initial reference image list includes the reference image(s) selected in step S401. Thereafter, the system measures the similarities of the respective features and/or feature elements in the selected reference images (step S413), and determines the weights of the respective features and/or feature elements in the selected reference image by Equations 1a 1c, 2a and 2b (step S414). Thus, the system updates the weights of the respective features and/or feature elements, and re-performs the image search utilizing the updated weights (step S415).
-
FIG. 5 is a flowchart showing another multimedia data search process for when a plurality of reference images are and are not selected. Generally, if the number of reference images is not plural, the image search is performed using features with equivalent weights. However, if the search does not result in a desired image, other reference images, both similar and dissimilar to the specified image to be searched are selected and respectively added to a reference image list or a dissimilar image list. Then, the weights of the respective features and/or feature elements are updated using the reference image list and the dissimilar image list. - Referring to
FIG. 5 , when a user selects a plurality or reference images (steps S501 and S502), the process (steps S504-S50B) is same as described with reference to steps S302-S306 ofFIG. 3 . Thus, an explanation will be omitted. Thereafter, if the user is satisfied with the result of the image search (step S509), the search operation terminates. However, if the user is not satisfied with the search result, the user selects other reference image(s) similar and dissimilar to the specified image to be searched among the resultant images of the search. On the other hand, when a user selects only one reference image, the image search is performed using features with weights as assigned (step S503). Here, if the reference image is selected for the first time, the search is performed using features with equal weights assigned. If the user is not satisfied with the search result, the user then selects other reference image(s) similar and dissimilar to the specified image to be searched among the resultant images of the search. - Accordingly, the system adds the selected similar image(s) to an initial reference image list managed by the system (step S511) and adds the dissimilar image(s) to an initial dissimilar image list (step S312). Here, the initial reference image list includes the reference image(s) selected in step S501. Thereafter, the system measures the similarities of the respective features and/or feature elements in the images included in the reference image list (step S513), and measures the similarities of the respective features and/or feature elements in the images included in the dissimilar image list (step S514).
- Using Equations 1a˜1c, 2a and 2b, the system determines the weights of the respective features and/or feature elements using the images included in the reference image list and the dissimilar image list (step S515). Thus, the system updates the weights of the respective features and/or feature elements, and re-performs the image search utilizing the updated weights (step S516). Particularly, the weights of the features by the similarly measurement of the features and/or feature elements in the images included in the reference/dissimilar image lists are calculated by Equations 3a˜3d.
- In the above equations, n denotes the reference number in the reference image list or the dissimilar image list, m denotes the number of features used or the similarity measurement, Weight_k denotes the final weight of the k-th feature, Simi(i,j,k) denotes the similarity between the I-th reference image and the j-th reference image when the k-th feature is used, Cont(k) denotes how much the k-th feature contribute to raise the similarity, WeightI
— k denotes the weight of the k-th feature in the reference image list, and WeightR— k denotes the weight of the k-th feature in the dissimilar image list. Generally, the similarities of the images included in the reference image list and the dissimilar image list are calculated respectively. As a result, the weights of the respective features Weight_k increase as the similarities of the images included in the reference image list rises, while the weights decrease as the similarities in the images included in the dissimilar image list rises. - Also, after measuring the similarities of the feature elements in the respective features of the images included in the reference/dissimilar image list, the weights of the feature elements of the respective features are determined by Equations 4a˜4b.
Weight of an element i: wi=afI(i)+bfR(i) [4a] -
- [4b] Similarity of an element i in the reference images:
- [4b] Similarity of an element i in the reference images:
- In the above equations, fR(I)=pmi×vi, denotes the dissimilarity of an element i in the images included in the dissimilar image list, the values of a, b, p, and q denote constants, mi denotes an average of the element i in the images included in the corresponding (reference and dissimilar) image lists, and vi denotes the distribution of the element i in the images included in the corresponding (reference and dissimilar) image lists. Generally, the similarities of the images included in the reference image list and the dissimilar image list are calculated respectively. As a result, the weights of the respective feature elements Weight_k increase as the similarities of the images included in the reference image list rises, while the weights decrease as the similarities in the images included in the dissimilar image list rises.
- If the weights of the features and the weights of the feature elements are determined as above, the similarities will be calculated using
Equation 5 during the image search. - Here, A is s constant, Diff(Fk
— lp,q) denotes the difference between the I-th elements of the k-th feature of the image p and image q, wk— i denotes the weight of the 1-th feature element of the k-th feature, wk denotes the weight of the k-th feature, n denotes the number of features, and km denotes the number of feature elements of the k-th feature. Thus, the difference is obtained by multiplying the feature difference value of the respective image, the feature element weight of the respective feature, and the weight of the respective feature. Also, the similarity is obtained by subtracting the difference from the constant. - As described above in reference to FIGS. 3˜5, the system automatically determines and updates both the feature element weights of respective features and the weights of the features of the image to be searched when the user searches an image, Therefore, a rapid and effective search can be performed.
- Nevertheless, if the user wishes to perform a further search of the specified image after viewing a previously searched result, the user may raise and enter various kinds of queries to the search system. Table 1 shows examples of queries by users, and Table 2 shows the feature information required according to the type of query when colors and textures are used as the basic features of an image.
TABLE 1 Query Type 1 What color does image have as a whole? 2 Does any portion of the image have a certain color feature? 3 About what degree does the portion of the image have a certain color feature? 4 What texture does the image as a whole? 5 Does any portion of the image have a certain texture feature? 6 About what degree does the portion of the image have a certain texture feature? 7 Does the image have a certain color and texture feature or have any portion having such features? 8 Does any portion of the image have a certain color and texture feature? 9 About what degree does the portion of the image have a certain color and texture feature? 10 What color does the image have at a specified position? 11 What texture does the image have at a specified position? 12 What color and texture does the image have at a specified position? -
TABLE 2 Query Type Main Feature Type 1 What color does image have as a Global color whole? information 2 Does any portion have a certain Local color color feature? information 3 What degree does the portion Local color have a certain color feature information 4 What texture does the image as a Global texture whole? formation 5 Does any portion have a certain Local texture texture feature? information 6 what degree does the portion Local texture have a certain texture feature? information 7 Does the image have a certain Global color texture color and texture feature or information have any portion having such features? 8 Does any portion have a certain Local color texture color and texture feature? information 9 What degree does the portion local color texture have a certain color and texture information feature? 10 What color does the image have Local color position at a specified position? information 11 What texture does the image have local texture at a specified position? position information 12 What color and texture does the local color texture image have at a specified position information position? - In Table 2, 12 query types are presented and to satisfy the characteristics with respect to such queries, the search system should have at least the following 8 image feature information.
- The first image feature information is a global color information which represents the color feature of the whole image. A color histogram may be an example of the global color information. The second image feature information is a global texture information which represents the texture feature of the whole image. A texture histogram may be an example of the global texture information. The feature information of the color and the texture of the whole image may be represented by a combination of the global color information and the global texture information.
- The third image feature information is a local color information which represents the color feature of a region, i.e. grid region, in the image. A representative color for each local grid may be an example of the local color information. Alternatively, the weights of color elements obtained from the global color information may be utilized as the local color in formation.
- The fourth image feature element is a local texture information representing the texture feature of a grid region in the image. A representative texture for each grid may be an example of the local texture information. Alternatively, the weights of texture elements obtained from the global texture information may be utilized as the local texture information.
- The fifth image feature element is a local color and texture information which represents the color and texture features or a grid region in the image. A representative color and texture for each grid may be an example of the local color and texture information. Alternatively, the weights of color and texture elements respectively obtained from the global color information and the global texture information may be utilized as the local color and texture information.
- The sixth image feature element is a local color position information which represents a color feature in a region at a particular position of the image. A color local grid feature may be an example of the local color position information. The seventh image feature element is a local texture position information which represents a texture feature in a region at a particular position of the image. A texture local grid feature may be an example of the local texture position information. Also, the specified color and texture feature in a region at the particular position of the image can be represented as a combination of the sixth and seventh information.
- The eighth image feature information is a local color and texture information which represents a specified color and texture feature in a region at a particular position of the image. A color and texture local grid feature may be an example of the local color and texture position information.
- The system can perform an effective search by constructing a set of feature information, i.e. image characteristics, as described above using the analyzed results based upon the contents of the queries and add element weights to the constructed features. Thus, if a user requests a search, the system adjusts the importance of the image characteristics, i.e. the weights of the features and/or feature element, and performs the image search.
- The search method using a reference multimedia data determines a multimedia data having the highest similarity to the reference multimedia data by adjusting the weights of the features and/or feature elements of the respective features included in the multimedia data. Here, the weight adjustment of the feature and/or feature elements of the respective features can be performed using one of a direct adjusting method by the user, an automatic adjusting method by the system, or an adjusting method using the relevance information (i.e., positive and negative information) fed back to the system by the user. The meanings of the features as described above will now be explained in detail.
- First, a color histogram represents the color distribution in an image. Similarly, the texture histogram represents the texture distribution in an image.
- The color image grid represents the color information of a grid region generated by dividing an image into n*m grid regions.
- The texture image grid represents the texture information of a grid region generated by dividing an image into n*m grid regions. The color-texture joint local grid represents the color texture information of a grid region generated by dividing an image into n*m grid regions.
-
FIG. 6 shows the structure of texture description which can be constructed in consideration of the query types and relevance feedbacks of theuser 601. The structure comprises comprises aglobal information 602 which represents a feature of a whole image, aspatial information 603 which represents a feature of an image region, andweight information 604 which represents the importance of the constructed features 602 and 603. The global information includes aglobal feature descriptor 605 of the whole image, and anelement weight descriptor 606 of the feature elements of the global feature descriptor of the whole image. Thespatial information 603 includes aspatial feature descriptor 607 of an image region, and aposition weight descriptor 608 of the image region. - The
global information 602 of the whole image and thespatial information 603 of the image region can be constructed by a selective combination of features included in the image such as the color, texture, and shape. Here, the possible combinations of the basic features can be obtained usingEquation 6 below, where n denotes the number of the basic features. - Thus, the number of feature types obtained by
Equation 6 applies to local positions and for global information, since there n number of basic features, the total number of feature types can be obtained byEquation 7. - The present invention will be explained utilizing two basic features of color and texture. In such case, the total number of feature types required by the system would be 3+2*2=7. However, if the feature of shape is added, the total number of required feature type would be 7:2*3=13.
-
FIG. 7 shows image characteristics constructed using the features of color and texture. Referring toFIG. 7 , the relevance feedback image(s) 701 used for adjusting the weights of the image is features according to the user feedback includesglobal color information 702 a of the whole image,global texture information 702 b of the whole image,spatial information 703 a of image regions,spatial color information 703 b of image regions, andweight descriptor 704 of theglobal informations spatial informations - In
FIG. 7 , fourfeature informations global color information 702 a includes aglobal color histogram 705 representing the color feature information of the whole image, and anelement weight descriptor 706 of the respective bins of the global color histogram. Theglobal texture information 702 b includes aglobal texture histogram 707 representing the texture information of the whole image, and anelement weight descriptor 708 of the respective bins of the global texture histogram. - Also, the
spatial color information 703 a includes acolor image grid 709, and aposition weight descriptor 710 of the color image grid. Thespatial texture information 703 b includes atexture image grid 711, and aposition weight descriptor 712 or the texture image grid. - The
color histogram 705 is used as a feature information of the whole image and the weight of each color element in thecolor histogram 705 are represented by theelement weight descriptor 706. Also, theglobal texture histogram 707 is used as another feature information of the whole image and the weight of each texture element in theglobal texture histogram 707 are represented by theelement weight descriptor 708. - Moreover, the
color image grid 709 is used as a feature information of the image regions and the weight or each grid position in thecolor image grid 709 is represented by theposition weight descriptor 710. Similarly, thetexture image grid 711 is used as another feature information of the image regions, and the weight of each grid position in thetexture image grid 711 is represented by theposition weight descriptor 712. - As shown in
FIG. 7 , an image characteristic structure having four feature information was explained in order to satisfy the twelve query types in Table 2. However, all nine feature types for the twelve query types is not necessary. For example, if a color-texture joint local grid is used as a feature, the local color, local texture, local color and texture, local position color, local position texture and local position color and texture can be obtained from the color-texture joint local grid. - Furthermore, in the image characteristic structure of
FIG. 7 , the feature weights are represented the same level as the feature information, and the feature element weights are represented in a level below the respective feature information. However, image characteristics may be constructed alternatively is with the feature weights in a level below the respective feature information as shown inFIG. 8 . For example, assuming that aglobal color information 801 is the feature information, theglobal color information 801 includes 2global color feature 802, andweights 803. Here theweights 803 is composed offeature weights 804 corresponding to the global color feature and featureelement weights 805. -
FIG. 9 shows another embodiment of the image characteristics used for adjusting the weights of the image features according to the user feedback. In this image characteristic, all information related to weight characteristics are grouped into a set and represented separately. - Referring to
FIG. 9 , theimage feature structure 902, i.e. the reference feedback, for adjusting the weights of the image features when searching theimage 901 includesimage characteristics 903 and theweight characteristics 904. Theimage characteristics 903 includeglobal information 905,local information 906, andlocal position information 907. Theweight characteristics 904 includefeature weights 908 andfeature element weights 909. Moreover, theglobal information 905 includesn feature units 910, thelocal information 906 includes a number offeature units 911 equivalent to a sum of the number or features and possible combinations of the features, and thelocal position information 907 also includes n feature units. -
FIG. 10 shows another example of the image data structure ofFIG. 9 when the image information includes two basic features or color and texture. Particularly, theimage characteristics 1001 includesglobal information 1002,local information 1003 and localpositional information 1004. Theglobal information 1002 includes a globalcolor feature unit 1005 and a globaltexture feature unit 1007. Thelocal information 1003 includes a localcolor feature unit 1009, a localtexture feature unit 1010 and a local color andtexture feature unit 1011. Thelocal position information 1004 includes a local positioncolor feature unit 1013 and a local positiontexture feature unit 1014. - Moreover, the global
color feature unit 1005 is represented by aglobal color histogram 1006, the globaltexture feature unit 1007 is represented by aglobal texture histogram 1008, and the localcolor feature unit 1009 and the local positioncolor feature unit 1013 are represented by acolor image grid 1012. Also, the localtexture feature unit 1010 and the local positiontexture feature unit 1014 are represented by atexture image grid 1015. Finally, the local color andtexture feature unit 1011 is represented by both thecolor image grid 1012 and thetexture image grid 1015. - Therefore, the query types in Table 2 can be satisfied by constructing image characteristics of the seven features as described above, and the weights are updated by adjusting the weights in the feature weights and the feature element weights as shown in
FIG. 11 . Referring toFIG. 11 , theimage feature structure 1102, i.e. the reference feedback, for adjusting the weights of the image features when searching theimage 1101 includesimage characteristics 1103 andweight characteristics 1104. - Particularly, the
image characteristics 1103 includesglobal information 1105,local information 1106 and localpositional information 1107. Theweight characteristics 1104 includesfeature weights 1108 andfeature element weights 1109. Here, theglobal information 1105 includes a globalcolor feature unit 1110 and a globaltexture feature unit 1111. Thelocal information 1106 includes a localcolor feature unit 1112, a localtexture feature unit 1113 and a local color andtexture feature unit 1114. Thelocal position information 1109 includes a local positioncolor feature unit 1115 and a local positiontexture feature unit 1116. - As described above, according to the present invention, the system analyzes all possible queries of the user, and provides minimum image characteristics which satisfy all judgement standards during the image search. Accordingly, a rapid and effective image search can be performed by adjusting the weights of the features and feature elements to reflect the user feedbacks.
- The foregoing embodiments are merely exemplary and are not to be construed as limiting the present invention. The present teachings can be readily applied to other types of apparatuses. The description of the present invention is intended to be illustrative, and not to limit the scope of the claims. Many alternatives, modifications, and variations will be apparent to those skilled in the art.
Claims (9)
1-20. (canceled)
21. A method of constructing a multimedia data comprising:
incorporating a feature information including feature and feature elements of an image; and
incorporating a weight information including weight information of said features and weight information of said feature elements.
22. A method of claim 21 , wherein the feature and the feature elements are represented by an image characteristic structure comprising:
a global information which represents a feature of a whole image; and
a spatial information which represents a feature of an image region, wherein the image characteristic structure further comprises a weight information which represents the importance of the global information and the spatial information.
23. A method of searching a desired image from reference images using visual feature information comprising:
generating visual feature information with respect to an interested region in each reference image, the visual feature information utilized in an image search system, including:
providing a global feature information;
incorporating a local visual feature information, wherein the local visual feature information includes a set of visual feature information assigned to each sub region in each image:
incorporating a weight information of the global visual feature information; and
incorporating an importance information assigned to each sub region of the interested region in each image;
measuring the similarities of reference images based on the global and the local visual feature information of reference images; and
selecting an image based on the measured similarities.
24. The method of claim 23 , wherein the global and local visual feature information are plurally provided.
25. The method of claim 23 , wherein the visual feature includes color, texture, and shape.
26. The method of claim 25 , wherein the global and the local visual feature information include color histogram, region-representing color, and texture histogram.
27. The method of claim 23 , wherein the importance information represents whether the visual feature information assigned to corresponding sub region of the interested region of the interested region in the image is utilized in image search,
28. The method of claim 27 , wherein search performance is determined based on weight information and the importance information.
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US11/179,511 US20050262067A1 (en) | 1999-02-01 | 2005-07-13 | Method of searching multimedia data |
US12/821,502 US20100318522A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
US12/821,524 US20100318523A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
Applications Claiming Priority (6)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
KR3183/1999 | 1999-02-01 | ||
KR1019990003182A KR100319154B1 (en) | 1999-02-01 | 1999-02-01 | Method of image composition and method of image search system |
KR1019990003183A KR100319150B1 (en) | 1999-02-01 | 1999-02-01 | Method for image checking by weight automatic decision for each type and element feature element |
KR3182/1999 | 1999-02-01 | ||
US09/495,250 US7016916B1 (en) | 1999-02-01 | 2000-01-31 | Method of searching multimedia data |
US11/179,511 US20050262067A1 (en) | 1999-02-01 | 2005-07-13 | Method of searching multimedia data |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/495,250 Continuation US7016916B1 (en) | 1999-02-01 | 2000-01-31 | Method of searching multimedia data |
Related Child Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/821,502 Continuation US20100318522A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
US12/821,524 Continuation US20100318523A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
Publications (1)
Publication Number | Publication Date |
---|---|
US20050262067A1 true US20050262067A1 (en) | 2005-11-24 |
Family
ID=26634660
Family Applications (4)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/495,250 Expired - Fee Related US7016916B1 (en) | 1999-02-01 | 2000-01-31 | Method of searching multimedia data |
US11/179,511 Abandoned US20050262067A1 (en) | 1999-02-01 | 2005-07-13 | Method of searching multimedia data |
US12/821,502 Abandoned US20100318522A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
US12/821,524 Abandoned US20100318523A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US09/495,250 Expired - Fee Related US7016916B1 (en) | 1999-02-01 | 2000-01-31 | Method of searching multimedia data |
Family Applications After (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US12/821,502 Abandoned US20100318522A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
US12/821,524 Abandoned US20100318523A1 (en) | 1999-02-01 | 2010-06-23 | Method of searching multimedia data |
Country Status (6)
Country | Link |
---|---|
US (4) | US7016916B1 (en) |
EP (2) | EP1635270A3 (en) |
JP (1) | JP3564068B2 (en) |
CN (2) | CN1201267C (en) |
AU (1) | AU2463900A (en) |
WO (1) | WO2000045342A1 (en) |
Cited By (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20060101011A1 (en) * | 2004-11-05 | 2006-05-11 | International Business Machines Corporation | Method, system and program for executing a query having a union operator |
US20080310725A1 (en) * | 2007-06-15 | 2008-12-18 | Masatomo Kurata | Image processing apparatus and method and program |
US20100017389A1 (en) * | 2006-05-29 | 2010-01-21 | University Of Wollongong | Content based image retrieval |
US20110202543A1 (en) * | 2010-02-16 | 2011-08-18 | Imprezzeo Pty Limited | Optimising content based image retrieval |
US20120177294A1 (en) * | 2011-01-10 | 2012-07-12 | Microsoft Corporation | Image retrieval using discriminative visual features |
US20140379528A1 (en) * | 2013-06-25 | 2014-12-25 | Overstock.Com, Inc. | System and method for graphically building weighted search queries |
DE102014214851A1 (en) * | 2014-07-29 | 2016-02-04 | picalike GmbH | Computer-implemented method and computer system for carrying out a similarity analysis |
US20160189396A1 (en) * | 2013-09-13 | 2016-06-30 | Cortexica Vision Systems Limited | Image processing |
US9747622B1 (en) | 2009-03-24 | 2017-08-29 | Overstock.Com, Inc. | Point-and-shoot product lister |
US9760795B2 (en) | 2014-02-24 | 2017-09-12 | Electronics And Telecommunications Research Institute | Method and apparatus for extracting image feature |
US9805425B2 (en) | 2004-06-02 | 2017-10-31 | Overstock.Com, Inc. | System and methods for electronic commerce using personal and business networks |
US9928752B2 (en) | 2011-03-24 | 2018-03-27 | Overstock.Com, Inc. | Social choice engine |
US10210179B2 (en) * | 2008-11-18 | 2019-02-19 | Excalibur Ip, Llc | Dynamic feature weighting |
US10269081B1 (en) | 2007-12-21 | 2019-04-23 | Overstock.Com, Inc. | System, program product, and methods for social network advertising and incentives for same |
US10546262B2 (en) | 2012-10-19 | 2020-01-28 | Overstock.Com, Inc. | Supply chain management system |
US10810654B1 (en) | 2013-05-06 | 2020-10-20 | Overstock.Com, Inc. | System and method of mapping product attributes between different schemas |
US10872350B1 (en) | 2013-12-06 | 2020-12-22 | Overstock.Com, Inc. | System and method for optimizing online marketing based upon relative advertisement placement |
US10929890B2 (en) | 2013-08-15 | 2021-02-23 | Overstock.Com, Inc. | System and method of personalizing online marketing campaigns |
US10970769B2 (en) | 2017-03-02 | 2021-04-06 | Overstock.Com, Inc. | Method and system for optimizing website searching with user pathing |
US10970463B2 (en) | 2016-05-11 | 2021-04-06 | Overstock.Com, Inc. | System and method for optimizing electronic document layouts |
US11023947B1 (en) | 2013-03-15 | 2021-06-01 | Overstock.Com, Inc. | Generating product recommendations using a blend of collaborative and content-based data |
US11205179B1 (en) | 2019-04-26 | 2021-12-21 | Overstock.Com, Inc. | System, method, and program product for recognizing and rejecting fraudulent purchase attempts in e-commerce |
WO2022050687A1 (en) * | 2020-09-02 | 2022-03-10 | 최광석 | Terminal for providing patent search and control method therefor, server for providing patent search and control method therefor, and patent search system and control method therefor |
US20220147573A1 (en) * | 2020-11-11 | 2022-05-12 | Hitachi, Ltd. | Search condition presentation apparatus, search condition presentation method, and recording medium |
US11463578B1 (en) | 2003-12-15 | 2022-10-04 | Overstock.Com, Inc. | Method, system and program product for communicating e-commerce content over-the-air to mobile devices |
US11514493B1 (en) | 2019-03-25 | 2022-11-29 | Overstock.Com, Inc. | System and method for conversational commerce online |
US11676192B1 (en) | 2013-03-15 | 2023-06-13 | Overstock.Com, Inc. | Localized sort of ranked product recommendations based on predicted user intent |
US11734368B1 (en) | 2019-09-26 | 2023-08-22 | Overstock.Com, Inc. | System and method for creating a consistent personalized web experience across multiple platforms and channels |
Families Citing this family (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7099860B1 (en) | 2000-10-30 | 2006-08-29 | Microsoft Corporation | Image retrieval systems and methods with semantic and feature based relevance feedback |
US6970860B1 (en) | 2000-10-30 | 2005-11-29 | Microsoft Corporation | Semi-automatic annotation of multimedia objects |
KR100422710B1 (en) * | 2000-11-25 | 2004-03-12 | 엘지전자 주식회사 | Multimedia query and retrieval system using multi-weighted feature |
JP4889159B2 (en) | 2001-05-14 | 2012-03-07 | 富士通株式会社 | Data search system and data search method |
GB2389927A (en) * | 2002-06-20 | 2003-12-24 | Peter Foot | Method for searching a data source by building an abstract composite image |
US7536365B2 (en) * | 2005-12-08 | 2009-05-19 | Northrop Grumman Corporation | Hybrid architecture for acquisition, recognition, and fusion |
TWI403912B (en) * | 2006-06-08 | 2013-08-01 | Univ Nat Chiao Tung | Method and system of image retrieval |
US20080155426A1 (en) * | 2006-12-21 | 2008-06-26 | Microsoft Corporation | Visualization and navigation of search results |
CN101510217B (en) * | 2009-03-09 | 2013-06-05 | 阿里巴巴集团控股有限公司 | Image updating method in image database, server and system |
US9015139B2 (en) | 2010-05-14 | 2015-04-21 | Rovi Guides, Inc. | Systems and methods for performing a search based on a media content snapshot image |
JP7018001B2 (en) | 2018-09-20 | 2022-02-09 | 株式会社日立製作所 | Information processing systems, methods and programs for controlling information processing systems |
CN112597321B (en) * | 2021-03-05 | 2022-02-22 | 腾讯科技(深圳)有限公司 | Multimedia processing method based on block chain and related equipment |
Citations (22)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5020019A (en) * | 1989-05-29 | 1991-05-28 | Ricoh Company, Ltd. | Document retrieval system |
US5297042A (en) * | 1989-10-05 | 1994-03-22 | Ricoh Company, Ltd. | Keyword associative document retrieval system |
US5321833A (en) * | 1990-08-29 | 1994-06-14 | Gte Laboratories Incorporated | Adaptive ranking system for information retrieval |
US5579471A (en) * | 1992-11-09 | 1996-11-26 | International Business Machines Corporation | Image query system and method |
US5619347A (en) * | 1994-09-28 | 1997-04-08 | Matsushita Electric Industrial Co., Ltd. | Apparatus for calculating a degree of white balance adjustment for a picture |
US5724567A (en) * | 1994-04-25 | 1998-03-03 | Apple Computer, Inc. | System for directing relevance-ranked data objects to computer users |
US5793888A (en) * | 1994-11-14 | 1998-08-11 | Massachusetts Institute Of Technology | Machine learning apparatus and method for image searching |
US5802361A (en) * | 1994-09-30 | 1998-09-01 | Apple Computer, Inc. | Method and system for searching graphic images and videos |
US5855015A (en) * | 1995-03-20 | 1998-12-29 | Interval Research Corporation | System and method for retrieval of hyperlinked information resources |
US5893095A (en) * | 1996-03-29 | 1999-04-06 | Virage, Inc. | Similarity engine for content-based retrieval of images |
US5982931A (en) * | 1995-06-07 | 1999-11-09 | Ishimaru; Mikio | Apparatus and method for the manipulation of image containing documents |
US6041140A (en) * | 1994-10-04 | 2000-03-21 | Synthonics, Incorporated | Apparatus for interactive image correlation for three dimensional image production |
US6067539A (en) * | 1998-03-02 | 2000-05-23 | Vigil, Inc. | Intelligent information retrieval system |
US6081276A (en) * | 1996-11-14 | 2000-06-27 | International Business Machines Corporation | Method and apparatus for creating a color name dictionary and for querying an image by color name |
US6128398A (en) * | 1995-01-31 | 2000-10-03 | Miros Inc. | System, method and application for the recognition, verification and similarity ranking of facial or other object patterns |
US6163622A (en) * | 1997-12-18 | 2000-12-19 | U.S. Philips Corporation | Image retrieval system |
US6175829B1 (en) * | 1998-04-22 | 2001-01-16 | Nec Usa, Inc. | Method and apparatus for facilitating query reformulation |
US6285995B1 (en) * | 1998-06-22 | 2001-09-04 | U.S. Philips Corporation | Image retrieval system using a query image |
US6353823B1 (en) * | 1999-03-08 | 2002-03-05 | Intel Corporation | Method and system for using associative metadata |
US6446060B1 (en) * | 1999-01-26 | 2002-09-03 | International Business Machines Corporation | System and method for sequential processing for content-based retrieval of composite objects |
US6445834B1 (en) * | 1998-10-19 | 2002-09-03 | Sony Corporation | Modular image query system |
US6519360B1 (en) * | 1997-09-17 | 2003-02-11 | Minolta Co., Ltd. | Image processing apparatus for comparing images based on color feature information and computer program product in a memory |
Family Cites Families (31)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH04320665A (en) | 1991-04-19 | 1992-11-11 | Uofuji:Kk | Processing of shark's fin using vegetable-mixed high-viscosity material |
JPH0628059A (en) | 1991-09-30 | 1994-02-04 | Toshiba Corp | Initializing device for hardware simulator |
JPH05145289A (en) | 1991-11-20 | 1993-06-11 | Tokico Ltd | Component mounting apparatus |
JPH05309135A (en) | 1992-05-11 | 1993-11-22 | Teijin Ltd | Gas supply system for breath |
JP3143532B2 (en) * | 1992-11-30 | 2001-03-07 | キヤノン株式会社 | Image retrieval apparatus and method |
JPH0721198A (en) * | 1993-06-17 | 1995-01-24 | Nippon Telegr & Teleph Corp <Ntt> | Image retrieving method |
JP3229453B2 (en) | 1993-08-12 | 2001-11-19 | 財団法人ダム水源地環境整備センター | Ultrasonic Fish Counting System |
JP3229451B2 (en) | 1993-08-12 | 2001-11-19 | 財団法人ダム水源地環境整備センター | Ultrasonic fish counting device |
JP3026712B2 (en) * | 1993-12-09 | 2000-03-27 | キヤノン株式会社 | Image search method and apparatus |
JPH07239856A (en) * | 1994-02-25 | 1995-09-12 | Canon Inc | Method and device for retrieving image |
JPH07260148A (en) | 1994-03-22 | 1995-10-13 | Senshin Zairyo Riyou Gas Jienereeta Kenkyusho:Kk | Burner for gas turbine |
JPH08249353A (en) * | 1995-03-15 | 1996-09-27 | Omron Corp | Method and device for image retrieval |
JP3284528B2 (en) * | 1995-03-15 | 2002-05-20 | オムロン株式会社 | Image search method and apparatus |
JPH08262965A (en) * | 1995-03-20 | 1996-10-11 | Mitsubishi Electric Corp | Closed caption decoder with pause function for language learning |
JPH09101970A (en) * | 1995-10-06 | 1997-04-15 | Omron Corp | Method and device for retrieving image |
US5696964A (en) * | 1996-04-16 | 1997-12-09 | Nec Research Institute, Inc. | Multimedia database retrieval system which maintains a posterior probability distribution that each item in the database is a target of a search |
JPH1066833A (en) | 1996-08-26 | 1998-03-10 | Riken Corp | Removal of nitrogen oxide |
US5873080A (en) * | 1996-09-20 | 1999-02-16 | International Business Machines Corporation | Using multiple search engines to search multimedia data |
JPH1095840A (en) | 1996-09-25 | 1998-04-14 | Teijin Ltd | Production of unsaturated polyester |
US5898001A (en) | 1996-10-29 | 1999-04-27 | Council Of Scientific And Industrial Research | Tissue culture process for producing a large number of viable mint plants in vitro from internodal segments |
JP3609225B2 (en) * | 1996-11-25 | 2005-01-12 | 日本電信電話株式会社 | Similar object retrieval device |
JP3446797B2 (en) * | 1996-12-11 | 2003-09-16 | 日本電信電話株式会社 | Similar object search method and apparatus |
JP3754791B2 (en) * | 1997-03-19 | 2006-03-15 | キヤノン株式会社 | Image search apparatus and method |
JPH10289241A (en) * | 1997-04-14 | 1998-10-27 | Canon Inc | Image processor and its control method |
JP3349066B2 (en) | 1997-05-20 | 2002-11-20 | 株式会社クボタ | Seedling transplanter |
JPH10326286A (en) * | 1997-05-27 | 1998-12-08 | Mitsubishi Electric Corp | Similarity retrieval device and recording medium where similarity retrival program is recorded |
JPH10330813A (en) | 1997-06-04 | 1998-12-15 | Nippon Steel Corp | Smelting reduction and decarburizing equipment and operating method thereof |
JP3673615B2 (en) * | 1997-06-19 | 2005-07-20 | キヤノン株式会社 | Image processing apparatus and control method thereof |
WO1999022318A1 (en) * | 1997-10-27 | 1999-05-06 | Massachusetts Institute Of Technology | Image search and retrieval system |
JP2970755B2 (en) | 1997-12-01 | 1999-11-02 | 日本電気株式会社 | Semiconductor device |
US7272593B1 (en) * | 1999-01-26 | 2007-09-18 | International Business Machines Corporation | Method and apparatus for similarity retrieval from iterative refinement |
-
2000
- 2000-01-31 US US09/495,250 patent/US7016916B1/en not_active Expired - Fee Related
- 2000-02-01 CN CNB00800126XA patent/CN1201267C/en not_active Expired - Fee Related
- 2000-02-01 CN CN2005100547495A patent/CN1661601B/en not_active Expired - Fee Related
- 2000-02-01 AU AU24639/00A patent/AU2463900A/en not_active Abandoned
- 2000-02-01 WO PCT/KR2000/000079 patent/WO2000045342A1/en active Application Filing
- 2000-02-01 EP EP05077645A patent/EP1635270A3/en not_active Withdrawn
- 2000-02-01 JP JP2000596530A patent/JP3564068B2/en not_active Expired - Fee Related
- 2000-02-01 EP EP00902998A patent/EP1066596A1/en not_active Withdrawn
-
2005
- 2005-07-13 US US11/179,511 patent/US20050262067A1/en not_active Abandoned
-
2010
- 2010-06-23 US US12/821,502 patent/US20100318522A1/en not_active Abandoned
- 2010-06-23 US US12/821,524 patent/US20100318523A1/en not_active Abandoned
Patent Citations (23)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5020019A (en) * | 1989-05-29 | 1991-05-28 | Ricoh Company, Ltd. | Document retrieval system |
US5297042A (en) * | 1989-10-05 | 1994-03-22 | Ricoh Company, Ltd. | Keyword associative document retrieval system |
US5321833A (en) * | 1990-08-29 | 1994-06-14 | Gte Laboratories Incorporated | Adaptive ranking system for information retrieval |
US5579471A (en) * | 1992-11-09 | 1996-11-26 | International Business Machines Corporation | Image query system and method |
US5751286A (en) * | 1992-11-09 | 1998-05-12 | International Business Machines Corporation | Image query system and method |
US5724567A (en) * | 1994-04-25 | 1998-03-03 | Apple Computer, Inc. | System for directing relevance-ranked data objects to computer users |
US5619347A (en) * | 1994-09-28 | 1997-04-08 | Matsushita Electric Industrial Co., Ltd. | Apparatus for calculating a degree of white balance adjustment for a picture |
US5802361A (en) * | 1994-09-30 | 1998-09-01 | Apple Computer, Inc. | Method and system for searching graphic images and videos |
US6041140A (en) * | 1994-10-04 | 2000-03-21 | Synthonics, Incorporated | Apparatus for interactive image correlation for three dimensional image production |
US5793888A (en) * | 1994-11-14 | 1998-08-11 | Massachusetts Institute Of Technology | Machine learning apparatus and method for image searching |
US6128398A (en) * | 1995-01-31 | 2000-10-03 | Miros Inc. | System, method and application for the recognition, verification and similarity ranking of facial or other object patterns |
US5855015A (en) * | 1995-03-20 | 1998-12-29 | Interval Research Corporation | System and method for retrieval of hyperlinked information resources |
US5982931A (en) * | 1995-06-07 | 1999-11-09 | Ishimaru; Mikio | Apparatus and method for the manipulation of image containing documents |
US5893095A (en) * | 1996-03-29 | 1999-04-06 | Virage, Inc. | Similarity engine for content-based retrieval of images |
US6081276A (en) * | 1996-11-14 | 2000-06-27 | International Business Machines Corporation | Method and apparatus for creating a color name dictionary and for querying an image by color name |
US6519360B1 (en) * | 1997-09-17 | 2003-02-11 | Minolta Co., Ltd. | Image processing apparatus for comparing images based on color feature information and computer program product in a memory |
US6163622A (en) * | 1997-12-18 | 2000-12-19 | U.S. Philips Corporation | Image retrieval system |
US6067539A (en) * | 1998-03-02 | 2000-05-23 | Vigil, Inc. | Intelligent information retrieval system |
US6175829B1 (en) * | 1998-04-22 | 2001-01-16 | Nec Usa, Inc. | Method and apparatus for facilitating query reformulation |
US6285995B1 (en) * | 1998-06-22 | 2001-09-04 | U.S. Philips Corporation | Image retrieval system using a query image |
US6445834B1 (en) * | 1998-10-19 | 2002-09-03 | Sony Corporation | Modular image query system |
US6446060B1 (en) * | 1999-01-26 | 2002-09-03 | International Business Machines Corporation | System and method for sequential processing for content-based retrieval of composite objects |
US6353823B1 (en) * | 1999-03-08 | 2002-03-05 | Intel Corporation | Method and system for using associative metadata |
Cited By (43)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11463578B1 (en) | 2003-12-15 | 2022-10-04 | Overstock.Com, Inc. | Method, system and program product for communicating e-commerce content over-the-air to mobile devices |
US10853891B2 (en) | 2004-06-02 | 2020-12-01 | Overstock.Com, Inc. | System and methods for electronic commerce using personal and business networks |
US9805425B2 (en) | 2004-06-02 | 2017-10-31 | Overstock.Com, Inc. | System and methods for electronic commerce using personal and business networks |
US7539667B2 (en) * | 2004-11-05 | 2009-05-26 | International Business Machines Corporation | Method, system and program for executing a query having a union operator |
US20060101011A1 (en) * | 2004-11-05 | 2006-05-11 | International Business Machines Corporation | Method, system and program for executing a query having a union operator |
US20100017389A1 (en) * | 2006-05-29 | 2010-01-21 | University Of Wollongong | Content based image retrieval |
US8346014B2 (en) * | 2007-06-15 | 2013-01-01 | Sony Corporation | Image processing apparatus and method and program |
US20080310725A1 (en) * | 2007-06-15 | 2008-12-18 | Masatomo Kurata | Image processing apparatus and method and program |
US10269081B1 (en) | 2007-12-21 | 2019-04-23 | Overstock.Com, Inc. | System, program product, and methods for social network advertising and incentives for same |
US10210179B2 (en) * | 2008-11-18 | 2019-02-19 | Excalibur Ip, Llc | Dynamic feature weighting |
US10074118B1 (en) | 2009-03-24 | 2018-09-11 | Overstock.Com, Inc. | Point-and-shoot product lister |
US10896451B1 (en) | 2009-03-24 | 2021-01-19 | Overstock.Com, Inc. | Point-and-shoot product lister |
US9747622B1 (en) | 2009-03-24 | 2017-08-29 | Overstock.Com, Inc. | Point-and-shoot product lister |
US20110202543A1 (en) * | 2010-02-16 | 2011-08-18 | Imprezzeo Pty Limited | Optimising content based image retrieval |
US9229956B2 (en) * | 2011-01-10 | 2016-01-05 | Microsoft Technology Licensing, Llc | Image retrieval using discriminative visual features |
US20120177294A1 (en) * | 2011-01-10 | 2012-07-12 | Microsoft Corporation | Image retrieval using discriminative visual features |
US9928752B2 (en) | 2011-03-24 | 2018-03-27 | Overstock.Com, Inc. | Social choice engine |
US10546262B2 (en) | 2012-10-19 | 2020-01-28 | Overstock.Com, Inc. | Supply chain management system |
US11023947B1 (en) | 2013-03-15 | 2021-06-01 | Overstock.Com, Inc. | Generating product recommendations using a blend of collaborative and content-based data |
US11676192B1 (en) | 2013-03-15 | 2023-06-13 | Overstock.Com, Inc. | Localized sort of ranked product recommendations based on predicted user intent |
US11631124B1 (en) | 2013-05-06 | 2023-04-18 | Overstock.Com, Inc. | System and method of mapping product attributes between different schemas |
US10810654B1 (en) | 2013-05-06 | 2020-10-20 | Overstock.Com, Inc. | System and method of mapping product attributes between different schemas |
US10102287B2 (en) | 2013-06-25 | 2018-10-16 | Overstock.Com, Inc. | System and method for graphically building weighted search queries |
US10769219B1 (en) * | 2013-06-25 | 2020-09-08 | Overstock.Com, Inc. | System and method for graphically building weighted search queries |
US9483788B2 (en) * | 2013-06-25 | 2016-11-01 | Overstock.Com, Inc. | System and method for graphically building weighted search queries |
US20140379528A1 (en) * | 2013-06-25 | 2014-12-25 | Overstock.Com, Inc. | System and method for graphically building weighted search queries |
US11475484B1 (en) | 2013-08-15 | 2022-10-18 | Overstock.Com, Inc. | System and method of personalizing online marketing campaigns |
US10929890B2 (en) | 2013-08-15 | 2021-02-23 | Overstock.Com, Inc. | System and method of personalizing online marketing campaigns |
US20160189396A1 (en) * | 2013-09-13 | 2016-06-30 | Cortexica Vision Systems Limited | Image processing |
US10134149B2 (en) * | 2013-09-13 | 2018-11-20 | Cortexica Vision Systems Limited | Image processing |
US10872350B1 (en) | 2013-12-06 | 2020-12-22 | Overstock.Com, Inc. | System and method for optimizing online marketing based upon relative advertisement placement |
US11694228B1 (en) | 2013-12-06 | 2023-07-04 | Overstock.Com, Inc. | System and method for optimizing online marketing based upon relative advertisement placement |
US9760795B2 (en) | 2014-02-24 | 2017-09-12 | Electronics And Telecommunications Research Institute | Method and apparatus for extracting image feature |
DE102014214851A1 (en) * | 2014-07-29 | 2016-02-04 | picalike GmbH | Computer-implemented method and computer system for carrying out a similarity analysis |
US11526653B1 (en) | 2016-05-11 | 2022-12-13 | Overstock.Com, Inc. | System and method for optimizing electronic document layouts |
US10970463B2 (en) | 2016-05-11 | 2021-04-06 | Overstock.Com, Inc. | System and method for optimizing electronic document layouts |
US10970769B2 (en) | 2017-03-02 | 2021-04-06 | Overstock.Com, Inc. | Method and system for optimizing website searching with user pathing |
US11514493B1 (en) | 2019-03-25 | 2022-11-29 | Overstock.Com, Inc. | System and method for conversational commerce online |
US11205179B1 (en) | 2019-04-26 | 2021-12-21 | Overstock.Com, Inc. | System, method, and program product for recognizing and rejecting fraudulent purchase attempts in e-commerce |
US11928685B1 (en) | 2019-04-26 | 2024-03-12 | Overstock.Com, Inc. | System, method, and program product for recognizing and rejecting fraudulent purchase attempts in e-commerce |
US11734368B1 (en) | 2019-09-26 | 2023-08-22 | Overstock.Com, Inc. | System and method for creating a consistent personalized web experience across multiple platforms and channels |
WO2022050687A1 (en) * | 2020-09-02 | 2022-03-10 | 최광석 | Terminal for providing patent search and control method therefor, server for providing patent search and control method therefor, and patent search system and control method therefor |
US20220147573A1 (en) * | 2020-11-11 | 2022-05-12 | Hitachi, Ltd. | Search condition presentation apparatus, search condition presentation method, and recording medium |
Also Published As
Publication number | Publication date |
---|---|
JP2002536731A (en) | 2002-10-29 |
CN1661601B (en) | 2010-06-02 |
EP1635270A3 (en) | 2010-03-03 |
EP1066596A1 (en) | 2001-01-10 |
WO2000045342A8 (en) | 2001-04-19 |
CN1661601A (en) | 2005-08-31 |
US20100318522A1 (en) | 2010-12-16 |
WO2000045342A1 (en) | 2000-08-03 |
CN1201267C (en) | 2005-05-11 |
EP1635270A2 (en) | 2006-03-15 |
US20100318523A1 (en) | 2010-12-16 |
AU2463900A (en) | 2000-08-18 |
JP3564068B2 (en) | 2004-09-08 |
CN1294721A (en) | 2001-05-09 |
US7016916B1 (en) | 2006-03-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US7016916B1 (en) | Method of searching multimedia data | |
US6795818B1 (en) | Method of searching multimedia data | |
US6704725B1 (en) | Method of searching multimedia data | |
EP1066595B1 (en) | Method of searching or browsing multimedia data and data structure | |
US6430312B1 (en) | Image subregion querying using color correlograms | |
US6240423B1 (en) | Method and system for image querying using region based and boundary based image matching | |
US6826316B2 (en) | System and method for determining image similarity | |
US6014664A (en) | Method and apparatus for incorporating weights into data combinational rules | |
CN100388279C (en) | Method and device for measuring visual similarity | |
US6556710B2 (en) | Image searching techniques | |
US6522782B2 (en) | Image and text searching techniques | |
KR101548438B1 (en) | Method and apparatus for comparing videos | |
US20070043774A1 (en) | Method and Apparatus for Incremental Computation of the Accuracy of a Categorization-by-Example System | |
JPH07160731A (en) | Method and device for picture retrieval | |
US6522780B1 (en) | Indexing of images and/or text | |
JP2002288657A (en) | Representative color setting method utilizing spatial dense component | |
US6522779B2 (en) | Representing an image with a posterized joint histogram | |
KR100319150B1 (en) | Method for image checking by weight automatic decision for each type and element feature element | |
JP4209104B2 (en) | Multimedia search method using multiple weight structure | |
Zhang et al. | NetView: integrating large scale distributed visual databases | |
Amato et al. | A query language for similarity-based retrieval of multimedia data | |
Chan | Empirical comparison of image retrieval color similarity methods with human judgment | |
Amato et al. | Advances in Databases and Information Systems 1997 | |
Loots et al. | Relevance Feedback with Continuous Learning in Content Based Image Retrieval |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
STCB | Information on status: application discontinuation |
Free format text: ABANDONED -- FAILURE TO RESPOND TO AN OFFICE ACTION |