US8131716B2 - Tuning of relevancy ranking for federated search - Google Patents

Tuning of relevancy ranking for federated search Download PDF

Info

Publication number
US8131716B2
US8131716B2 US12/834,710 US83471010A US8131716B2 US 8131716 B2 US8131716 B2 US 8131716B2 US 83471010 A US83471010 A US 83471010A US 8131716 B2 US8131716 B2 US 8131716B2
Authority
US
United States
Prior art keywords
score
relevancy ranking
ranking score
search
search result
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
US12/834,710
Other versions
US20100274783A1 (en
Inventor
Pierre-Yves Chevalier
Bruno Roustant
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
EMC Corp
Original Assignee
EMC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by EMC Corp filed Critical EMC Corp
Priority to US12/834,710 priority Critical patent/US8131716B2/en
Assigned to EMC CORPORATION reassignment EMC CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: CHEVALIER, PIERRE-YVES, ROUSTANT, BRUNO
Publication of US20100274783A1 publication Critical patent/US20100274783A1/en
Application granted granted Critical
Publication of US8131716B2 publication Critical patent/US8131716B2/en
Assigned to CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT reassignment CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT SECURITY AGREEMENT Assignors: ASAP SOFTWARE EXPRESS, INC., AVENTAIL LLC, CREDANT TECHNOLOGIES, INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL SOFTWARE INC., DELL SYSTEMS CORPORATION, DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., MAGINATICS LLC, MOZY, INC., SCALEIO LLC, SPANNING CLOUD APPS LLC, WYSE TECHNOLOGY L.L.C.
Assigned to THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT reassignment THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT SECURITY AGREEMENT Assignors: ASAP SOFTWARE EXPRESS, INC., AVENTAIL LLC, CREDANT TECHNOLOGIES, INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL SOFTWARE INC., DELL SYSTEMS CORPORATION, DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., MAGINATICS LLC, MOZY, INC., SCALEIO LLC, SPANNING CLOUD APPS LLC, WYSE TECHNOLOGY L.L.C.
Assigned to EMC IP Holding Company LLC reassignment EMC IP Holding Company LLC ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: EMC CORPORATION
Assigned to THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. reassignment THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. SECURITY AGREEMENT Assignors: CREDANT TECHNOLOGIES, INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., WYSE TECHNOLOGY L.L.C.
Assigned to THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. reassignment THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A. SECURITY AGREEMENT Assignors: CREDANT TECHNOLOGIES INC., DELL INTERNATIONAL L.L.C., DELL MARKETING L.P., DELL PRODUCTS L.P., DELL USA L.P., EMC CORPORATION, EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., WYSE TECHNOLOGY L.L.C.
Assigned to DELL INTERNATIONAL, L.L.C., SCALEIO LLC, DELL SYSTEMS CORPORATION, DELL MARKETING L.P., CREDANT TECHNOLOGIES, INC., MOZY, INC., EMC CORPORATION, AVENTAIL LLC, DELL USA L.P., MAGINATICS LLC, ASAP SOFTWARE EXPRESS, INC., WYSE TECHNOLOGY L.L.C., EMC IP Holding Company LLC, FORCE10 NETWORKS, INC., DELL PRODUCTS L.P., DELL SOFTWARE INC. reassignment DELL INTERNATIONAL, L.L.C. RELEASE BY SECURED PARTY (SEE DOCUMENT FOR DETAILS). Assignors: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
Assigned to DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), DELL INTERNATIONAL L.L.C., DELL USA L.P., SCALEIO LLC, DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), DELL PRODUCTS L.P., DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.) reassignment DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.) RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Assignors: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Assigned to DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), DELL PRODUCTS L.P., EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), SCALEIO LLC, DELL INTERNATIONAL L.L.C., DELL USA L.P., EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC) reassignment DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.) RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Assignors: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2457Query processing with adaptation to user needs
    • G06F16/24578Query processing with adaptation to user needs using ranking
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/20Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
    • G06F16/24Querying
    • G06F16/245Query processing
    • G06F16/2458Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
    • G06F16/2471Distributed queries

Definitions

  • Searching multiple sources returns a variety of search results. These search results need to be sorted or ranked when presenting them to the user. Ideally, the results are sorted or ranked in terms of what the searcher most desires to see as a result of the search the searcher initiated.
  • One common method of sorting and ranking web search results involves pre-indexing the web sites to indicate factors such as links, frequency of terms of interest, etc. Search result rankings are determined based on comparing the pre-indexed results between the search results. However, pre-indexing all of the content is a processing and storage intensive effort. In addition, comparing pre-indexed results provides no flexibility to account for a searcher's preferences.
  • a computer programmer searcher may have a preference for lay medical information sources whereas a medical researcher searcher may have a preference for medical library or journal repository sources. It would be better if search results could be sorted or ranked without the use of pre-indexed results and, instead, could be sorted or ranked to include preferences based on the searcher.
  • FIG. 1 is a block diagram illustrating an embodiment of a system for relevancy scoring.
  • FIG. 2 is a flow diagram illustrating an embodiment of a process for relevancy scoring.
  • FIG. 3 is a flow diagram illustrating an embodiment of a process for determining relevancy score.
  • FIG. 4 is a flow diagram illustrating an embodiment of a process for tuning a relevancy score.
  • FIG. 5 is a flow diagram illustrating an embodiment of a process for tuning of elements of the relevancy score.
  • the invention can be implemented in numerous ways, including as a process, an apparatus, a system, a composition of matter, a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or communication links.
  • these implementations, or any other form that the invention may take, may be referred to as techniques.
  • a component such as a processor or a memory described as being configured to perform a task includes both a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task.
  • the order of the steps of disclosed processes may be altered within the scope of the invention.
  • Tuning relevancy scoring is disclosed.
  • a query is received that includes a constraint.
  • a ranking score for a search result includes a consideration of whether a constraint of the search was met or not met by the search result.
  • a constraint expresses allowed desired values for a match in the search result.
  • a constraint includes a term or a term and an operator.
  • Metadata or attribute information is also used in some embodiments in determining a ranking score.
  • the ranking score is designed to mimic a user's assessment of relevancy of a search result and includes in various embodiments one or more of a number of perceived relevance features. Examples of relevancy features include scope/depth, accuracy/validity, currency, quality of source, and clarity.
  • the various feature scores are combined to create a relevancy score.
  • the feature scores and/or the combining of feature scores can be tuned for or by a user, a system, a super user, an administrator, an individual search, or for any other appropriate searcher or search.
  • scores are tuned using constants or functions that are added to or multiplied with elements in the scoring functions.
  • a user, a super user, an administrator, a search criteria, or any other appropriate configuration or criteria setting method sets a tunable element indirectly whereby the user, super user, administrator, search criteria tune or set an intuitively understandable criteria for search (e.g., computer code search, contract search, math article search, etc.) which in turn adjust multiple weights or added factors of features/elements in the relevancy score.
  • a contract search adds a greater weight to a document-type attribute of contract, a format-type attribute of PDF, and a department-type attribute of legal; the contract search can also tune clarity to lower the score of documents with computer code, mathematical symbols, or other non-English text type content.
  • a relevancy scoring of a result that is returned from searching heterogeneous sources is computed based on a set of independent features. These features simulate evaluation criteria of an end user.
  • the features include use of query structure and data structure.
  • the features are combined into a single score using a weight based formula. This enables the features to be tuned as well depending on a user, super user, administrator, or search criteria for searching.
  • the tuning includes adding or multiplying a constant or a function to one or more of the factors included in the relevancy score calculation.
  • the tuning is achieved by a user, super user, administrator or search criteria that indirectly sets weights or adds or multiplies constants or factors in a relevancy score calculation.
  • a code searching tuning, a medical article tuning, a contract tuning, etc. may have different constants or functions adding or multiplying the different features that are added/multiplied together for calculating a relevancy score.
  • the rank is received as a metadata associated with a search result from a search source.
  • the feature includes a constraint which includes an operator such as CONTAINS, DOES_NOT_CONTAIN, BEGINS_WITH, ENDS_WITH, ALWAYS_WITHIN, AND, OR, NOT, EQUALS, GREATER_THAN, LESS_THAN, GREATER_THAN_OR_EQUAL_TO, LESS_THAN_OR_EQUAL_TO, or any other appropriate operator.
  • an operator such as CONTAINS, DOES_NOT_CONTAIN, BEGINS_WITH, ENDS_WITH, ALWAYS_WITHIN, AND, OR, NOT, EQUALS, GREATER_THAN, LESS_THAN, GREATER_THAN_OR_EQUAL_TO, LESS_THAN_OR_EQUAL_TO, or any other appropriate operator.
  • a feature includes scope/depth, accuracy/validity, clarity, currency, quality of source, or any other appropriate feature useful in calculating a relevancy score.
  • a score for each result can be computed on the fly. That is, a whole set of results does not have to be received to be able to compute the score by comparing them to each other.
  • Each result gets its own score computable based on its own content and/or metadata. After this, there is a sorting by score (and maybe resorting as additional results are received, e.g., from additional sources). In some embodiments, there is a merging based on the score results or rank results from two scoring or ranking systems.
  • computing a score for each result on the fly as that result is received is disclosed, based on the result itself and not a relative comparison of results, and applying to metadata the criteria applied previously only to the structured content itself.
  • the metadata or attributes of the search result are synthesized (e.g., a title, a date, an abstract, a body, etc.) from the search result.
  • FIG. 1 is a block diagram illustrating an embodiment of a system for relevancy scoring.
  • one or more users connect to web-based content management application 102 using a computer system, represented in FIG. 1 by computer 100 .
  • web-based content management application 102 is built on a content management system vendor-provided web development kit (WDK) 104 .
  • WDK 104 comprises a set of tools to facilitate development of web-based applications, e.g., by providing reusable components pre-configured to interact with a content server and/or other repository to provide content management services.
  • Web-based content management application 102 in some embodiments provides to users content type aware content management services and/or functions.
  • An example of a content type aware service or function is providing a view (display) based at least in part on content item type, e.g., displaying a “calendar” as a grid and an email message with “to”, “from”, “subject”, and/or other fields in a header display region at the top of the display, attachment names/icons in an attachment display region below the header, and message text in a message body display region at the bottom; enabling a user to search by content type; and displaying query results in a manner determined based at least in part on content type.
  • content types comprise one or more of the following: contracts, news, code, financial, spread sheet, portable document format (PDF), HTML, XML, graphics, multimedia, or any other appropriate content type.
  • Web-based content management application 102 also includes search services 106 .
  • a user requests a search to a content management system.
  • the content management service searches both the internal repositories of the content management system and external repositories and other attached resources.
  • the content management system receives the search request and the WDK 104 translates the search request for search services 106 .
  • Search services 106 translates the search request for a direct search of internal content management system repository 108 and an indirect search of external repositories/resources via enterprise content integration server 110 .
  • search services 106 are provided by business logic and/or foundation classes that comprise a content management framework on which web-based content management application 102 and WDK 104 are based.
  • Search services 106 computes a score that is used to merge and rank results from different sources including external repositories/resources and internal sources/repositories for display to the user.
  • the score is computed based on a probability of interest of the user performing a search.
  • the score is based on the match of a result to the query terms and the precision of the query.
  • Enterprise content integration (ECI) server 110 targets the search request for a search of one or more external repositories, represented in FIG. 1 by 114 and 118 , and/or other resources, represented in FIG. 1 by 122 .
  • external repositories comprise other content management systems, database systems, legacy content management systems not capable of the type of search desired, (e.g., FilenetTM, LexisNexisTM, Lotus NotesTM, etc.).
  • other resources include information sources or search services—for example, GoogleTM, YahooTM, MSNTM, MySimonTM, PricewatchTM, AskJeevesTM, etc.
  • the targeted search request is translated and adapted using a configuration adapter, represented in FIG.
  • configuration adapter 112 translates and adapts the responses to the search request and passes the responses to ECI server 110 .
  • the configuration adapter includes associated information in the response to the search request.
  • the associated information is ultimately used as metadata information that enables the content management system to treat the search results as typed information for the results of the search.
  • type information is synthesized for search results based on information in search result or information about sources of the search results.
  • a query broker (not shown) at the content management system is configured to communicate with the ECI server 110 and to use information about the external content sources accessed via ECI server 110 to associate type information with results received from ECI server 110 .
  • the results from ECI server 110 in some embodiments indicate the external content source for each responsive content item (or group of items), and the query broker at the content management system associates type information with each content item (or set of content items) based on the source, as identified for example by data associated with a configuration file and/or adapter used to retrieve and/or identify the content from the external source.
  • more than one configuration adapter is associated with a specific repository or resource—for example, one configuration adapter is configured to facilitate searching for contract information in a specific database and another configuration adapter is configured to facilitate searching for financial information or specification information in the same specific database.
  • a configuration adapter is easily configured by a user, system administrator, system manager using an application with a graphical user interface (e.g., a WYSIWYG editor) to establish general parsing and syntax rules for communication between the ECI server and the external repository or resource.
  • a graphical user interface e.g., a WYSIWYG editor
  • the configuration adapter configuration is based on preconfigured templates, has a self learning engine in order to self-configure based at least in part on the external repository type and/or external resource type.
  • FIG. 2 is a flow diagram illustrating an embodiment of a process for relevancy scoring.
  • a query is received including constraints.
  • constraints in a query include constraints using the one or more of the following operators: DOES NOT CONTAIN, CONTAINS, BEGINS WITH, ENDS WITH, AND, OR, EQUALS, GREATER THAN, LESS THAN, DOES NOT EQUAL, or any other appropriate operator.
  • search results(s) are received including attribute(s).
  • An attribute comprises a metadata attached to a search result; in a typical case, the search result comprises a document.
  • an attribute comprises a title or document name, a creation or modification date, a summary or abstract for the document, a document type, or any other appropriate attribute.
  • attributes are called properties and include a title, a category, keywords, etc.
  • a relevancy score is calculated. The relevancy score evaluates relevancy using a small set of criteria that are similar to what user uses to evaluate relevancy. The criteria include measures of depth/scope, accuracy/validity, clarity, currency, and quality of sources. The criteria are then combined using weights.
  • a relevancy ranking is calculated based on the relevancy score. Scores are evaluated with respect to each other and other scores, and in some cases merged, to determine a ranking. In some embodiments, scores are weighted or normalized while being evaluated and/or merged for the ranking.
  • FIG. 3 is a flow diagram illustrating an embodiment of a process for determining relevancy score.
  • the process of FIG. 3 is used to implement 204 of FIG. 2 .
  • a scope/depth score is calculated.
  • the scope/depth score uses constraints with the operators: CONTAINS, DOES_NOT_CONTAIN, ALWAYS_WITHIN, BEGINS_WITH, and ENDS_WITH. Operators like EQUALS, GREATER_THAN, LESS_THAN, AND, NOT, and OR are measured using the accuracy/validity score.
  • the scope/depth score includes a measure of the number of CONTAINS-like constraints that are satisfied.
  • Attributes with multiple constraints are not penalized in the scope/depth score, and attributes with only one constraint are not weighted too heavily.
  • One goal for the scope/depth score is that a satisfied constraint must always increase the scope/depth score; for example, if in a first case we have only one constraint on the title attribute, and in a second case we have the same constraint plus a second satisfied constraint on title, then the second case must always be assigned a greater score than the first case.
  • Another goal for the scope/depth score is that an unsatisfied constraint must always decrease the score.
  • a bonus score is given to a primary attribute (e.g., a name or title of a document) that matches its constraint. If the primary attribute does not match its specific constraint, then try to match its constraint with the result full-text content.
  • a primary attribute e.g., a name or title of a document
  • a scope/depth score uses the following:
  • the scope/depth score is tuned for a particular user, super user, administrator, type of search, or any other appropriate tuning. For example, a search for patents may set the primary attribute as the abstract whereas a search for doctoral theses may set the primary attribute as the title. In some embodiments, bonus weight or attribute weights are tuned.
  • an accuracy/validity score is calculated.
  • the scores for accuracy and validity are merged because it is not a goal for the accuracy/validity score to count a constraint that improves the query precision, where the constraint is unsatisfied.
  • some constraint operators are considered more precise than others; For example, EQUALS is more precise than GREATER_THAN (or LESS_THAN_OR_EQUAL_TO, etc.), which is more precise than CONTAINS.
  • an AND (or NOT) operator improves the query precision with multiple constraints, but an OR operator does not improve much the query precision with multiple constraints.
  • each satisfied constraint must be taken into account for the accuracy/validity score, not only the overall constraint with logical operators.
  • each time a query term is matched it is checked to see if it is a full-word, same-stem-word, or a sub-word match.
  • the scope/depth score and/or accuracy/validity score can favour full-word matches that are obviously more relevant to the user.
  • Other matches do not indicate as precisely whether the document being scored is relevant for the user. For example, a full-word match counts as a 1.0 match. A same-stem-word match counts as 0.7 match. A sub-word match counts as 0.3 match.
  • the validity/accuracy score is tuned for a particular user, super user, administrator, type of search, or any other appropriate tuning.
  • a search for products may set the constraint weight with different values as compared to a search for human resource record.
  • a clarity score is calculated.
  • the clarity score is designed to capture the extent to which information is presented in a clear and well-organized manner.
  • special attributes e.g., title, abstract, or date
  • the attributes are checked for readability. For example, the attributes are checked to see if they are not a file name, not a uniform record locator (URL), or that the proportion of special characters (e.g., ‘/’, ‘.’, ‘ ⁇ ’, ‘ ⁇ ’, ‘>’, ‘?’, ‘&’, etc.) is not high, or any other appropriate measure for clarity.
  • clarity function includes a tuning for specialized searches—for example, emphasis given to the presence or absence of mathematical, scientific, medical, computer code, legal terms, or any other appropriate specialized clarity terms.
  • a search for code, for math or science or medical records may have a higher value for content that includes a high degree of math symbols, computer code, scientific, medical, or any other appropriate terminology.
  • an automated analysis is used to determine clarity.
  • a currency score is calculated.
  • the currency score analyzes the date attribute and gives a better score to recent results than to old results.
  • recent and old can have different time frames dependent on the content; for example, stock news is considered recent with a different time scale as compared to housing prices.
  • currency score may be a peaked function (e.g., pyramid, Gaussian, or any other peaked function) with the peak corresponding to a date of interest and a width corresponding to an appropriate time scale width.
  • a search may be performed regarding a weather event where documents are given highest priority on the day of the weather event and lower priorities just prior and after, but dwindling down to no priority for documents more than a week before or a week after.
  • the peaked function can be asymmetric where there is no interest in content from before the date (e.g., only interested in the after-effects or analysis regarding an event or date) or after the date (e.g., only interested in content predictions or speculations prior to an event).
  • a tuning of the currency function for a specialized search allows the adjustment of the time weighting function including the peak, the width, and the shape of the function.
  • a quality of source score is calculated.
  • the quality of source score provides a means to enhance sources based on an assessment of quality of the source.
  • the quality or source score may emphasize a type of repository based on the type of search.
  • code searches may emphasize code repositories or web resources with code stored; news searches may emphasize news sources or repositories, financial searches may emphasize financial sites or repositories; medical searches may emphasize medical repositories or web resources.
  • the tuning of the quality of source score includes the specialized tuning for searches of legal, financial, medical, scientific, or any other appropriate source tunings.
  • a final relevancy score is calculated.
  • tuning of the relevancy score for a specific application, source, search, user, a super user, an administrator or any other appropriate tuning Tuning may affect one or multiple parameters in the score or ranking.
  • the tuning may include adding or multiplying one or more constants or one or more functions to the score or ranking algorithm to have a desired effect. For example, a user may perform many searches for medical articles so that emphasis is given to current articles (e.g., within the past three years) from journals (e.g., from journal sites, medical article sources/repositories, reputable web sites, etc.) with good reputations that include a high degree of medical terminology (e.g., medical specialty terms in the title, abstract, and body).
  • a person may search for events near and around a family member's birthday so that emphasis is given to content from that date from sources showing current events (e.g., news repositories, blogs, video sites, etc.) with common usage clarity (e.g., no computer code).
  • sources showing current events e.g., news repositories, blogs, video sites, etc.
  • common usage clarity e.g., no computer code
  • FIG. 4 is a flow diagram illustrating an embodiment of a process for tuning a relevancy score.
  • an indication is received that a relevancy score is to be tuned.
  • a user, a super user, an administrator, or any other searcher indicates a tuning by adjusting a weight/constant/function shape for a set of user-friendly criteria.
  • the indication is using a graphical user interface, a configuration file, a script, or any other appropriate manner of indicating a tuning.
  • a weight or constant is adjusted using a button, radio dial, slider switch, a numerical value, or any other appropriate manner of adjusting a weight or constant.
  • a function shape is selected by indicating a button near a graphic of a function graph, a math formula, or a self drawn function shape, or any other appropriate manner of selecting a function shape.
  • tuning of elements of the relevancy score is determined. Each part of the relevancy score includes a number of elements that are added or multiplied or have a functional form. Each element is adjusted appropriately based on the input easily adjusted user selection. For example, an indication of a computer code search would favor sources that are code repositories and search results with clarity score that was adjusted to include computer code vocabulary as well as symbols associated with computer coding. The computer code search would also favor more recent results to avoid compatibility issues with older operating systems/plug-ins/compilers/hardware environments etc.
  • FIG. 5 is a flow diagram illustrating an embodiment of a process for tuning of elements of the relevancy score.
  • the process of FIG. 5 is used to implement the 402 of FIG. 4 .
  • scope/depth score elements are tuned, if appropriate.
  • accuracy/validity score elements are tuned, if appropriate.
  • clarity score elements are tuned, if appropriate.
  • currency score elements are tuned, if appropriate.
  • quality of source score elements are tuned, if appropriate.
  • final relevancy score elements are tuned, if appropriate.
  • W 1 , W 2 , W 3 , W 4 , W 5 may be tuned by adding or multiplying a factor or function.
  • the function fcurr(date) can take on a different form being any specified shape with value sloping up or down, saturating, or peaking before, after, at, or near the date input as an input parameter.

Abstract

Determining a relevancy ranking score is disclosed. An indication is received that a relevancy ranking score algorithm is to be tuned to a selected preference. The relevancy ranking score algorithm is updated based at least in part on the selected preference, wherein the relevancy ranking score of a search result resulting from a search query is based at least in part on one or more constraints of the search query.

Description

CROSS REFERENCE TO OTHER APPLICATIONS
This application is a continuation of co-pending U.S. patent application Ser. No. 11/824,304, entitled TUNING OF RELEVANCY RANKING FOR FEDERATED SEARCH filed Jun. 29, 2007 which is incorporated herein by reference for all purposes.
BACKGROUND OF THE INVENTION
Searching multiple sources (e.g., web sites, repositories, databases, etc.) returns a variety of search results. These search results need to be sorted or ranked when presenting them to the user. Ideally, the results are sorted or ranked in terms of what the searcher most desires to see as a result of the search the searcher initiated. One common method of sorting and ranking web search results involves pre-indexing the web sites to indicate factors such as links, frequency of terms of interest, etc. Search result rankings are determined based on comparing the pre-indexed results between the search results. However, pre-indexing all of the content is a processing and storage intensive effort. In addition, comparing pre-indexed results provides no flexibility to account for a searcher's preferences. For example, when performing a search for medical information, a computer programmer searcher may have a preference for lay medical information sources whereas a medical researcher searcher may have a preference for medical library or journal repository sources. It would be better if search results could be sorted or ranked without the use of pre-indexed results and, instead, could be sorted or ranked to include preferences based on the searcher.
BRIEF DESCRIPTION OF THE DRAWINGS
Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
FIG. 1 is a block diagram illustrating an embodiment of a system for relevancy scoring.
FIG. 2 is a flow diagram illustrating an embodiment of a process for relevancy scoring.
FIG. 3 is a flow diagram illustrating an embodiment of a process for determining relevancy score.
FIG. 4 is a flow diagram illustrating an embodiment of a process for tuning a relevancy score.
FIG. 5 is a flow diagram illustrating an embodiment of a process for tuning of elements of the relevancy score.
DETAILED DESCRIPTION
The invention can be implemented in numerous ways, including as a process, an apparatus, a system, a composition of matter, a computer readable medium such as a computer readable storage medium or a computer network wherein program instructions are sent over optical or communication links. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. A component such as a processor or a memory described as being configured to perform a task includes both a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. In general, the order of the steps of disclosed processes may be altered within the scope of the invention.
A detailed description of one or more embodiments of the invention is provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
Tuning relevancy scoring is disclosed. A query is received that includes a constraint. A ranking score for a search result includes a consideration of whether a constraint of the search was met or not met by the search result. A constraint expresses allowed desired values for a match in the search result. In various embodiments, a constraint includes a term or a term and an operator. Metadata or attribute information is also used in some embodiments in determining a ranking score. The ranking score is designed to mimic a user's assessment of relevancy of a search result and includes in various embodiments one or more of a number of perceived relevance features. Examples of relevancy features include scope/depth, accuracy/validity, currency, quality of source, and clarity. The various feature scores are combined to create a relevancy score. The feature scores and/or the combining of feature scores can be tuned for or by a user, a system, a super user, an administrator, an individual search, or for any other appropriate searcher or search. In various embodiments, scores are tuned using constants or functions that are added to or multiplied with elements in the scoring functions. In some embodiments, a user, a super user, an administrator, a search criteria, or any other appropriate configuration or criteria setting method sets a tunable element indirectly whereby the user, super user, administrator, search criteria tune or set an intuitively understandable criteria for search (e.g., computer code search, contract search, math article search, etc.) which in turn adjust multiple weights or added factors of features/elements in the relevancy score. For example, a contract search adds a greater weight to a document-type attribute of contract, a format-type attribute of PDF, and a department-type attribute of legal; the contract search can also tune clarity to lower the score of documents with computer code, mathematical symbols, or other non-English text type content.
In some embodiments, tuning includes a conditional favoring of search results with attribute B=‘Y’ if attribute A=‘X’. For example, an administrator sets that if content-type is ‘contract,’ then relevancy score is higher for documents are format-type ‘PDF’. In some embodiments, the conditional tuning includes if attribute A=‘X’, then relevancy score uses a added or multiplied constant (e.g., +Z or *Z) or function (e.g., +f′(input_parameter) or *f′(input_parameter) instead of constant or other function).
A relevancy scoring of a result that is returned from searching heterogeneous sources (e.g., repositories, web resource, databases, etc.) is computed based on a set of independent features. These features simulate evaluation criteria of an end user. The features include use of query structure and data structure. The features are combined into a single score using a weight based formula. This enables the features to be tuned as well depending on a user, super user, administrator, or search criteria for searching. In various embodiments, the tuning includes adding or multiplying a constant or a function to one or more of the factors included in the relevancy score calculation. In some embodiments, the tuning is achieved by a user, super user, administrator or search criteria that indirectly sets weights or adds or multiplies constants or factors in a relevancy score calculation. For example, a code searching tuning, a medical article tuning, a contract tuning, etc. may have different constants or functions adding or multiplying the different features that are added/multiplied together for calculating a relevancy score. In some embodiments, the rank is received as a metadata associated with a search result from a search source. In some embodiments, the feature includes a constraint which includes an operator such as CONTAINS, DOES_NOT_CONTAIN, BEGINS_WITH, ENDS_WITH, ALWAYS_WITHIN, AND, OR, NOT, EQUALS, GREATER_THAN, LESS_THAN, GREATER_THAN_OR_EQUAL_TO, LESS_THAN_OR_EQUAL_TO, or any other appropriate operator.
In various embodiments, a feature includes scope/depth, accuracy/validity, clarity, currency, quality of source, or any other appropriate feature useful in calculating a relevancy score.
In some embodiments, a score for each result can be computed on the fly. That is, a whole set of results does not have to be received to be able to compute the score by comparing them to each other. Each result gets its own score computable based on its own content and/or metadata. After this, there is a sorting by score (and maybe resorting as additional results are received, e.g., from additional sources). In some embodiments, there is a merging based on the score results or rank results from two scoring or ranking systems.
In some embodiments, computing a score for each result on the fly as that result is received is disclosed, based on the result itself and not a relative comparison of results, and applying to metadata the criteria applied previously only to the structured content itself. In some embodiments, the metadata or attributes of the search result are synthesized (e.g., a title, a date, an abstract, a body, etc.) from the search result.
FIG. 1 is a block diagram illustrating an embodiment of a system for relevancy scoring. In the example shown, one or more users connect to web-based content management application 102 using a computer system, represented in FIG. 1 by computer 100. In the example shown, web-based content management application 102 is built on a content management system vendor-provided web development kit (WDK) 104. In some embodiments, WDK 104 comprises a set of tools to facilitate development of web-based applications, e.g., by providing reusable components pre-configured to interact with a content server and/or other repository to provide content management services. Web-based content management application 102 in some embodiments provides to users content type aware content management services and/or functions. An example of a content type aware service or function is providing a view (display) based at least in part on content item type, e.g., displaying a “calendar” as a grid and an email message with “to”, “from”, “subject”, and/or other fields in a header display region at the top of the display, attachment names/icons in an attachment display region below the header, and message text in a message body display region at the bottom; enabling a user to search by content type; and displaying query results in a manner determined based at least in part on content type. In various embodiments, content types comprise one or more of the following: contracts, news, code, financial, spread sheet, portable document format (PDF), HTML, XML, graphics, multimedia, or any other appropriate content type.
Web-based content management application 102 also includes search services 106. In some embodiments, a user requests a search to a content management system. The content management service searches both the internal repositories of the content management system and external repositories and other attached resources. The content management system receives the search request and the WDK 104 translates the search request for search services 106. Search services 106 translates the search request for a direct search of internal content management system repository 108 and an indirect search of external repositories/resources via enterprise content integration server 110. In some embodiments, search services 106 are provided by business logic and/or foundation classes that comprise a content management framework on which web-based content management application 102 and WDK 104 are based.
Search services 106 computes a score that is used to merge and rank results from different sources including external repositories/resources and internal sources/repositories for display to the user. The score is computed based on a probability of interest of the user performing a search. The score is based on the match of a result to the query terms and the precision of the query.
Enterprise content integration (ECI) server 110 targets the search request for a search of one or more external repositories, represented in FIG. 1 by 114 and 118, and/or other resources, represented in FIG. 1 by 122. In various embodiments, external repositories comprise other content management systems, database systems, legacy content management systems not capable of the type of search desired, (e.g., Filenet™, LexisNexis™, Lotus Notes™, etc.). In various embodiments, other resources include information sources or search services—for example, Google™, Yahoo™, MSN™, MySimon™, Pricewatch™, AskJeeves™, etc. The targeted search request is translated and adapted using a configuration adapter, represented in FIG. 1 by configuration adapter 112, 116, and 120. The configuration adapter also translates and adapts the responses to the search request and passes the responses to ECI server 110. In some embodiments, the configuration adapter includes associated information in the response to the search request.
In some embodiments, the associated information is ultimately used as metadata information that enables the content management system to treat the search results as typed information for the results of the search. In some embodiments, type information is synthesized for search results based on information in search result or information about sources of the search results. In some embodiments, a query broker (not shown) at the content management system is configured to communicate with the ECI server 110 and to use information about the external content sources accessed via ECI server 110 to associate type information with results received from ECI server 110. The results from ECI server 110 in some embodiments indicate the external content source for each responsive content item (or group of items), and the query broker at the content management system associates type information with each content item (or set of content items) based on the source, as identified for example by data associated with a configuration file and/or adapter used to retrieve and/or identify the content from the external source. In some embodiments, more than one configuration adapter is associated with a specific repository or resource—for example, one configuration adapter is configured to facilitate searching for contract information in a specific database and another configuration adapter is configured to facilitate searching for financial information or specification information in the same specific database. In some embodiments, a configuration adapter is easily configured by a user, system administrator, system manager using an application with a graphical user interface (e.g., a WYSIWYG editor) to establish general parsing and syntax rules for communication between the ECI server and the external repository or resource. In various embodiments, the configuration adapter configuration is based on preconfigured templates, has a self learning engine in order to self-configure based at least in part on the external repository type and/or external resource type.
FIG. 2 is a flow diagram illustrating an embodiment of a process for relevancy scoring. In the example shown, in 200 a query is received including constraints. In various embodiments, constraints in a query include constraints using the one or more of the following operators: DOES NOT CONTAIN, CONTAINS, BEGINS WITH, ENDS WITH, AND, OR, EQUALS, GREATER THAN, LESS THAN, DOES NOT EQUAL, or any other appropriate operator. In 202, search results(s) are received including attribute(s). An attribute comprises a metadata attached to a search result; in a typical case, the search result comprises a document. In various embodiments, an attribute comprises a title or document name, a creation or modification date, a summary or abstract for the document, a document type, or any other appropriate attribute. In some embodiments, attributes are called properties and include a title, a category, keywords, etc. In 204, a relevancy score is calculated. The relevancy score evaluates relevancy using a small set of criteria that are similar to what user uses to evaluate relevancy. The criteria include measures of depth/scope, accuracy/validity, clarity, currency, and quality of sources. The criteria are then combined using weights. In 206, a relevancy ranking is calculated based on the relevancy score. Scores are evaluated with respect to each other and other scores, and in some cases merged, to determine a ranking. In some embodiments, scores are weighted or normalized while being evaluated and/or merged for the ranking.
FIG. 3 is a flow diagram illustrating an embodiment of a process for determining relevancy score. In some embodiments, the process of FIG. 3 is used to implement 204 of FIG. 2. In the example shown, in 300 a scope/depth score is calculated. The scope/depth score uses constraints with the operators: CONTAINS, DOES_NOT_CONTAIN, ALWAYS_WITHIN, BEGINS_WITH, and ENDS_WITH. Operators like EQUALS, GREATER_THAN, LESS_THAN, AND, NOT, and OR are measured using the accuracy/validity score. The scope/depth score includes a measure of the number of CONTAINS-like constraints that are satisfied. Attributes with multiple constraints are not penalized in the scope/depth score, and attributes with only one constraint are not weighted too heavily. One goal for the scope/depth score is that a satisfied constraint must always increase the scope/depth score; for example, if in a first case we have only one constraint on the title attribute, and in a second case we have the same constraint plus a second satisfied constraint on title, then the second case must always be assigned a greater score than the first case. Another goal for the scope/depth score is that an unsatisfied constraint must always decrease the score. For example, if in a first case we have only one constraint on the title attribute, and in a second case we have the same constraint plus a second unsatisfied constraint on tile, then the second case must always be assigned a lower score than the first case. In addition, a bonus score is given to a primary attribute (e.g., a name or title of a document) that matches its constraint. If the primary attribute does not match its specific constraint, then try to match its constraint with the result full-text content.
In some embodiments, a scope/depth score uses the following:
    • If there is no attribute with contains-like constraints, then
      Score[scope/depth]=a constant value such as 0.9;
    • If there are attributes with contains-like constraints, then
      Score[scope/depth]=Σ(nbCTtotal[attribute]*Weight[attribute]* Score[attribute])/Σ(nbCTtotal[attribute]*Weight[attribute]);
    • Where
    • Σ:sum for all attributes;
    • nbCTtotal[attribute] is the number of contains-like constraints (satisfied or not) for the attribute (just counting CONTAINS like constraints, ignoring all EQUALS, GREATER_THAN like constraints), counting up to a maximum of constraints (e.g., four constraints);
    • Weight[attribute] is the weight factor for a specific attribute, a real number value that is greater than or equal to 0; Note that in some embodiments, this value can be set by an user, a super user, and/or administrator; One can define a weight for any specific attribute, this weight is relative to other weights and is designed to represent the importance of the attribute compared to (e.g., in relation to) other attributes. By default all attributes have a neutral weight of 1; For example, it is desired to increase the importance of the ‘summary’ attribute because it contains relevant data in the system (e.g., more relevant than other attributes' data); So, a weight of 2 is set for it; Each match in this particular attribute will improve the score twice as much as other matches in other attributes; For another example, this weighting can be used to increase the importance of all ‘*format*’ attributes (i.e., attributes whose name contains ‘format’) or the importance of attributes with the value ‘*pdf*’ (e.g., in the event that PDF formatted documents are important compared to other documents that match in other formats) by setting a weight of 4 for them;
      Score[attribute]=(nbCTok[attribute]+bonusOcc)/nbCTtotal[attribute]* Fscope(nbCTtotal[attribute]);
    • If the attribute is primary and has a positive score, then multiply it by the Primary Bonus (e.g., a value such as 1.3);
    • If the attribute is primary but has a zero score, then recompute its score by comparing its constraint with the result of the full-text content score and multiply the score by the Primary Full-Text Factor (e.g., a value such as 0.3); Note that in some embodiments, this value can be set by an user, a super user, and/or administrator; Normally, the full-text content is the full textual content of the document; But in some cases (e.g., search), the results do not provide a document's full text content (e.g., for performance reasons), they only provide document metadata. In this case, an artificial ‘full-text content’ attribute can be constructed by aggregating the values of all or some (e.g., excepting particular ones based on a configuration) attributes;
    • nbCTok[attribute] is the number of satisfied contains-like constraints for a specific attribute, counting up to a maximum of number satisfied constraints (e.g., four constraints);
    • bonusOcc is added (e.g., a value of 1) if the CONTAINS constraint value occurs multiple times. The score may be greater than 1 in rare cases;
    • Fscope(x) is a function that takes a number of constraints as parameter and returns a real number value within [0,1]; Fscope(x) increases as x is increased; and
    • Note that score[scope/depth] may be greater than 1 in rare cases.
In some embodiments, the scope/depth score is tuned for a particular user, super user, administrator, type of search, or any other appropriate tuning. For example, a search for patents may set the primary attribute as the abstract whereas a search for doctoral theses may set the primary attribute as the title. In some embodiments, bonus weight or attribute weights are tuned.
In 302, an accuracy/validity score is calculated. The scores for accuracy and validity are merged because it is not a goal for the accuracy/validity score to count a constraint that improves the query precision, where the constraint is unsatisfied. For the accuracy/validity score, some constraint operators are considered more precise than others; For example, EQUALS is more precise than GREATER_THAN (or LESS_THAN_OR_EQUAL_TO, etc.), which is more precise than CONTAINS. Also, for the accuracy/validity score an AND (or NOT) operator improves the query precision with multiple constraints, but an OR operator does not improve much the query precision with multiple constraints. In addition, each satisfied constraint must be taken into account for the accuracy/validity score, not only the overall constraint with logical operators.
In some embodiments, a accuracy/validity score uses the following:
Score[accuracy/validity]=Score[overall constraint];
    • Where
    • Score[overall constraint] is computed by visiting the overall constraint tree that mixes all attributes;
    • Score[single constraint] depends on whether it is satisfied (score=0 if unsatisfied), and if it is satisfied it depends on the type of the constraint: for example, EQUAL=1, GREATER_THAN=0.95, ENDS_WITH=0.9, CONTAINS=0.85, DOESN'T_CONTAIN=0.8; Score[single constraint] always has a value within [0,1];
      Score[AND]=Σ(constraint score*constraint weight)/Σ(constraint weight)* Facc_and(nb constraint total);
    • Σ is the sum for all ANDed constraints;
      Score[OR]=Σ(constraint score*constraint weight)/Σ(constraint weight)* Facc_or(nb constraint total);
    • Σ is the sum for all ORed constraints;
    • Constraint weight is the weight of the constraint attribute; Note that in some embodiments, this value can be set by an user, a super user, and/or administrator; One can define a weight for any specific constraint attribute, this weight is relative to other weights and is designed to represent the importance of the attribute compared to (e.g., in relation to) other attributes. By default all attributes have a neutral weight of 1; For example, it is desired to increase the importance of the ‘summary’ attribute because it contains relevant data in the system (e.g., more relevant than other attributes' data); So, a weight of 2 is set for it; Each match in this particular attribute will improve the score twice as much as other matches in other attributes; For another example, this weighting can be used to increase the importance of all ‘*format*’ attributes (i.e., attributes whose name contains ‘format’) or the importance of attributes with the value ‘*pdf*’ (e.g., in the event that PDF formatted documents are important compared to other documents that match in other formats) by setting a weight of 4 for them;
    • Facc_and(x) is a function that takes a number of constraint as parameter (unlimited positive integer) and returns a real number value within [0,1];
    • Facc_and(x) provides an output value that increases with increasing values of x;
    • Facc_or(x) is a function that takes a number of constraint as parameter (unlimited positive integer) and returns a real number value within [0,1];
    • Facc_or(x) provides an output value that decreases with increasing values of x; and
    • Score[accuracy/validity] output value is within [0,1].
In various embodiments, for the scope/depth score and/or accuracy/validity score, each time a query term is matched it is checked to see if it is a full-word, same-stem-word, or a sub-word match. In this way, the scope/depth score and/or accuracy/validity score can favour full-word matches that are obviously more relevant to the user. Other matches do not indicate as precisely whether the document being scored is relevant for the user. For example, a full-word match counts as a 1.0 match. A same-stem-word match counts as 0.7 match. A sub-word match counts as 0.3 match. So, if the query term to look for is ‘test’, then: The full-text “Test suite” will also get one full match; The full-text “document about testing” will get 0.7 match; And, the full-text “Attestation” will get 0.3 match.
In some embodiments, the validity/accuracy score is tuned for a particular user, super user, administrator, type of search, or any other appropriate tuning. For example, a search for products may set the constraint weight with different values as compared to a search for human resource record.
In 304, a clarity score is calculated. The clarity score is designed to capture the extent to which information is presented in a clear and well-organized manner. For the clarity score the presence of special attributes (e.g., title, abstract, or date) are counted even if their associated constraints are not satisfied. Also, the attributes are checked for readability. For example, the attributes are checked to see if they are not a file name, not a uniform record locator (URL), or that the proportion of special characters (e.g., ‘/’, ‘.’, ‘\’, ‘<’, ‘>’, ‘?’, ‘&’, etc.) is not high, or any other appropriate measure for clarity.
In some embodiments, a clarity score uses the following:
Score[clarity]=Fclar(nbClearAttributes);
    • Where
    • Fclar(x) is a function that takes a number of clear attributes as parameter and returns a real within [0,1]; Fclar(x) output value increases as x increases;
    • nbClearAttributes is the number of clear attributes (counting only the special attributes that are readable); and
    • Score[clarity] output value is within [0,1].
In various embodiments, clarity function includes a tuning for specialized searches—for example, emphasis given to the presence or absence of mathematical, scientific, medical, computer code, legal terms, or any other appropriate specialized clarity terms. For example a search for code, for math or science or medical records may have a higher value for content that includes a high degree of math symbols, computer code, scientific, medical, or any other appropriate terminology. In some embodiments, an automated analysis is used to determine clarity.
In 306, a currency score is calculated. The currency score analyzes the date attribute and gives a better score to recent results than to old results. In some embodiments, recent and old can have different time frames dependent on the content; for example, stock news is considered recent with a different time scale as compared to housing prices.
In some embodiments, a currency score uses the following:
Score[currency]=Fcurr(date);
    • Where
    • Fcurr(x) is a function that takes a date as parameter and returns a real number value within [0,1]; For a recent date, the function returns a real number value near 1; For an old date, the function returns a real number value near 0. In some embodiments, the maximum age is capped—for example, dates older than five years are considered to be five years old, or, in the case of a faster changing information arena, dates older than four days are considered to be four days old; Fcurr(x) has output that decreases as a function of increasing x; and
    • Score[currency] is within [0,1].
In some embodiments, currency score may be a peaked function (e.g., pyramid, Gaussian, or any other peaked function) with the peak corresponding to a date of interest and a width corresponding to an appropriate time scale width. For example, a search may be performed regarding a weather event where documents are given highest priority on the day of the weather event and lower priorities just prior and after, but dwindling down to no priority for documents more than a week before or a week after. In some embodiments, the peaked function can be asymmetric where there is no interest in content from before the date (e.g., only interested in the after-effects or analysis regarding an event or date) or after the date (e.g., only interested in content predictions or speculations prior to an event). In some embodiments, a tuning of the currency function for a specialized search allows the adjustment of the time weighting function including the peak, the width, and the shape of the function.
In 308, a quality of source score is calculated. The quality of source score provides a means to enhance sources based on an assessment of quality of the source. In some embodiments, a quality of source score uses the following:
Score[source]=Fsrc(resultRank[source])*RankConfidence[source];
    • Where
    • Fsrc(x) is a function that takes a result rank as parameter and returns a real number value within [0,1]; Fsrc(x) output decreases with increasing x;
    • RankConfidence[source] is a weight for the source ranking confidence, a real number value within [0,1]; This value can be set by the administrator; and
    • Score[source] output value is within [0,1].
In various embodiments, the quality or source score may emphasize a type of repository based on the type of search. For example, code searches may emphasize code repositories or web resources with code stored; news searches may emphasize news sources or repositories, financial searches may emphasize financial sites or repositories; medical searches may emphasize medical repositories or web resources. In some embodiments, the tuning of the quality of source score includes the specialized tuning for searches of legal, financial, medical, scientific, or any other appropriate source tunings.
In 310, a final relevancy score is calculated. In some embodiments, a final relevancy score uses the following:
W1=factor_scope_depth;
W2=factor_accuracy_validity;
W3=factor_clarity;
W4=factor_currency;
W5=factor_quality_of_sources*rank_confidence_in_a_specific_source;
Relevancy13 score=(W1*Score[scope/depth]+W2*Score[accuracy/validity]+W3*Score[clarity]+W4*Score[currency]+W5*Score[sources])/ (W1+W2+W3+W4+W5)+(source_bonus);
    • Where
    • factor_scope_depth is a weighting factor for the scope/depth score used when combining the scope/depth score is combined into an intermediate score S; This weighting factor is set empirically so that the resultant relevancy score matches a users assessment of relevancy;
    • factor_accuracy_validity is a weighting factor for the accuracy/validity score used when combining the accuracy/validity score is combined into an intermediate score S; This weighting factor is set empirically so that the resultant relevancy score matches a users assessment of relevancy;
    • factor_clarity is a weighting factor for the clarity score used when combining the clarity score is combined into an intermediate score S; This weighting factor is set empirically so that the resultant relevancy score matches a users assessment of relevancy;
    • factor_currency is a weighting factor for the currency score used when combining the currency score is combined into an intermediate score S; This weighting factor is set empirically so that the resultant relevancy score matches a users assessment of relevancy;
    • factor_sources is a weighting factor for the quality of sources score used when combining the quality of sources score is combined into an intermediate score S; This weighting factor is set empirically so that the resultant relevancy score matches a users assessment of relevancy;
    • rank_confidence_in_a_specific_source is a user, super-user, or administrator settable weighting for source; For each source, the confidence in the source's ranking can be defined as a value within [0,1]. ‘0’ indicates that the result rankings returned by the source are not taken into account at all. ‘1’ indicates that the result rankings returned by the source are taken into account completely; Note that in some embodiments, this value can be set by an user, a super user, and/or administrator; and
    • source_bonus is a user, super-user, or administrator settable weighting for a specific source; This bonus is added to the score of all results coming from a specific source (or a specific type of source), depending whether the query sent to the source contains a constraint on full-text or on attributes; For example, a source bonus of 0.3 is defined for all internal sources (e.g., source of type ‘documentum docbase’) if the query sent contains a constraint on attributes; Then, if the query is ‘find documents which title contains “test”’ then all results coming from internal sources will be given a bonus of 0.3; On the other hand, if the query is ‘find documents which full-text contains “test”’, then no bonus is applied; As another example, for all internal sources a 0.3 bonus is defined for queries with constraint on attribute, and another 0.2 bonus for queries with constraint on full-text; Then, if the query contains both constraints, the results returned by internal sources will be given a 0.3 bonus (i.e., the maximum of 0.3 and 0.2); Note that in some embodiments, this value can be set by an user, a super user, and/or administrator.
In some embodiments, the relevancy score is merged with an internal source (e.g., Documentum database score) or other source score using the formula:
Merged_score=(Relevancy_score+Source_score*rating_weight_of_the_source)/ (1+rating_weight_of_the_source);
    • Where
    • Source_score is the score from the internal or other source; and
    • Rating weight of the source is the relative weight of the internal source; With this weight the score of some important sources can be favoured or, on the other hand, the score of some unreliable sources can be ignored.
      In some embodiments, the merging occurs when calculating or determining a relevancy ranking (e.g., as in 206 of FIG. 2).
In some embodiments, there is a tuning of the relevancy score for a specific application, source, search, user, a super user, an administrator or any other appropriate tuning Tuning may affect one or multiple parameters in the score or ranking. In some embodiments, the tuning may include adding or multiplying one or more constants or one or more functions to the score or ranking algorithm to have a desired effect. For example, a user may perform many searches for medical articles so that emphasis is given to current articles (e.g., within the past three years) from journals (e.g., from journal sites, medical article sources/repositories, reputable web sites, etc.) with good reputations that include a high degree of medical terminology (e.g., medical specialty terms in the title, abstract, and body). In another example, a person may search for events near and around a family member's birthday so that emphasis is given to content from that date from sources showing current events (e.g., news repositories, blogs, video sites, etc.) with common usage clarity (e.g., no computer code).
FIG. 4 is a flow diagram illustrating an embodiment of a process for tuning a relevancy score. In the example shown, in 400 an indication is received that a relevancy score is to be tuned. A user, a super user, an administrator, or any other searcher indicates a tuning by adjusting a weight/constant/function shape for a set of user-friendly criteria. In various embodiments, the indication is using a graphical user interface, a configuration file, a script, or any other appropriate manner of indicating a tuning. In various embodiments, a weight or constant is adjusted using a button, radio dial, slider switch, a numerical value, or any other appropriate manner of adjusting a weight or constant. In various embodiments, a function shape is selected by indicating a button near a graphic of a function graph, a math formula, or a self drawn function shape, or any other appropriate manner of selecting a function shape. In 402, tuning of elements of the relevancy score is determined. Each part of the relevancy score includes a number of elements that are added or multiplied or have a functional form. Each element is adjusted appropriately based on the input easily adjusted user selection. For example, an indication of a computer code search would favor sources that are code repositories and search results with clarity score that was adjusted to include computer code vocabulary as well as symbols associated with computer coding. The computer code search would also favor more recent results to avoid compatibility issues with older operating systems/plug-ins/compilers/hardware environments etc.
FIG. 5 is a flow diagram illustrating an embodiment of a process for tuning of elements of the relevancy score. In some embodiments, the process of FIG. 5 is used to implement the 402 of FIG. 4. In the example shown, in 500 scope/depth score elements are tuned, if appropriate. In 502, accuracy/validity score elements are tuned, if appropriate. In 504, clarity score elements are tuned, if appropriate. In 506, currency score elements are tuned, if appropriate. In 508, quality of source score elements are tuned, if appropriate. In 510, final relevancy score elements are tuned, if appropriate.
For example, in terms of the relevancy score as calculated in 310 of FIG. 3 (and similarly for the other scores), W1, W2, W3, W4, W5 (including both factor_quality_of_sources and rank_confidence_in_a_specific_source), and source_bonus may be tuned by adding or multiplying a factor or function. As another example, in terms of the currency score as calculated in 306 of FIG. 3 (and similarly for functions for the other scores), the function fcurr(date) can take on a different form being any specified shape with value sloping up or down, saturating, or peaking before, after, at, or near the date input as an input parameter.
Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.

Claims (19)

What is claimed is:
1. A system for determining a relevancy ranking score, comprising:
a processor; and
a memory coupled with the processor, wherein the memory is configured to provide the processor with instructions which when executed cause the processor to:
receive an indication of a selection that a relevancy ranking score is to be tuned to a selected preference; and
update a relevancy ranking score algorithm based at least in part on the indication of the selection of the selected preference, wherein the relevancy ranking score of a search result resulting from a search query is based at least in part on one or more constraints of the search query, wherein the relevancy ranking score of the search result is based at least in part on a currency score of the search result, wherein the currency score comprises a score indicating recent results relative to a date of interest associated with the search query, wherein the currency score is based at least in part on a currency function evaluating a date attribute, and wherein updating the relevancy ranking score includes adjusting a peak, a width, and a shape of the currency function, wherein the peak corresponds to a date of interest, wherein the width corresponds to an appropriate time scale width, and wherein the shape of the currency function includes a peaked shape.
2. The system of claim 1, wherein the relevancy ranking score is used to determine a search result ranking for a plurality of search results resulting from the search query.
3. The system of claim 1, wherein the memory is further configured to provide the processor with instructions which when executed cause the processor to calculate the relevancy ranking score for the search result based on the updated relevancy ranking score algorithm.
4. The system of claim 1, wherein the relevancy ranking score includes a conditional favoring of the search result with attribute B=‘Y’ if attribute A=‘X’.
5. The system of claim 1, wherein the search result is one of a plurality of search results from a plurality of search sources.
6. The system of claim 1, wherein updating the relevancy ranking score algorithm includes adding a constant to an element of the relevancy ranking score algorithm.
7. The system of claim 1, wherein updating the relevancy ranking score algorithm includes multiplying by a constant an element of the relevancy ranking score algorithm.
8. The system of claim 1, wherein updating the relevancy ranking score algorithm includes adding a function to an element of the relevancy ranking score algorithm.
9. The system of claim 1, wherein updating the relevancy ranking score algorithm includes multiplying by a function an element of the relevancy ranking score algorithm.
10. The system of claim 1, wherein the relevancy ranking score of the search result is based at least in part on a clarity score of the search result.
11. The system of claim 1, wherein the relevancy ranking score of the search result is based at least in part on one or more attributes that are synthesized from the search result.
12. A computer program product for determining a relevancy ranking score, the computer program product being embodied in a non-transitory computer readable medium and comprising computer instructions for:
receiving an indication of a selection that a relevancy ranking score is to be tuned to a selected preference; and
updating a relevancy ranking score algorithm based at least in part on the indication of the selection of the selected preference, wherein the relevancy ranking score of a search result resulting from a search query is based at least in part on one or more constraints of the search query, wherein the relevancy ranking score of the search result is based at least in part on a currency score of the search result, wherein the currency score comprises a score indicating recent results relative to a date of interest associated with the search query, wherein the currency score is based at least in part on a currency function evaluating a date attribute, and wherein updating the relevancy ranking score includes adjusting a peak, a width, and a shape of the currency function, wherein the peak corresponds to a date of interest, wherein the width corresponds to an appropriate time scale width, and wherein the shape of the currency function includes a peaked shape.
13. A method of determining a relevancy ranking score, comprising:
receiving an indication that a relevancy ranking score algorithm is to be tuned to a selected preference; and
updating, using a processor, the relevancy ranking score algorithm based at least in part on the selected preference, wherein the relevancy ranking score of a search result resulting from a search query is based at least in part on one or more constraints of the search query, wherein the relevancy ranking score of the search result is based at least in part on a currency score of the search result, wherein the currency score comprises a score indicating recent results relative to a date of interest associated with the search query, wherein the currency score is based at least in part on a currency function evaluating a date attribute, and wherein updating the relevancy ranking score includes adjusting a peak, a width, and a shape of the currency function, wherein the peak corresponds to a date of interest, wherein the width corresponds to an appropriate time scale width, and wherein the shape of the currency function includes a peaked shape.
14. The method of claim 13, wherein the relevancy ranking score is used to determine the search result ranking for a plurality of search results resulting from the search query.
15. The method of claim 13, further comprising calculating the relevancy ranking score for the search result based on the updated relevancy ranking score algorithm.
16. The method of claim 13, wherein the relevancy ranking score includes a conditional favoring of the search result with attribute B=‘Y’ if attribute A=‘X’.
17. The method of claim 13, wherein the search result is one of a plurality of search results from a plurality of search sources.
18. The method of claim 13, wherein updating the relevancy ranking score algorithm includes adding or multiplying by a constant or a function to an element of the relevancy ranking score algorithm.
19. The method of claim 13, wherein the relevancy ranking score of the search result is based at least in part on one or more attributes that are synthesized from the search result.
US12/834,710 2007-06-29 2010-07-12 Tuning of relevancy ranking for federated search Active US8131716B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US12/834,710 US8131716B2 (en) 2007-06-29 2010-07-12 Tuning of relevancy ranking for federated search

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US11/824,304 US7783630B1 (en) 2007-06-29 2007-06-29 Tuning of relevancy ranking for federated search
US12/834,710 US8131716B2 (en) 2007-06-29 2010-07-12 Tuning of relevancy ranking for federated search

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
US11/824,304 Continuation US7783630B1 (en) 2007-06-29 2007-06-29 Tuning of relevancy ranking for federated search

Publications (2)

Publication Number Publication Date
US20100274783A1 US20100274783A1 (en) 2010-10-28
US8131716B2 true US8131716B2 (en) 2012-03-06

Family

ID=42583426

Family Applications (2)

Application Number Title Priority Date Filing Date
US11/824,304 Active 2028-06-17 US7783630B1 (en) 2007-06-29 2007-06-29 Tuning of relevancy ranking for federated search
US12/834,710 Active US8131716B2 (en) 2007-06-29 2010-07-12 Tuning of relevancy ranking for federated search

Family Applications Before (1)

Application Number Title Priority Date Filing Date
US11/824,304 Active 2028-06-17 US7783630B1 (en) 2007-06-29 2007-06-29 Tuning of relevancy ranking for federated search

Country Status (1)

Country Link
US (2) US7783630B1 (en)

Cited By (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120030201A1 (en) * 2010-07-30 2012-02-02 International Business Machines Corporation Querying documents using search terms
US20130262460A1 (en) * 2012-03-28 2013-10-03 Microsoft Corporation Publisher Ecosystem for Preferring Search Sources
US20170169111A1 (en) * 2015-12-09 2017-06-15 Oracle International Corporation Search query task management for search system tuning
US9760608B2 (en) 2013-11-01 2017-09-12 Microsoft Technology Licensing, Llc Real-time search tuning
US9959356B2 (en) 2012-11-02 2018-05-01 Swiftype, Inc. Automatically modifying a custom search engine for a web site based on administrator input to search results of a specific search query
US10540365B2 (en) 2014-05-30 2020-01-21 Apple Inc. Federated search
US10885039B2 (en) 2014-05-30 2021-01-05 Apple Inc. Machine learning based search improvement
US11023520B1 (en) 2012-06-01 2021-06-01 Google Llc Background audio identification for query disambiguation
US11308097B2 (en) 2018-12-13 2022-04-19 Yandex Europe Ag Method of and server for generating meta-feature for ranking documents
US11409755B2 (en) 2020-12-30 2022-08-09 Elasticsearch B.V. Asynchronous search of electronic assets via a distributed search engine
US11734279B2 (en) 2021-04-29 2023-08-22 Elasticsearch B.V. Event sequences search
US11899677B2 (en) 2021-04-27 2024-02-13 Elasticsearch B.V. Systems and methods for automatically curating query responses

Families Citing this family (95)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7975142B2 (en) * 2006-12-04 2011-07-05 Electronics And Telecommunications Research Institute Ring authentication method for concurrency environment
US7783620B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Relevancy scoring using query structure and data structure for federated search
US7783630B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Tuning of relevancy ranking for federated search
US8713001B2 (en) * 2007-07-10 2014-04-29 Asim Roy Systems and related methods of user-guided searching
KR101136726B1 (en) * 2009-06-15 2012-05-04 차영철 User-generated rule based multi-function search box service method and system
US9633121B2 (en) 2010-04-19 2017-04-25 Facebook, Inc. Personalizing default search queries on online social networks
CN103229168B (en) * 2010-09-28 2016-10-19 国际商业机器公司 The method and system that evidence spreads between multiple candidate answers during question and answer
US8924394B2 (en) * 2011-02-18 2014-12-30 Mmodal Ip Llc Computer-assisted abstraction for reporting of quality measures
US9043358B2 (en) 2011-03-09 2015-05-26 Microsoft Technology Licensing, Llc Enterprise search over private and public data
US9959326B2 (en) * 2011-03-23 2018-05-01 International Business Machines Corporation Annotating schema elements based on associating data instances with knowledge base entities
EP2727017A4 (en) * 2011-06-30 2015-06-24 Nokia Corp Method and apparatus for providing user-corrected search results
US11010432B2 (en) * 2011-10-24 2021-05-18 Imagescan, Inc. Apparatus and method for displaying multiple display panels with a progressive relationship using cognitive pattern recognition
US8935255B2 (en) 2012-07-27 2015-01-13 Facebook, Inc. Social static ranking for search
US9619528B2 (en) * 2012-11-02 2017-04-11 Swiftype, Inc. Automatically creating a custom search engine for a web site based on social input
US9398104B2 (en) * 2012-12-20 2016-07-19 Facebook, Inc. Ranking test framework for search results on an online social network
US9223826B2 (en) 2013-02-25 2015-12-29 Facebook, Inc. Pushing suggested search queries to mobile devices
US9514191B2 (en) 2013-03-14 2016-12-06 Microsoft Technology Licensing, Llc Visualizing ranking factors for items in a search result list
US9659058B2 (en) 2013-03-22 2017-05-23 X1 Discovery, Inc. Methods and systems for federation of results from search indexing
US9910887B2 (en) 2013-04-25 2018-03-06 Facebook, Inc. Variable search query vertical access
US9330183B2 (en) 2013-05-08 2016-05-03 Facebook, Inc. Approximate privacy indexing for search queries on online social networks
US9223898B2 (en) 2013-05-08 2015-12-29 Facebook, Inc. Filtering suggested structured queries on online social networks
US9305322B2 (en) 2013-07-23 2016-04-05 Facebook, Inc. Native application testing
US11238056B2 (en) 2013-10-28 2022-02-01 Microsoft Technology Licensing, Llc Enhancing search results with social labels
US9542440B2 (en) 2013-11-04 2017-01-10 Microsoft Technology Licensing, Llc Enterprise graph search based on object and actor relationships
US9336300B2 (en) 2014-01-17 2016-05-10 Facebook, Inc. Client-side search templates for online social networks
US11645289B2 (en) 2014-02-04 2023-05-09 Microsoft Technology Licensing, Llc Ranking enterprise graph queries
US9870432B2 (en) 2014-02-24 2018-01-16 Microsoft Technology Licensing, Llc Persisted enterprise graph queries
US11657060B2 (en) 2014-02-27 2023-05-23 Microsoft Technology Licensing, Llc Utilizing interactivity signals to generate relationships and promote content
US10757201B2 (en) 2014-03-01 2020-08-25 Microsoft Technology Licensing, Llc Document and content feed
US10255563B2 (en) 2014-03-03 2019-04-09 Microsoft Technology Licensing, Llc Aggregating enterprise graph content around user-generated topics
US10169457B2 (en) 2014-03-03 2019-01-01 Microsoft Technology Licensing, Llc Displaying and posting aggregated social activity on a piece of enterprise content
US10394827B2 (en) 2014-03-03 2019-08-27 Microsoft Technology Licensing, Llc Discovering enterprise content based on implicit and explicit signals
US10346550B1 (en) 2014-08-28 2019-07-09 X1 Discovery, Inc. Methods and systems for searching and indexing virtual environments
US10061826B2 (en) 2014-09-05 2018-08-28 Microsoft Technology Licensing, Llc. Distant content discovery
US10002165B2 (en) 2014-11-07 2018-06-19 International Business Machines Corporation Holistic federated ranking algorithm
US10628636B2 (en) 2015-04-24 2020-04-21 Facebook, Inc. Live-conversation modules on online social networks
US10298535B2 (en) 2015-05-19 2019-05-21 Facebook, Inc. Civic issues platforms on online social networks
US10509834B2 (en) 2015-06-05 2019-12-17 Apple Inc. Federated search results scoring
US10592572B2 (en) 2015-06-05 2020-03-17 Apple Inc. Application view index and search
US10621189B2 (en) 2015-06-05 2020-04-14 Apple Inc. In-application history search
US10509833B2 (en) 2015-06-05 2019-12-17 Apple Inc. Proximity search scoring
US10755032B2 (en) 2015-06-05 2020-08-25 Apple Inc. Indexing web pages with deep links
US10397167B2 (en) 2015-06-19 2019-08-27 Facebook, Inc. Live social modules on online social networks
US10509832B2 (en) 2015-07-13 2019-12-17 Facebook, Inc. Generating snippet modules on online social networks
US10268664B2 (en) 2015-08-25 2019-04-23 Facebook, Inc. Embedding links in user-created content on online social networks
US10810217B2 (en) 2015-10-07 2020-10-20 Facebook, Inc. Optionalization and fuzzy search on online social networks
US9602965B1 (en) 2015-11-06 2017-03-21 Facebook, Inc. Location-based place determination using online social networks
US10795936B2 (en) 2015-11-06 2020-10-06 Facebook, Inc. Suppressing entity suggestions on online social networks
US10270868B2 (en) 2015-11-06 2019-04-23 Facebook, Inc. Ranking of place-entities on online social networks
US10534814B2 (en) 2015-11-11 2020-01-14 Facebook, Inc. Generating snippets on online social networks
US10387511B2 (en) 2015-11-25 2019-08-20 Facebook, Inc. Text-to-media indexes on online social networks
US10740368B2 (en) 2015-12-29 2020-08-11 Facebook, Inc. Query-composition platforms on online social networks
US10282434B2 (en) 2016-01-11 2019-05-07 Facebook, Inc. Suppression and deduplication of place-entities on online social networks
US10162899B2 (en) 2016-01-15 2018-12-25 Facebook, Inc. Typeahead intent icons and snippets on online social networks
US10262039B1 (en) 2016-01-15 2019-04-16 Facebook, Inc. Proximity-based searching on online social networks
US10740375B2 (en) 2016-01-20 2020-08-11 Facebook, Inc. Generating answers to questions using information posted by users on online social networks
US10216850B2 (en) 2016-02-03 2019-02-26 Facebook, Inc. Sentiment-modules on online social networks
US10242074B2 (en) 2016-02-03 2019-03-26 Facebook, Inc. Search-results interfaces for content-item-specific modules on online social networks
US10270882B2 (en) 2016-02-03 2019-04-23 Facebook, Inc. Mentions-modules on online social networks
US10157224B2 (en) 2016-02-03 2018-12-18 Facebook, Inc. Quotations-modules on online social networks
US10452671B2 (en) 2016-04-26 2019-10-22 Facebook, Inc. Recommendations from comments on online social networks
US10635661B2 (en) 2016-07-11 2020-04-28 Facebook, Inc. Keyboard-based corrections for search queries on online social networks
US10282483B2 (en) 2016-08-04 2019-05-07 Facebook, Inc. Client-side caching of search keywords for online social networks
US10223464B2 (en) 2016-08-04 2019-03-05 Facebook, Inc. Suggesting filters for search on online social networks
US10726022B2 (en) 2016-08-26 2020-07-28 Facebook, Inc. Classifying search queries on online social networks
US10534815B2 (en) 2016-08-30 2020-01-14 Facebook, Inc. Customized keyword query suggestions on online social networks
US10102255B2 (en) 2016-09-08 2018-10-16 Facebook, Inc. Categorizing objects for queries on online social networks
US10645142B2 (en) 2016-09-20 2020-05-05 Facebook, Inc. Video keyframes display on online social networks
US10026021B2 (en) 2016-09-27 2018-07-17 Facebook, Inc. Training image-recognition systems using a joint embedding model on online social networks
US10083379B2 (en) 2016-09-27 2018-09-25 Facebook, Inc. Training image-recognition systems based on search queries on online social networks
US10579688B2 (en) 2016-10-05 2020-03-03 Facebook, Inc. Search ranking and recommendations for online social networks based on reconstructed embeddings
US10311117B2 (en) 2016-11-18 2019-06-04 Facebook, Inc. Entity linking to query terms on online social networks
US10650009B2 (en) 2016-11-22 2020-05-12 Facebook, Inc. Generating news headlines on online social networks
US10235469B2 (en) 2016-11-30 2019-03-19 Facebook, Inc. Searching for posts by related entities on online social networks
US10185763B2 (en) 2016-11-30 2019-01-22 Facebook, Inc. Syntactic models for parsing search queries on online social networks
US10313456B2 (en) 2016-11-30 2019-06-04 Facebook, Inc. Multi-stage filtering for recommended user connections on online social networks
US10162886B2 (en) 2016-11-30 2018-12-25 Facebook, Inc. Embedding-based parsing of search queries on online social networks
US10607148B1 (en) 2016-12-21 2020-03-31 Facebook, Inc. User identification with voiceprints on online social networks
US11223699B1 (en) 2016-12-21 2022-01-11 Facebook, Inc. Multiple user recognition with voiceprints on online social networks
US10535106B2 (en) 2016-12-28 2020-01-14 Facebook, Inc. Selecting user posts related to trending topics on online social networks
US10489472B2 (en) 2017-02-13 2019-11-26 Facebook, Inc. Context-based search suggestions on online social networks
US10614141B2 (en) 2017-03-15 2020-04-07 Facebook, Inc. Vital author snippets on online social networks
US10769222B2 (en) 2017-03-20 2020-09-08 Facebook, Inc. Search result ranking based on post classifiers on online social networks
US11379861B2 (en) 2017-05-16 2022-07-05 Meta Platforms, Inc. Classifying post types on online social networks
US10248645B2 (en) 2017-05-30 2019-04-02 Facebook, Inc. Measuring phrase association on online social networks
US10268646B2 (en) 2017-06-06 2019-04-23 Facebook, Inc. Tensor-based deep relevance model for search on online social networks
US10489468B2 (en) 2017-08-22 2019-11-26 Facebook, Inc. Similarity search using progressive inner products and bounds
US10776437B2 (en) 2017-09-12 2020-09-15 Facebook, Inc. Time-window counters for search results on online social networks
US10678786B2 (en) 2017-10-09 2020-06-09 Facebook, Inc. Translating search queries on online social networks
US10810214B2 (en) 2017-11-22 2020-10-20 Facebook, Inc. Determining related query terms through query-post associations on online social networks
US10963514B2 (en) 2017-11-30 2021-03-30 Facebook, Inc. Using related mentions to enhance link probability on online social networks
US10129705B1 (en) 2017-12-11 2018-11-13 Facebook, Inc. Location prediction using wireless signals on online social networks
US11604968B2 (en) 2017-12-11 2023-03-14 Meta Platforms, Inc. Prediction of next place visits on online social networks
US11061877B2 (en) 2018-09-10 2021-07-13 Salesforce.Com, Inc. Indexing items in a database environment
US11586820B2 (en) * 2020-08-10 2023-02-21 Ebay Inc. Techniques for enhancing the quality of human annotation

Citations (46)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5963938A (en) 1996-05-17 1999-10-05 Novell Inc. Automatic, context-organizing, query interface
US6243713B1 (en) 1998-08-24 2001-06-05 Excalibur Technologies Corp. Multimedia document retrieval by application of multimedia queries to a unified index of multimedia data for a plurality of multimedia data types
US6263328B1 (en) 1999-04-09 2001-07-17 International Business Machines Corporation Object oriented query model and process for complex heterogeneous database queries
US6341277B1 (en) 1998-11-17 2002-01-22 International Business Machines Corporation System and method for performance complex heterogeneous database queries using a single SQL expression
US6363374B1 (en) 1998-12-31 2002-03-26 Microsoft Corporation Text proximity filtering in search systems using same sentence restrictions
US6385600B1 (en) 1997-04-03 2002-05-07 At&T Corp. System and method for searching on a computer using an evidence set
US20020073079A1 (en) 2000-04-04 2002-06-13 Merijn Terheggen Method and apparatus for searching a database and providing relevance feedback
US20020091686A1 (en) 2000-03-09 2002-07-11 The Web Access, Inc. Method and apparatus for performing a research task by interchangeably utilizing a multitude of search methodologies
US20020103787A1 (en) 2000-08-08 2002-08-01 Surendra Goel Category searching
US20020103797A1 (en) 2000-08-08 2002-08-01 Surendra Goel Displaying search results
US20020103788A1 (en) 2000-08-08 2002-08-01 Donaldson Thomas E. Filtering search results
US20030120654A1 (en) * 2000-01-14 2003-06-26 International Business Machines Corporation Metadata search results ranking system
US6622138B1 (en) * 2000-09-06 2003-09-16 Oracle International Corporation Method and apparatus for optimizing computation of OLAP ranking functions
US6694331B2 (en) 2001-03-21 2004-02-17 Knowledge Management Objects, Llc Apparatus for and method of searching and organizing intellectual property information utilizing a classification system
US6738764B2 (en) 2001-05-08 2004-05-18 Verity, Inc. Apparatus and method for adaptively ranking search results
US6785668B1 (en) 2000-11-28 2004-08-31 Sas Institute Inc. System and method for data flow analysis of complex data filters
US20040260680A1 (en) 2003-06-19 2004-12-23 International Business Machines Corporation Personalized indexing and searching for information in a distributed data processing system
US6918096B2 (en) 1996-11-07 2005-07-12 Thebrain Technologies, Corp. Method and apparatus for displaying a network of thoughts from a thought's perspective
US6944612B2 (en) 2002-11-13 2005-09-13 Xerox Corporation Structured contextual clustering method and system in a federated search engine
US6963867B2 (en) * 1999-12-08 2005-11-08 A9.Com, Inc. Search query processing to provide category-ranked presentation of search results
US20060041553A1 (en) 2004-08-19 2006-02-23 Claria Corporation Method and apparatus for responding to end-user request for information-ranking
US7133867B2 (en) 2003-02-13 2006-11-07 Sap Aktiengesellschaft Text and attribute searches of data stores that include business objects
US7181438B1 (en) 1999-07-21 2007-02-20 Alberti Anemometer, Llc Database access system
US7194454B2 (en) 2001-03-12 2007-03-20 Lucent Technologies Method for organizing records of database search activity by topical relevance
US7243093B2 (en) 2002-11-27 2007-07-10 International Business Machines Corporation Federated query management
US7249127B2 (en) 2001-05-24 2007-07-24 Microsoft Corporation Method and apparatus for deriving logical relations from linguistic relations with multiple relevance ranking strategies for information retrieval
US20070260598A1 (en) * 2005-11-29 2007-11-08 Odom Paul S Methods and systems for providing personalized contextual search results
US20070288438A1 (en) 2006-06-12 2007-12-13 Zalag Corporation Methods and apparatuses for searching content
US20070294225A1 (en) 2006-06-19 2007-12-20 Microsoft Corporation Diversifying search results for improved search and personalization
US20080033797A1 (en) * 2006-08-01 2008-02-07 Microsoft Corporation Search query monetization-based ranking and filtering
US20080104032A1 (en) 2004-09-29 2008-05-01 Sarkar Pte Ltd. Method and System for Organizing Items
US20080140521A1 (en) 2006-12-12 2008-06-12 Sivakumar Jambunathan Dynamic Modification Of Advertisements Displayed In Response To A Search Engine Query
US20080243786A1 (en) 2007-03-30 2008-10-02 Tyron Jerrod Stading System and method of goal-oriented searching
US20090006360A1 (en) * 2007-06-28 2009-01-01 Oracle International Corporation System and method for applying ranking svm in query relaxation
US7526425B2 (en) 2001-08-14 2009-04-28 Evri Inc. Method and system for extending keyword searching to syntactically and semantically annotated data
US7571157B2 (en) * 2004-12-29 2009-08-04 Aol Llc Filtering search results
US7634474B2 (en) * 2006-03-30 2009-12-15 Microsoft Corporation Using connectivity distance for relevance feedback in search
US7685097B2 (en) 2002-08-15 2010-03-23 Her Majesty The Queen In Right Of Canada As Represented By The Minister Of Health Method and system for aggregating and disseminating time-sensitive information
US20100185611A1 (en) * 2006-03-01 2010-07-22 Oracle International Corporation Re-ranking search results from an enterprise system
US7783630B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Tuning of relevancy ranking for federated search
US7783620B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Relevancy scoring using query structure and data structure for federated search
US7827181B2 (en) 2004-09-30 2010-11-02 Microsoft Corporation Click distance determination
US7840893B2 (en) 2005-04-25 2010-11-23 Kulas Charles J Display and manipulation of web page-based search results
US7849079B2 (en) * 2006-07-31 2010-12-07 Microsoft Corporation Temporal ranking of search results
US7865496B1 (en) 2004-11-30 2011-01-04 Schiller Victor H Systems, device, and methods for searching
US7945553B2 (en) 1997-02-13 2011-05-17 Nortel Networks Limited Associative search engine

Family Cites Families (21)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US744358A (en) * 1903-04-20 1903-11-17 Charles T Kinsman Dentimeter.
US6009422A (en) * 1997-11-26 1999-12-28 International Business Machines Corporation System and method for query translation/semantic translation using generalized query language
US6434556B1 (en) * 1999-04-16 2002-08-13 Board Of Trustees Of The University Of Illinois Visualization of Internet search information
US6434550B1 (en) * 2000-04-14 2002-08-13 Rightnow Technologies, Inc. Temporal updates of relevancy rating of retrieved information in an information search system
US6968332B1 (en) * 2000-05-25 2005-11-22 Microsoft Corporation Facility for highlighting documents accessed through search or browsing
US7925967B2 (en) * 2000-11-21 2011-04-12 Aol Inc. Metadata quality improvement
US6714929B1 (en) * 2001-04-13 2004-03-30 Auguri Corporation Weighted preference data search system and method
US7188106B2 (en) * 2001-05-01 2007-03-06 International Business Machines Corporation System and method for aggregating ranking results from various sources to improve the results of web searching
JP4003468B2 (en) * 2002-02-05 2007-11-07 株式会社日立製作所 Method and apparatus for retrieving similar data by relevance feedback
US20060230040A1 (en) * 2003-12-08 2006-10-12 Andy Curtis Methods and systems for providing a response to a query
US7447678B2 (en) * 2003-12-31 2008-11-04 Google Inc. Interface for a universal search engine
US7281002B2 (en) * 2004-03-01 2007-10-09 International Business Machine Corporation Organizing related search results
US7565630B1 (en) * 2004-06-15 2009-07-21 Google Inc. Customization of search results for search queries received from third party sites
US8010460B2 (en) * 2004-09-02 2011-08-30 Linkedin Corporation Method and system for reputation evaluation of online users in a social networking scheme
WO2006078794A2 (en) * 2005-01-18 2006-07-27 Yahoo! Inc. Matching and ranking of sponsored search listings incorporating web search technology and web content
US20060277098A1 (en) * 2005-06-06 2006-12-07 Chung Tze D Media playing system and method for delivering multimedia content with up-to-date and targeted marketing messages over a communication network
US20080015968A1 (en) * 2005-10-14 2008-01-17 Leviathan Entertainment, Llc Fee-Based Priority Queuing for Insurance Claim Processing
US20070271146A1 (en) * 2005-10-20 2007-11-22 Ebags.Com Method and apparatus for matching and/or coordinating shoes handbags and other consumer products
US20070208730A1 (en) * 2006-03-02 2007-09-06 Microsoft Corporation Mining web search user behavior to enhance web search relevance
CN100495398C (en) * 2006-03-30 2009-06-03 国际商业机器公司 Method for searching order in file system and correlation search engine
US20070266025A1 (en) * 2006-05-12 2007-11-15 Microsoft Corporation Implicit tokenized result ranking

Patent Citations (48)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5963938A (en) 1996-05-17 1999-10-05 Novell Inc. Automatic, context-organizing, query interface
US6918096B2 (en) 1996-11-07 2005-07-12 Thebrain Technologies, Corp. Method and apparatus for displaying a network of thoughts from a thought's perspective
US7945553B2 (en) 1997-02-13 2011-05-17 Nortel Networks Limited Associative search engine
US6385600B1 (en) 1997-04-03 2002-05-07 At&T Corp. System and method for searching on a computer using an evidence set
US6243713B1 (en) 1998-08-24 2001-06-05 Excalibur Technologies Corp. Multimedia document retrieval by application of multimedia queries to a unified index of multimedia data for a plurality of multimedia data types
US6341277B1 (en) 1998-11-17 2002-01-22 International Business Machines Corporation System and method for performance complex heterogeneous database queries using a single SQL expression
US6363374B1 (en) 1998-12-31 2002-03-26 Microsoft Corporation Text proximity filtering in search systems using same sentence restrictions
US6263328B1 (en) 1999-04-09 2001-07-17 International Business Machines Corporation Object oriented query model and process for complex heterogeneous database queries
US7181438B1 (en) 1999-07-21 2007-02-20 Alberti Anemometer, Llc Database access system
US6963867B2 (en) * 1999-12-08 2005-11-08 A9.Com, Inc. Search query processing to provide category-ranked presentation of search results
US20030120654A1 (en) * 2000-01-14 2003-06-26 International Business Machines Corporation Metadata search results ranking system
US6718324B2 (en) 2000-01-14 2004-04-06 International Business Machines Corporation Metadata search results ranking system
US20020091686A1 (en) 2000-03-09 2002-07-11 The Web Access, Inc. Method and apparatus for performing a research task by interchangeably utilizing a multitude of search methodologies
US20020073079A1 (en) 2000-04-04 2002-06-13 Merijn Terheggen Method and apparatus for searching a database and providing relevance feedback
US20020103787A1 (en) 2000-08-08 2002-08-01 Surendra Goel Category searching
US20020103788A1 (en) 2000-08-08 2002-08-01 Donaldson Thomas E. Filtering search results
US20020103797A1 (en) 2000-08-08 2002-08-01 Surendra Goel Displaying search results
US6622138B1 (en) * 2000-09-06 2003-09-16 Oracle International Corporation Method and apparatus for optimizing computation of OLAP ranking functions
US6785668B1 (en) 2000-11-28 2004-08-31 Sas Institute Inc. System and method for data flow analysis of complex data filters
US7194454B2 (en) 2001-03-12 2007-03-20 Lucent Technologies Method for organizing records of database search activity by topical relevance
US6694331B2 (en) 2001-03-21 2004-02-17 Knowledge Management Objects, Llc Apparatus for and method of searching and organizing intellectual property information utilizing a classification system
US6738764B2 (en) 2001-05-08 2004-05-18 Verity, Inc. Apparatus and method for adaptively ranking search results
US7249127B2 (en) 2001-05-24 2007-07-24 Microsoft Corporation Method and apparatus for deriving logical relations from linguistic relations with multiple relevance ranking strategies for information retrieval
US7526425B2 (en) 2001-08-14 2009-04-28 Evri Inc. Method and system for extending keyword searching to syntactically and semantically annotated data
US7685097B2 (en) 2002-08-15 2010-03-23 Her Majesty The Queen In Right Of Canada As Represented By The Minister Of Health Method and system for aggregating and disseminating time-sensitive information
US6944612B2 (en) 2002-11-13 2005-09-13 Xerox Corporation Structured contextual clustering method and system in a federated search engine
US7243093B2 (en) 2002-11-27 2007-07-10 International Business Machines Corporation Federated query management
US7133867B2 (en) 2003-02-13 2006-11-07 Sap Aktiengesellschaft Text and attribute searches of data stores that include business objects
US20040260680A1 (en) 2003-06-19 2004-12-23 International Business Machines Corporation Personalized indexing and searching for information in a distributed data processing system
US20060041553A1 (en) 2004-08-19 2006-02-23 Claria Corporation Method and apparatus for responding to end-user request for information-ranking
US7444358B2 (en) 2004-08-19 2008-10-28 Claria Corporation Method and apparatus for responding to end-user request for information-collecting
US20080104032A1 (en) 2004-09-29 2008-05-01 Sarkar Pte Ltd. Method and System for Organizing Items
US7827181B2 (en) 2004-09-30 2010-11-02 Microsoft Corporation Click distance determination
US7865496B1 (en) 2004-11-30 2011-01-04 Schiller Victor H Systems, device, and methods for searching
US7571157B2 (en) * 2004-12-29 2009-08-04 Aol Llc Filtering search results
US7840893B2 (en) 2005-04-25 2010-11-23 Kulas Charles J Display and manipulation of web page-based search results
US20070260598A1 (en) * 2005-11-29 2007-11-08 Odom Paul S Methods and systems for providing personalized contextual search results
US20100185611A1 (en) * 2006-03-01 2010-07-22 Oracle International Corporation Re-ranking search results from an enterprise system
US7634474B2 (en) * 2006-03-30 2009-12-15 Microsoft Corporation Using connectivity distance for relevance feedback in search
US20070288438A1 (en) 2006-06-12 2007-12-13 Zalag Corporation Methods and apparatuses for searching content
US20070294225A1 (en) 2006-06-19 2007-12-20 Microsoft Corporation Diversifying search results for improved search and personalization
US7849079B2 (en) * 2006-07-31 2010-12-07 Microsoft Corporation Temporal ranking of search results
US20080033797A1 (en) * 2006-08-01 2008-02-07 Microsoft Corporation Search query monetization-based ranking and filtering
US20080140521A1 (en) 2006-12-12 2008-06-12 Sivakumar Jambunathan Dynamic Modification Of Advertisements Displayed In Response To A Search Engine Query
US20080243786A1 (en) 2007-03-30 2008-10-02 Tyron Jerrod Stading System and method of goal-oriented searching
US20090006360A1 (en) * 2007-06-28 2009-01-01 Oracle International Corporation System and method for applying ranking svm in query relaxation
US7783630B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Tuning of relevancy ranking for federated search
US7783620B1 (en) * 2007-06-29 2010-08-24 Emc Corporation Relevancy scoring using query structure and data structure for federated search

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20120030201A1 (en) * 2010-07-30 2012-02-02 International Business Machines Corporation Querying documents using search terms
US8548989B2 (en) * 2010-07-30 2013-10-01 International Business Machines Corporation Querying documents using search terms
US20130262460A1 (en) * 2012-03-28 2013-10-03 Microsoft Corporation Publisher Ecosystem for Preferring Search Sources
US11640426B1 (en) 2012-06-01 2023-05-02 Google Llc Background audio identification for query disambiguation
US11023520B1 (en) 2012-06-01 2021-06-01 Google Llc Background audio identification for query disambiguation
US10467309B2 (en) 2012-11-02 2019-11-05 Elasticsearch B.V. Automatically modifying a custom search engine for a web site based on administrator input to search results of a specific search query
US9959352B2 (en) 2012-11-02 2018-05-01 Swiftype, Inc. Automatically modifying a custom search engine for a web site based on administrator input to search results of a specific search query
US9959356B2 (en) 2012-11-02 2018-05-01 Swiftype, Inc. Automatically modifying a custom search engine for a web site based on administrator input to search results of a specific search query
US10579693B2 (en) 2012-11-02 2020-03-03 Elasticsearch B.V. Modifying a custom search engine
US9760608B2 (en) 2013-11-01 2017-09-12 Microsoft Technology Licensing, Llc Real-time search tuning
US10540365B2 (en) 2014-05-30 2020-01-21 Apple Inc. Federated search
US10885039B2 (en) 2014-05-30 2021-01-05 Apple Inc. Machine learning based search improvement
US10229210B2 (en) * 2015-12-09 2019-03-12 Oracle International Corporation Search query task management for search system tuning
US20170169111A1 (en) * 2015-12-09 2017-06-15 Oracle International Corporation Search query task management for search system tuning
US11308097B2 (en) 2018-12-13 2022-04-19 Yandex Europe Ag Method of and server for generating meta-feature for ranking documents
US11409755B2 (en) 2020-12-30 2022-08-09 Elasticsearch B.V. Asynchronous search of electronic assets via a distributed search engine
US11899677B2 (en) 2021-04-27 2024-02-13 Elasticsearch B.V. Systems and methods for automatically curating query responses
US11734279B2 (en) 2021-04-29 2023-08-22 Elasticsearch B.V. Event sequences search

Also Published As

Publication number Publication date
US20100274783A1 (en) 2010-10-28
US7783630B1 (en) 2010-08-24

Similar Documents

Publication Publication Date Title
US8131716B2 (en) Tuning of relevancy ranking for federated search
US7783620B1 (en) Relevancy scoring using query structure and data structure for federated search
US10824682B2 (en) Enhanced online user-interaction tracking and document rendition
US11023513B2 (en) Method and apparatus for searching using an active ontology
Bornmann et al. The application of bibliometrics to research evaluation in the humanities and social sciences: An exploratory study using normalized G oogle S cholar data for the publications of a research institute
US10509861B2 (en) Systems, methods, and software for manuscript recommendations and submissions
US7885918B2 (en) Creating a taxonomy from business-oriented metadata content
US9171078B2 (en) Automatic recommendation of vertical search engines
US11625426B2 (en) Incorporating opinion information with semantic graph data
AU2014318392B2 (en) Systems, methods, and software for manuscript recommendations and submissions
US8019758B2 (en) Generation of a blended classification model
US10552429B2 (en) Discovery of data assets using metadata
US8171015B2 (en) Optimizing, distributing, and tracking online content
US20090012969A1 (en) Systems and processes for evaluating webpages
US20080071763A1 (en) Dynamic updating of display and ranking for search results
US20080027910A1 (en) Web object retrieval based on a language model
JP2004213675A (en) Search of structured document
US9760600B2 (en) Serving recurrent calendar events
US20210209118A1 (en) Techniques for determining relevant electronic content in response to queries
US20060036461A1 (en) Active relationship management
US20090013068A1 (en) Systems and processes for evaluating webpages
CN105975508A (en) Personalized meta-search engine searched result merging and sorting method
US20100287136A1 (en) Method and system for the recognition and tracking of entities as they become famous
KR20110127052A (en) System and method for utilizing personalized tag recommendation model in web page search

Legal Events

Date Code Title Description
AS Assignment

Owner name: EMC CORPORATION, MASSACHUSETTS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHEVALIER, PIERRE-YVES;ROUSTANT, BRUNO;REEL/FRAME:024668/0184

Effective date: 20090417

STCF Information on status: patent grant

Free format text: PATENTED CASE

FPAY Fee payment

Year of fee payment: 4

AS Assignment

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT, TEXAS

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040136/0001

Effective date: 20160907

Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT, NORTH CAROLINA

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040134/0001

Effective date: 20160907

Owner name: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLAT

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040134/0001

Effective date: 20160907

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., A

Free format text: SECURITY AGREEMENT;ASSIGNORS:ASAP SOFTWARE EXPRESS, INC.;AVENTAIL LLC;CREDANT TECHNOLOGIES, INC.;AND OTHERS;REEL/FRAME:040136/0001

Effective date: 20160907

AS Assignment

Owner name: EMC IP HOLDING COMPANY LLC, MASSACHUSETTS

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:EMC CORPORATION;REEL/FRAME:040203/0001

Effective date: 20160906

AS Assignment

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., T

Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES, INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:049452/0223

Effective date: 20190320

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., TEXAS

Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES, INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:049452/0223

Effective date: 20190320

MAFP Maintenance fee payment

Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

Year of fee payment: 8

AS Assignment

Owner name: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., TEXAS

Free format text: SECURITY AGREEMENT;ASSIGNORS:CREDANT TECHNOLOGIES INC.;DELL INTERNATIONAL L.L.C.;DELL MARKETING L.P.;AND OTHERS;REEL/FRAME:053546/0001

Effective date: 20200409

AS Assignment

Owner name: WYSE TECHNOLOGY L.L.C., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: SCALEIO LLC, MASSACHUSETTS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: MOZY, INC., WASHINGTON

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: MAGINATICS LLC, CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: FORCE10 NETWORKS, INC., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: EMC IP HOLDING COMPANY LLC, TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: EMC CORPORATION, MASSACHUSETTS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL SYSTEMS CORPORATION, TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL SOFTWARE INC., CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL PRODUCTS L.P., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL MARKETING L.P., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL INTERNATIONAL, L.L.C., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: DELL USA L.P., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: CREDANT TECHNOLOGIES, INC., TEXAS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: AVENTAIL LLC, CALIFORNIA

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

Owner name: ASAP SOFTWARE EXPRESS, INC., ILLINOIS

Free format text: RELEASE BY SECURED PARTY;ASSIGNOR:CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH;REEL/FRAME:058216/0001

Effective date: 20211101

AS Assignment

Owner name: SCALEIO LLC, MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL PRODUCTS L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL INTERNATIONAL L.L.C., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL USA L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061324/0001

Effective date: 20220329

AS Assignment

Owner name: SCALEIO LLC, MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC), MASSACHUSETTS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL PRODUCTS L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL INTERNATIONAL L.L.C., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL USA L.P., TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

Owner name: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.), TEXAS

Free format text: RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001);ASSIGNOR:THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT;REEL/FRAME:061753/0001

Effective date: 20220329

FEPP Fee payment procedure

Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY