CN104008126A - Method and device for segmentation on basis of webpage content classification - Google Patents
Method and device for segmentation on basis of webpage content classification Download PDFInfo
- Publication number
- CN104008126A CN104008126A CN201410126465.1A CN201410126465A CN104008126A CN 104008126 A CN104008126 A CN 104008126A CN 201410126465 A CN201410126465 A CN 201410126465A CN 104008126 A CN104008126 A CN 104008126A
- Authority
- CN
- China
- Prior art keywords
- participle
- word segmentation
- classification
- word
- text message
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 156
- 238000000034 method Methods 0.000 title claims abstract description 42
- 238000012545 processing Methods 0.000 claims abstract description 84
- 238000012549 training Methods 0.000 claims description 34
- 238000000605 extraction Methods 0.000 claims description 4
- 230000004044 response Effects 0.000 abstract description 3
- 230000008569 process Effects 0.000 description 7
- 235000013305 food Nutrition 0.000 description 6
- 230000007246 mechanism Effects 0.000 description 6
- 239000000463 material Substances 0.000 description 5
- 241000208340 Araliaceae Species 0.000 description 4
- 241000234295 Musa Species 0.000 description 4
- 235000018290 Musa x paradisiaca Nutrition 0.000 description 4
- 235000005035 Panax pseudoginseng ssp. pseudoginseng Nutrition 0.000 description 4
- 235000003140 Panax quinquefolius Nutrition 0.000 description 4
- 235000008434 ginseng Nutrition 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 230000015572 biosynthetic process Effects 0.000 description 3
- 230000009471 action Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000007812 deficiency Effects 0.000 description 2
- 238000013461 design Methods 0.000 description 2
- 230000003203 everyday effect Effects 0.000 description 2
- 230000036541 health Effects 0.000 description 2
- 230000010365 information processing Effects 0.000 description 2
- 238000005192 partition Methods 0.000 description 2
- 241000270322 Lepidosauria Species 0.000 description 1
- 206010028916 Neologism Diseases 0.000 description 1
- 230000032683 aging Effects 0.000 description 1
- 230000002146 bilateral effect Effects 0.000 description 1
- 235000012813 breadcrumbs Nutrition 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 230000002045 lasting effect Effects 0.000 description 1
- 230000013011 mating Effects 0.000 description 1
- 238000011430 maximum method Methods 0.000 description 1
- 239000000203 mixture Substances 0.000 description 1
- 238000004321 preservation Methods 0.000 description 1
- 230000001846 repelling effect Effects 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
- 230000009897 systematic effect Effects 0.000 description 1
- 230000007704 transition Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/30—Information retrieval; Database structures therefor; File system structures therefor of unstructured textual data
- G06F16/33—Querying
- G06F16/3331—Query processing
- G06F16/334—Query execution
- G06F16/3344—Query execution using natural language analysis
Abstract
The embodiment of the invention provides a method and a device for segmentation on the basis of webpage content classification. The method comprises the following steps of: extracting the text information of webpage contents in search resources; dividing the classes of the text information according to the classes of the webpage contents; segmenting the text information according to segmentation dictionaries corresponding to the classes of the text information. According to the embodiment of the invention, the classes of the text information of the webpage contents in the search resources are divided, and the text information is segmented on the basis of the segmentation dictionaries corresponding to the classes, so as to adapt to different classes of language characteristics better, meanwhile, the segmentation accuracy for different classes is also improved, and the optimal processing for local segmentation is realized; moreover, the improvement of the accuracy of segmentation is close to the intention of a user and improve the user experience, and then reduce the operations of re-input, search and the like of the user, and improve the simplicity of operation, meanwhile, the response of equipment on the operation of the user is reduced, and the consumption of the system resources of the equipment is reduced.
Description
Technical field
The technical field that the present invention relates to search, particularly relates to a kind of method and a kind of device that carries out word segmentation processing based on web page contents classification that carries out word segmentation processing based on web page contents classification.
Background technology
Along with the high speed development of internet, the diversification of network application trend, online quantity of information sharply increases.
Under various occasions, user often need to input key message and carry out obtaining of related information.For example, in search engine, input keyword search info web, in forum, input keyword search model etc.
Participle is the basis of carrying out information processing, information retrieval, and all information processing, information retrieval work are all carried out after participle.So the mistake of participle can be added in follow-up processing procedure, and be difficult to be eliminated.So the pursuit of the accuracy rate to participle is a lasting process Just because of this, simultaneously due to the intrinsic characteristic of Chinese language: do not have not have between definition, word and the word of clear and definite word these factors such as separator, neologisms, proper noun continue to bring out cause participle be difficult to accomplish 100% accurately.
The main method that current Words partition system uses is the participle based on statistics.From in form, word is the combination of stable word, and therefore, in context, the number of times that adjacent word occurs is simultaneously more, just more likely forms a word.Therefore the frequency of word and the adjacent co-occurrence of word or probability can reflect into the confidence level of word preferably.Can add up the frequency of the combination of each word of adjacent co-occurrence in language material, calculate their information that appears alternatively.The information that appears alternatively of two words of definition, the adjacent co-occurrence probabilities of calculating two Chinese character X, Y.The information of appearing alternatively has embodied the tightness degree of marriage relation between Chinese character.When tightness degree is during higher than some threshold values, just can think that this word group may form a word.This method only need be added up the word group frequency in language material, but this method also has certain limitation, can often extract some co-occurrence frequency out high but be not the everyday character group of word, for example " this ", " one of ", " having ", " I ", " many " etc., and the accuracy of identification to everyday words is poor, space-time expense is large.
On the one hand, the erroneous results of participle causes the related information that the later stage obtains to have very big difference with expection originally, and it is very poor that user experiences, and wasted apparatus system resource.On the other hand, user need to obtain related information, can again input key message search, equipment will again carry out search, contrast, screening of magnanimity information etc. and obtain the information relevant to searched key word, not only user's operation is more loaded down with trivial details, expend user's time, and will greatly increase the burden of equipment, expend more device resource.
Summary of the invention
In view of the above problems, the present invention has been proposed to a kind of a kind of method and corresponding a kind of device that carries out word segmentation processing based on web page contents classification that carries out word segmentation processing based on web page contents classification that overcomes the problems referred to above or address the above problem is at least in part provided.
According to one aspect of the present invention, a kind of method of carrying out word segmentation processing based on web page contents classification is provided, comprising:
Extract the text message of web page contents in searching resource;
According to classification under text message described in described web page contents category division;
According to dictionary for word segmentation corresponding to classification under described text message, described text message is carried out to word segmentation processing.
Alternatively, also comprise:
For described classification, the first participle that adopts word segmentation processing to obtain is set up inverted index.
Alternatively, described for described classification, the step that the first participle that adopts word segmentation processing to obtain is set up inverted index comprises:
For described classification, the appearance position of the first participle corresponding to described classification that record occurs;
The described first participle and corresponding appearance position thereof are recorded in inverted index.
Alternatively, described dictionary for word segmentation generates in the following manner:
Obtain the first training document;
Divide the affiliated classification of described the first training document;
The described first training document corresponding to described classification carries out word segmentation processing, obtains the second participle;
Add up word frequency number and the first co-occurrence rate of described the second participle corresponding to described classification;
Adopt described the second participle and the first co-occurrence rate thereof to generate dictionary for word segmentation corresponding to described classification.
Alternatively, described the first co-occurrence rate comprises the ratio of the first word frequency number and the second word frequency number;
Wherein, described the first word frequency number comprises that current the second participle appears at target the second participle word frequency number afterwards; Described target the second participle comprises one or more the second participles that appear at before current environment participle;
Described the second word frequency number comprises the total word frequency number of described target the second participle.
Alternatively, described dictionary for word segmentation upgrades in the following manner:
Obtain the second training document;
Divide the affiliated classification of described the second training document;
According to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing, obtain the 3rd participle
Add up word frequency number and the second co-occurrence rate of described the 3rd participle corresponding to described classification;
Adopt described the 3rd participle and the second co-occurrence rate thereof to upgrade dictionary for word segmentation corresponding to described classification.
Alternatively, described the second co-occurrence rate comprises the ratio of the 3rd word frequency number and the 4th word frequency number;
Wherein, described the 3rd word frequency number comprises that current the 3rd participle appears at target the 3rd participle word frequency number afterwards; Described target the 3rd participle comprises one or more the 3rd participles that appear at before current environment participle;
Described the 4th word frequency number comprises the total word frequency number of described target the 3rd participle.
Alternatively, described according to dictionary for word segmentation corresponding to classification under described text message, the step of described text message being carried out to word segmentation processing comprises:
According to dictionary for word segmentation corresponding to classification under described text message, and universaling dictionary, described text message is carried out to word segmentation processing;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
Alternatively, described according to dictionary for word segmentation corresponding to the classification under described text message, the step of described text message being carried out to word segmentation processing comprises:
When the classification under described text message is while being multiple, according to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing respectively;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
According to a further aspect in the invention, provide a kind of device that carries out word segmentation processing based on web page contents classification, having comprised:
Extraction module, is suitable for extracting the text message of web page contents in searching resource;
Divide module, be suitable for according to classification under text message described in described web page contents category division;
Word-dividing mode, is suitable for, according to dictionary for word segmentation corresponding to classification under described text message, described text message being carried out to word segmentation processing.
Alternatively, also comprise:
Set up module, be suitable for for described classification, the first participle that adopts word segmentation processing to obtain is set up inverted index.
Alternatively, the described module of setting up is also suitable for:
For described classification, the appearance position of the first participle corresponding to described classification that record occurs;
The described first participle and corresponding appearance position thereof are recorded in inverted index.
Alternatively, described dictionary for word segmentation generates in the following manner:
Obtain the first training document;
Divide the affiliated classification of described the first training document;
The described first training document corresponding to described classification carries out word segmentation processing, obtains the second participle;
Add up word frequency number and the first co-occurrence rate of described the second participle corresponding to described classification;
Adopt described the second participle and the first co-occurrence rate thereof to generate dictionary for word segmentation corresponding to described classification.
Alternatively, described the first co-occurrence rate comprises the ratio of the first word frequency number and the second word frequency number;
Wherein, described the first word frequency number comprises that current the second participle appears at target the second participle word frequency number afterwards; Described target the second participle comprises one or more the second participles that appear at before current environment participle;
Described the second word frequency number comprises the total word frequency number of described target the second participle.
Alternatively, described dictionary for word segmentation upgrades in the following manner:
Obtain the second training document;
Divide the affiliated classification of described the second training document;
According to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing, obtain the 3rd participle
Add up word frequency number and the second co-occurrence rate of described the 3rd participle corresponding to described classification;
Adopt described the 3rd participle and the second co-occurrence rate thereof to upgrade dictionary for word segmentation corresponding to described classification.
Alternatively, described the second co-occurrence rate comprises the ratio of the 3rd word frequency number and the 4th word frequency number;
Wherein, described the 3rd word frequency number comprises that current the 3rd participle appears at target the 3rd participle word frequency number afterwards; Described target the 3rd participle comprises one or more the 3rd participles that appear at before current environment participle;
Described the 4th word frequency number comprises the total word frequency number of described target the 3rd participle.
Alternatively, described word-dividing mode is also suitable for:
According to dictionary for word segmentation corresponding to classification under described text message, and universaling dictionary, described text message is carried out to word segmentation processing;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
Alternatively, described word-dividing mode is also suitable for:
When the classification under described text message is while being multiple, according to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing respectively;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
The embodiment of the present invention is divided classification to the text message of web page contents in searching resource, based on such other dictionary for word segmentation, this paper information is carried out to participle, adapt to better different classes of characteristic of speech sounds, also improved different classes of participle accuracy, realized the optimal processing of local participle simultaneously; And, the raising of participle accuracy, the intention of being more close to the users, promote user's experience, then reduced the operations such as user re-enters, search, improved the simplicity of operation, also reduced the response of equipment to user's operation, reduced expending of apparatus system resource simultaneously.
The embodiment of the present invention is divided classification to the text message of web page contents in searching resource, based on such other dictionary for word segmentation, this paper information is carried out to participle, the first participle that adopts again word segmentation processing to obtain is set up inverted index, unicity and the one-sidedness of the inverted index based on overall text message are avoided, promote the accuracy rate of inverted index in each classification, then promote the index operational efficiency of inverted index, reduced the time of index; And, in searching resource the text message of webpage comprise newly, strange, the various text messages that meet this classification characteristic of speech sounds of top grade, utilize other people and the collective wisdom in searching resource, collected, made up self definition, own artificial deficiency, greatly reduced artificial operation cost.
Above-mentioned explanation is only the general introduction of technical solution of the present invention, in order to better understand technological means of the present invention, and can be implemented according to the content of instructions, and for above and other objects of the present invention, feature and advantage can be become apparent, below especially exemplified by the specific embodiment of the present invention.
Brief description of the drawings
By reading below detailed description of the preferred embodiment, various other advantage and benefits will become cheer and bright for those of ordinary skill in the art.Accompanying drawing is only for the object of preferred implementation is shown, and do not think limitation of the present invention.And in whole accompanying drawing, represent identical parts by identical reference symbol.In the accompanying drawings:
Fig. 1 shows a kind of according to an embodiment of the invention flow chart of steps of carrying out the embodiment of the method for word segmentation processing based on web page contents classification; And
Fig. 2 shows a kind of according to an embodiment of the invention structured flowchart that carries out the device embodiment of word segmentation processing based on web page contents classification.
Embodiment
Exemplary embodiment of the present disclosure is described below with reference to accompanying drawings in more detail.Although shown exemplary embodiment of the present disclosure in accompanying drawing, but should be appreciated that and can realize the disclosure and the embodiment that should do not set forth limits here with various forms.On the contrary, it is in order more thoroughly to understand the disclosure that these embodiment are provided, and can be by the those skilled in the art that conveys to complete the scope of the present disclosure.
With reference to Fig. 1, show a kind of according to an embodiment of the invention flow chart of steps of carrying out the embodiment of the method for word segmentation processing based on web page contents classification, can comprise the steps:
Step 101, the text message of web page contents in extraction searching resource;
The treatment scheme of search engine generally can be divided into two parts, and Part I is front end subscriber request, and Part II is that rear end makes data.
One, front end subscriber request processing procedure can comprise:
1. user entered keyword;
2. query word analysis, search engine is to key word participle;
3. retrieval, according to word segmentation result, from the index of prior making, finds out relevant collections of web pages;
4. sequence, to candidate's collections of web pages, sorts according to content relevance, the dimension such as ageing;
5. represent: the webpage after sequence is represented.
Two, making data procedures in rear end can comprise:
1. webpage captures, and reptile, by the linking relationship between webpage, captures the webpage of internet and preserves;
2. compilation of index, analyzes the webpage that captures preservation, to web page title and page text participle, makes inverted index according to word segmentation result, retrieves for front end.
The webpage of crawler capturing can be kept at and in web database, form a large amount of searching resources, and can comprise a large amount of text messages in web page contents.In embodiments of the present invention, can from web database, extract the text message of web page contents in searching resource.
Step 102, according to classification under text message described in described web page contents category division;
In a kind of situation, can obtain webpage classification based on the network address of webpage, then according to classification under webpage category division text message.For example, the domain name that the webpage network address in animation field generally identifies with " comic ", the domain name that the webpage network address of sports field generally identifies with " sports ", as comic.XXX.com, sports.XXX.com, detect the domain name of " comic " or " sports " mark in webpage network address time, can identify this webpage classification is animation field or sports field, then text message can be divided into animation field or sports field.
In another situation, the webpage of access with label (tag) information, can obtain webpage classification based on label information conventionally, then according to classification under webpage category division text message.For example, certain webpage is with label informations such as video, film, comedy movies, and can identify this webpage classification is film field, then text message is divided into film field.
In another situation, can the middle specific word of analyzing web page content title (topic) know webpage classification, be sports field such as containing the specific word such as basketball, football, NBA or world cup in title known.
In another situation, in can navigate at webpage (such as crumbs navigation Breadcrumb Trail), know webpage classification, then according to classification under webpage category division text message.Crumbs navigation occurs in page top level conventionally, generally can be positioned at the below of title or top margin.Crumbs navigation offer user return before the link (these links are also the paths that can arrive current page) of any one page, the parent page of this page normally in level framework.Crumbs navigation offers user and dates back to a paths of website homepage or portal page, normally occurs with greater-than sign (>), and also having some designs is other symbols (as >>).For example, " homepage > classification page > subclassificatio page " or " homepage >> classification page >> subclassificatio page ".From crumbs navigation, classification can be known the self-defined classification of webpage, the crumbs navigation of for example webpage is " super in the > of XX portal website physical culture > Chinese football > ", its corresponding Web page classifying be physical culture, Chinese football, in super, choose and the classification of mating in embodiment of the present invention practical application.
It should be noted that, the embodiment of the present invention can be according to actual needs, divide the grade of classification, for example, for sports field, except dividing the classification for whole sports field, also can divide basketball, the classification of the next stage such as football, can also divide NBA(National Basketball Association, national basketball association), CBA(Chinese Basketball Association, China Basketball Association), Division A League Matches of Spanish Football (Spain's soccer league match), in the more classification of next stage such as super (Chinese football super league), the embodiment of the present invention is not limited this.
And along with the grade classification precision of classification is higher, the overlapping probability of its collected text message is also just lower, otherwise accurately just higher, in the embodiment of the present invention, the degree of accuracy of participle also can be higher.
Step 103, according to dictionary for word segmentation corresponding to classification under described text message, carries out word segmentation processing to described text message.
In specific implementation, each classification can be to there being a specific dictionary for word segmentation, text message is conformed with to the participle of this classification language feature.
In a preferred embodiment of the present invention, described dictionary for word segmentation can generate in the following manner:
Sub-step S11, obtains the first training document;
In the invention process, the first training document can be the text message of webpage in searching resource.
Sub-step S12, divides the affiliated classification of described the first training document;
In specific implementation, in the time that the first training document can be the text message of webpage in searching resource, can train classification under document according to this webpage category division first.
It should be noted that, because sub-step S11, sub-step S12 are substantially similar to the application of step 101, step 102, so that describes is fairly simple, relevant part is referring to the part explanation of step 101, step 102, and the embodiment of the present invention is not described in detail at this.
Sub-step S13, the described first training document corresponding to described classification carries out word segmentation processing, obtains the second participle;
The application embodiment of the present invention, can adopt general dictionary for word segmentation first to train document to carry out word segmentation processing.General dictionary for word segmentation can be the dictionary for word segmentation of general field, does not have technical term in specific specific field to divide, and such as Radix Angelicae Sinensis, the radix aconiti agrestis etc. of field of Chinese herbal, it mainly comprises general, the frequency of occurrences entry higher than predetermined threshold value.Particularly, in general dictionary for word segmentation, can comprise general term and definite meaning word.General term can comprise the verb of adjective, conjunction and some general meanings, for example glad, still, participation etc.Determine that meaning word can be the word that can express definite meaning scope, normally some nouns and verb.
Sub-step S14, word frequency number and the first co-occurrence rate of adding up described the second participle corresponding to described classification;
In a kind of preferred exemplary of the embodiment of the present invention, can be based on the second participle training N-Gram model.
In this example, the probability that the first co-occurrence rate can occur for two or more second participles simultaneously.Particularly, described the first co-occurrence rate can comprise the ratio of the first word frequency number and the second word frequency number;
Wherein, described the first word frequency number comprises that current the second participle appears at target the second participle word frequency number afterwards; Described target the second participle comprises one or more the second participles that appear at before current environment participle;
Described the second word frequency number comprises the total word frequency number of described target the second participle.
N-Gram model is a kind of language model conventional in large vocabulary continuous speech recognition, and based on Markov hypothesis, i.e. the appearance of a word only depends on one or several limited words that it occurs above.For a sentence T, can suppose that T is by word sequence W
1, W
2, W
3..., W
ncomposition, this sentence T is by W so
1, W
2, W
3..., W
nthe connection probability connecting to form is P (T)=P (W
1w
2w
3w
n)=P (W
1) P (W
2| W
1) P (W
3| W
1w
2) ... P (W
n| W
1w
2w
n-1).
If the appearance of a word only depends on the word that it occurs above, be referred to as bigram.Be P (T)=P (W
1w
2w
3w
n)=P (W
1) P (W
2|w
1) P (W
3| W
1w
2) ... P (W
n| W
1w
2w
n-1) ≈ P (W
1) P (W
2| W
1) P (W
3| W
2) ... P (W
n| W
n-1).
If the appearance of a word only depends on two words that it occurs above, be referred to as trigram.In the practical application of N-Gram model taking bigram and trigram as main, and less higher than the N-Gram model application of quaternary, because the N-Gram model of training quaternary needs huger language material, and Sparse is serious, time complexity is high, and it is few that precision but improves.
Describe as an example of text message " I want to eat Chinese food lunch " example below:
The second text message in certain environmental form " I want eat Chinese food lunch " is carried out to participle, obtain the second participle " I ", " want ", " to ", " eat ", " Chinese ", " food ", " lunch ", shown in this second participle and word frequency table 1 and table 2.
Total word frequency of table 1 the second participle is counted statistical form
| The second participle | Total word frequency number |
| I | 3437 |
| want | 1215 |
| to | 3256 |
| eat | 938 |
| Chinese | 213 |
| food | 1506 |
| lunch | 459 |
The word frequency that current the second participle of table 2 appears at before target the second participle is counted statistical form
| I | want | to | eat | Chinese | food | lunch | |
| I | 8 | 1087 | 0 | 13 | 0 | 0 | 0 |
| want | 3 | 0 | 786 | 0 | 6 | 8 | 6 |
| to | 3 | 0 | 10 | 860 | 3 | 0 | 12 |
| eat | 0 | 0 | 2 | 0 | 19 | 2 | 52 |
| Chinese | 2 | 0 | 0 | 0 | 0 | 120 | 1 |
| food | 19 | 0 | 17 | 0 | 0 | 0 | 0 |
| lunch | 4 | 0 | 0 | 0 | 0 | 1 | 0 |
For example, 1087 in the second row the 3rd row represent that it is 1087 that current the second participle " want " appears at target the second participle " I " word frequency number below.
Sub-step S15, adopts described the second participle and the first co-occurrence rate thereof to generate dictionary for word segmentation corresponding to described classification.
In the participle process based on dictionary for word segmentation, the inquiry mode of generally using can comprise: Hash inquiry, TRIE tree (claiming again word lookup tree or key tree) inquiry, two points of inquiries and sequential query etc.And in actual participle process, can use multiple inquiry mode; for example, in the dictionary for word segmentation of Hash mechanism; can adopt Hash inquiry and two points of modes that inquiry combines; TRIE tree query and two points of modes that inquiry combines in the dictionary for word segmentation of TRIE mechanism, are adopted; multiple inquiry mode is combined and inquired about, can improve search efficiency.
In embodiments of the present invention, can inquire about according to Hash, the inquiry mechanism of one or more formation in TRIE tree query, two points of inquiries and sequential query etc. inquiry mode, generate dictionary for word segmentation, with realize that Hash inquiry, TRIE tree query, two points are inquired about and sequential query etc. inquiry mode in the inquiry mechanism of one or more formation.
In a preferred embodiment of the present invention, described dictionary for word segmentation can upgrade in the following manner:
Sub-step S21, obtains the second training document;
In the invention process, the second training document can be the text message of webpage in searching resource.
Sub-step S22, divides the affiliated classification of described the second training document;
In specific implementation, in the time that the second training document can be the text message of webpage in searching resource, can train classification under document according to this webpage category division second.
It should be noted that, because sub-step S21, sub-step S22 are substantially similar to the application of step 101, step 102, so that describes is fairly simple, relevant part is referring to the part explanation of step 101, step 102, and the embodiment of the present invention is not described in detail at this.
Sub-step S23, according to dictionary for word segmentation corresponding to described classification, carries out word segmentation processing to described text message, obtains the 3rd participle;
The application embodiment of the present invention, can adopt affiliated dictionary for word segmentation second corresponding to classification of the second training document to train document to carry out word segmentation processing.To treating the character string of participle in given the second training document, according to certain definite principle, for example Forward Maximum Method method (MM), oppositely maximum matching method (RMM) or bilateral scanning method etc., cut the substring of character string, if certain entry matches in this substring and dictionary for word segmentation, think that this substring is the 3rd participle, insert cutting mark, continue to cut apart remaining part, know that remainder is empty; Otherwise this substring is not the 3rd participle, and the substring that again cuts character string mates next time.
Sub-step S24, word frequency number and the second co-occurrence rate of adding up described the 3rd participle corresponding to described classification;
In a kind of preferred exemplary of the embodiment of the present invention, can be based on the 3rd participle training N-Gram model.
In this example, the probability that the second co-occurrence rate can occur for two or more the 3rd participles simultaneously.Particularly, described the second co-occurrence rate can comprise the ratio of the 3rd word frequency number and the 4th word frequency number;
Wherein, described the 3rd word frequency number comprises that current the 3rd participle appears at target the 3rd participle word frequency number afterwards; Described target the 3rd participle comprises one or more the 3rd participles that appear at before current environment participle;
Described the 4th word frequency number comprises the total word frequency number of described target the 3rd participle.
Sub-step S25, adopts described the 3rd participle and the second co-occurrence rate thereof to upgrade dictionary for word segmentation corresponding to described classification.
In embodiments of the present invention, can inquire about according to Hash, the inquiry mechanism of one or more formation in TRIE tree query, two points of inquiries and sequential query etc. inquiry mode, upgrade dictionary for word segmentation.
In a preferred embodiment of the present invention, can directly adopt affiliated dictionary for word segmentation corresponding to classification of text information, text message is carried out to word segmentation processing.
In a preferred embodiment of the present invention, step 103 can comprise following sub-step:
Sub-step S31, according to dictionary for word segmentation corresponding to classification under described text message, and universaling dictionary, described text message is carried out to word segmentation processing;
Sub-step S32, by the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
In embodiments of the present invention, can use affiliated dictionary for word segmentation corresponding to classification of text message, and universaling dictionary (general dictionary for word segmentation) carries out word segmentation processing to text message simultaneously.
For example, for text message " people participates in Radix Angelicae Sinensis ", owing to must be the word frequency that the word frequency of " people ", " participation " is greater than " ginseng ", "AND" in overall language material, so based on universaling dictionary participle, this text message can be cut into " people ", " participation ", " Radix Angelicae Sinensis ", but obviously such cutting is wrong in fact." people participates in Radix Angelicae Sinensis " this text message often appears in the document of health care class, if the dictionary for word segmentation of the health care class based under it carries out participle, this information can be cut into " ginseng ", "AND", " Radix Angelicae Sinensis ".Relatively, after word frequency number, can find that the relative term frequency of " ginseng " can be significantly higher than the relative term frequency in overall language material, therefore, finally can choose the word segmentation result of " ginseng ", "AND", " Radix Angelicae Sinensis " conduct " people participates in Radix Angelicae Sinensis ".
In another preferred embodiment of the present invention, step 103 can comprise following sub-step:
Sub-step S41, when the classification under described text message is while being multiple, according to dictionary for word segmentation corresponding to described classification, carries out word segmentation processing to described text message respectively;
Sub-step S42, by the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
In embodiments of the present invention, text message can be divided into multiple classifications, belongs to crossing domain.For example, can be mechanical field about the classification of the text message of aircraft, can be also aviation field.
In the situation of this kind of crossing domain, can carry out word segmentation processing based on dictionary for word segmentation corresponding to affiliated classification to this paper information respectively, finally using word frequency the highest as word segmentation result.
The main method that current Words partition system uses is the participle based on statistics, is exactly briefly mainly to rely on the information such as word frequency, transition probability between candidate word in the cut-off that determines word.Since be statistics, must be to meet great majority to sacrifice fractions, what that is to say pursuit is the optimum in overall statistical significance, but not the optimum of each part makes in local word segmentation processing accuracy very low.
The embodiment of the present invention is divided classification to the text message of web page contents in searching resource, based on such other dictionary for word segmentation, this paper information is carried out to participle, adapt to better different classes of characteristic of speech sounds, also improved different classes of participle accuracy, realized the optimal processing of local participle simultaneously; And, the raising of participle accuracy, the intention of being more close to the users, promote user's experience, then reduced the operations such as user re-enters, search, improved the simplicity of operation, also reduced the response of equipment to user's operation, reduced expending of apparatus system resource simultaneously.
In a preferred embodiment of the present invention, can also comprise the steps:
Step 104, for described classification, the first participle that adopts word segmentation processing to obtain is set up inverted index.
Inverted index comes from practical application to be needed to search record according to the value of attribute.Each in this concordance list all comprises a property value and has the address of each record of this property value.Because not being determines property value by recording, but determined the position of recording by property value, thereby be called inverted index (inverted index).Be called inverted index file with the file of inverted index, be called for short inverted file (inverted file).
Inverted file (inverted index), index object is such as, word in document or collection of document (webpage) etc., being used for storing the memory location of these words in a document or one group of document, is a kind of conventional Indexing Mechanism to document or collection of document.
In a preferred embodiment of the present invention, step 104 can comprise following sub-step:
Sub-step S51, for described classification, the appearance position of the first participle corresponding to described classification that record occurs;
Sub-step S52, is recorded in the described first participle and corresponding appearance position thereof in inverted index.
In specific implementation, the appearance position of the first participle can comprise the webpage of appearance, or, the webpage of appearance and the position in this webpage thereof.
Taking English as example, be below the text message in webpage that will be indexed:
T1="it is what it is"；
T2="what is it"；
T3="it is a banana"；
Be below inverted index:
"a": {(2,2)}
"banana":{(2,3)}
"is": {(0,1),(0,4),(1,1),(2,1)}
"it": {(0,0),(0,3),(1,2),(2,0)}
"what":{(0,2),(1,0)}
Wherein, " banana ": { (2,3) } be " banana " in the text message of the 3rd webpage (T3), and be the 4th word (address is 3) in the position of the 3rd webpage.
The embodiment of the present invention is divided classification to the text message of web page contents in searching resource, based on such other dictionary for word segmentation, this paper information is carried out to participle, the first participle that adopts again word segmentation processing to obtain is set up inverted index, unicity and the one-sidedness of the inverted index based on overall text message are avoided, promote the accuracy rate of inverted index in each classification, then promote the index operational efficiency of inverted index, reduced the time of index; And, in searching resource the text message of webpage comprise newly, strange, the various text messages that meet this classification characteristic of speech sounds of top grade, utilize other people and the collective wisdom in searching resource, collected, made up self definition, own artificial deficiency, greatly reduced artificial operation cost.
For embodiment of the method, for simple description, therefore it is all expressed as to a series of combination of actions, but those skilled in the art should know, the embodiment of the present invention is not subject to the restriction of described sequence of movement, because according to the embodiment of the present invention, some step can adopt other orders or carry out simultaneously.Secondly, those skilled in the art also should know, the embodiment described in instructions all belongs to preferred embodiment, and related action might not be that the embodiment of the present invention is necessary.
With reference to Fig. 2, show and show according to an embodiment of the invention a kind of according to an embodiment of the invention structured flowchart that carries out the device embodiment of word segmentation processing based on web page contents classification, can comprise as lower module:
Extraction module 201, is suitable for extracting the text message of web page contents in searching resource;
Divide module 202, be suitable for according to classification under text message described in described web page contents category division;
Word-dividing mode 203, is suitable for, according to dictionary for word segmentation corresponding to classification under described text message, described text message being carried out to word segmentation processing.
In a preferred embodiment of the present invention, can also comprise as lower module:
Set up module, be suitable for for described classification, the first participle that adopts word segmentation processing to obtain is set up inverted index.
In a preferred embodiment of the present invention, the described module of setting up can also be suitable for:
For described classification, the appearance position of the first participle corresponding to described classification that record occurs;
The described first participle and corresponding appearance position thereof are recorded in inverted index.
In a preferred embodiment of the present invention, described dictionary for word segmentation can generate in the following manner:
Obtain the first training document;
Divide the affiliated classification of described the first training document;
The described first training document corresponding to described classification carries out word segmentation processing, obtains the second participle;
Add up word frequency number and the first co-occurrence rate of described the second participle corresponding to described classification;
Adopt described the second participle and the first co-occurrence rate thereof to generate dictionary for word segmentation corresponding to described classification.
In a preferred embodiment of the present invention, described the first co-occurrence rate can comprise the ratio of the first word frequency number and the second word frequency number;
Wherein, described the first word frequency number comprises that current the second participle appears at target the second participle word frequency number afterwards; Described target the second participle comprises one or more the second participles that appear at before current environment participle;
Described the second word frequency number comprises the total word frequency number of described target the second participle.
In a preferred embodiment of the present invention, described dictionary for word segmentation can upgrade in the following manner:
Obtain the second training document;
Divide the affiliated classification of described the second training document;
According to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing, obtain the 3rd participle
Add up word frequency number and the second co-occurrence rate of described the 3rd participle corresponding to described classification;
Adopt described the 3rd participle and the second co-occurrence rate thereof to upgrade dictionary for word segmentation corresponding to described classification.
In a preferred embodiment of the present invention, described the second co-occurrence rate can comprise the ratio of the 3rd word frequency number and the 4th word frequency number;
Wherein, described the 3rd word frequency number comprises that current the 3rd participle appears at target the 3rd participle word frequency number afterwards; Described target the 3rd participle comprises one or more the 3rd participles that appear at before current environment participle;
Described the 4th word frequency number comprises the total word frequency number of described target the 3rd participle.
In a preferred embodiment of the present invention, described word-dividing mode 203 can also be suitable for:
According to dictionary for word segmentation corresponding to classification under described text message, and universaling dictionary, described text message is carried out to word segmentation processing;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
In a preferred embodiment of the present invention, described word-dividing mode 203 can also be suitable for:
When the classification under described text message is while being multiple, according to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing respectively;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
For device embodiment, because it is substantially similar to embodiment of the method, so description is fairly simple, relevant part is referring to the part explanation of embodiment of the method.
The algorithm providing at this is intrinsic not relevant to any certain computer, virtual system or miscellaneous equipment with demonstration.Various general-purpose systems also can with based on using together with this teaching.According to description above, it is apparent constructing the desired structure of this type systematic.In addition, the present invention is not also for any certain programmed language.It should be understood that and can utilize various programming languages to realize content of the present invention described here, and the description of above language-specific being done is in order to disclose preferred forms of the present invention.
In the instructions that provided herein, a large amount of details are described.But, can understand, embodiments of the invention can be put into practice in the situation that there is no these details.In some instances, be not shown specifically known method, structure and technology, so that not fuzzy understanding of this description.
Similarly, be to be understood that, in order to simplify the disclosure and to help to understand one or more in each inventive aspect, in the above in the description of exemplary embodiment of the present invention, each feature of the present invention is grouped together into single embodiment, figure or sometimes in its description.But, the method for the disclosure should be construed to the following intention of reflection: the present invention for required protection requires than the more feature of feature of clearly recording in each claim.Or rather, as reflected in claims below, inventive aspect is to be less than all features of disclosed single embodiment above.Therefore, claims of following embodiment are incorporated to this embodiment thus clearly, and wherein each claim itself is as independent embodiment of the present invention.
Those skilled in the art are appreciated that and can the module in the equipment in embodiment are adaptively changed and they are arranged in one or more equipment different from this embodiment.Module in embodiment or unit or assembly can be combined into a module or unit or assembly, and can put them in addition multiple submodules or subelement or sub-component.At least some in such feature and/or process or unit are mutually repelling, and can adopt any combination to combine all processes or the unit of disclosed all features in this instructions (comprising claim, summary and the accompanying drawing followed) and disclosed any method like this or equipment.Unless clearly statement in addition, in this instructions (comprising claim, summary and the accompanying drawing followed) disclosed each feature can be by providing identical, be equal to or the alternative features of similar object replaces.
In addition, those skilled in the art can understand, although embodiment more described herein comprise some feature instead of further feature included in other embodiment, the combination of the feature of different embodiment means within scope of the present invention and forms different embodiment.For example, in the following claims, the one of any of embodiment required for protection can be used with array mode arbitrarily.
All parts embodiment of the present invention can realize with hardware, or realizes with the software module of moving on one or more processor, or realizes with their combination.It will be understood by those of skill in the art that can use in practice microprocessor or digital signal processor (DSP) to realize carries out the some or all functions of the some or all parts in word segmentation processing equipment according to the embodiment of the present invention based on web page contents classification.The present invention can also be embodied as part or all equipment or the device program (for example, computer program and computer program) for carrying out method as described herein.Realizing program of the present invention and can be stored on computer-readable medium like this, or can there is the form of one or more signal.Such signal can be downloaded and obtain from internet website, or provides on carrier signal, or provides with any other form.
It should be noted above-described embodiment the present invention will be described instead of limit the invention, and those skilled in the art can design alternative embodiment in the case of not departing from the scope of claims.In the claims, any reference symbol between bracket should be configured to limitations on claims.Word " comprises " not to be got rid of existence and is not listed as element or step in the claims.Being positioned at word " " before element or " one " does not get rid of and has multiple such elements.The present invention can be by means of including the hardware of some different elements and realizing by means of the computing machine of suitably programming.In the unit claim of having enumerated some devices, several in these devices can be to carry out imbody by same hardware branch.The use of word first, second and C grade does not represent any order.Can be title by these word explanations.
Claims (10)
1. a method of carrying out word segmentation processing based on web page contents classification, comprising:
Extract the text message of web page contents in searching resource;
According to classification under text message described in described web page contents category division;
According to dictionary for word segmentation corresponding to classification under described text message, described text message is carried out to word segmentation processing.
2. the method for claim 1, is characterized in that, also comprises:
For described classification, the first participle that adopts word segmentation processing to obtain is set up inverted index.
3. the method as described in claim 1 to 2 any one, is characterized in that, described for described classification, and the step that the first participle that adopts word segmentation processing to obtain is set up inverted index comprises:
For described classification, the appearance position of the first participle corresponding to described classification that record occurs;
The described first participle and corresponding appearance position thereof are recorded in inverted index.
4. the method as described in claims 1 to 3 any one, is characterized in that, described dictionary for word segmentation generates in the following manner:
Obtain the first training document;
Divide the affiliated classification of described the first training document;
The described first training document corresponding to described classification carries out word segmentation processing, obtains the second participle;
Add up word frequency number and the first co-occurrence rate of described the second participle corresponding to described classification;
Adopt described the second participle and the first co-occurrence rate thereof to generate dictionary for word segmentation corresponding to described classification.
5. the method as described in claim 1 to 4 any one, is characterized in that, described the first co-occurrence rate comprises the ratio of the first word frequency number and the second word frequency number;
Wherein, described the first word frequency number comprises that current the second participle appears at target the second participle word frequency number afterwards; Described target the second participle comprises one or more the second participles that appear at before current environment participle;
Described the second word frequency number comprises the total word frequency number of described target the second participle.
6. the method as described in claim 1 to 5 any one, is characterized in that, described dictionary for word segmentation upgrades in the following manner:
Obtain the second training document;
Divide the affiliated classification of described the second training document;
According to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing, obtain the 3rd participle
Add up word frequency number and the second co-occurrence rate of described the 3rd participle corresponding to described classification;
Adopt described the 3rd participle and the second co-occurrence rate thereof to upgrade dictionary for word segmentation corresponding to described classification.
7. the method as described in claim 1 to 6 any one, is characterized in that, described the second co-occurrence rate comprises the ratio of the 3rd word frequency number and the 4th word frequency number;
Wherein, described the 3rd word frequency number comprises that current the 3rd participle appears at target the 3rd participle word frequency number afterwards; Described target the 3rd participle comprises one or more the 3rd participles that appear at before current environment participle;
Described the 4th word frequency number comprises the total word frequency number of described target the 3rd participle.
8. the method as described in claim 1 to 7 any one, is characterized in that, described according to dictionary for word segmentation corresponding to classification under described text message, the step of described text message being carried out to word segmentation processing comprises:
According to dictionary for word segmentation corresponding to classification under described text message, and universaling dictionary, described text message is carried out to word segmentation processing;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
9. the method as described in claim 1 to 8 any one, is characterized in that, described according to dictionary for word segmentation corresponding to the classification under described text message, the step of described text message being carried out to word segmentation processing comprises:
When the classification under described text message is while being multiple, according to dictionary for word segmentation corresponding to described classification, described text message is carried out to word segmentation processing respectively;
By the highest participle of word frequency number obtaining after word segmentation processing, the first participle obtaining as word segmentation processing.
10. a device that carries out word segmentation processing based on web page contents classification, comprising:
Extraction module, is suitable for extracting the text message of web page contents in searching resource;
Divide module, be suitable for according to classification under text message described in described web page contents category division;
Word-dividing mode, is suitable for, according to dictionary for word segmentation corresponding to classification under described text message, described text message being carried out to word segmentation processing.
Priority Applications (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410126465.1A CN104008126A (en) | 2014-03-31 | 2014-03-31 | Method and device for segmentation on basis of webpage content classification |
| PCT/CN2014/093396 WO2015149533A1 (en) | 2014-03-31 | 2014-12-09 | Method and device for word segmentation processing on basis of webpage content classification |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410126465.1A CN104008126A (en) | 2014-03-31 | 2014-03-31 | Method and device for segmentation on basis of webpage content classification |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN104008126A true CN104008126A (en) | 2014-08-27 |
Family
ID=51368783
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410126465.1A Pending CN104008126A (en) | 2014-03-31 | 2014-03-31 | Method and device for segmentation on basis of webpage content classification |
Country Status (2)
| Country | Link |
|---|---|
| CN (1) | CN104008126A (en) |
| WO (1) | WO2015149533A1 (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104636465A (en) * | 2015-02-10 | 2015-05-20 | 百度在线网络技术（北京）有限公司 | Webpage abstract generating methods and displaying methods and corresponding devices |
| CN104881403A (en) * | 2015-06-04 | 2015-09-02 | 百度在线网络技术（北京）有限公司 | Word segmentation method and device |
| WO2015149533A1 (en) * | 2014-03-31 | 2015-10-08 | 北京奇虎科技有限公司 | Method and device for word segmentation processing on basis of webpage content classification |
| CN107368489A (en) * | 2016-05-12 | 2017-11-21 | 阿里巴巴集团控股有限公司 | A kind of information data processing method and device |
| CN108334610A (en) * | 2018-02-06 | 2018-07-27 | 北京神州泰岳软件股份有限公司 | A kind of newsletter archive sorting technique, device and server |
| CN108763200A (en) * | 2018-05-15 | 2018-11-06 | 达而观信息科技（上海）有限公司 | Chinese word cutting method and device |
| CN108874869A (en) * | 2018-04-24 | 2018-11-23 | 中国地质大学（武汉） | A kind of method for building up of the geological classes dictionary based on data collaborative |
| WO2019024755A1 (en) * | 2017-08-01 | 2019-02-07 | 阿里巴巴集团控股有限公司 | Webpage information extraction method, apparatus and system, and electronic device |
| CN109326279A (en) * | 2018-11-23 | 2019-02-12 | 北京羽扇智信息科技有限公司 | A kind of method, apparatus of text-to-speech, electronic equipment and storage medium |
| CN110096695A (en) * | 2018-01-30 | 2019-08-06 | 腾讯科技（深圳）有限公司 | Hyperlink label method and apparatus, file classification method and device |
| CN111079428A (en) * | 2019-12-27 | 2020-04-28 | 出门问问信息科技有限公司 | Word segmentation and industry dictionary construction method and device and readable storage medium |
| CN111414648A (en) * | 2020-03-04 | 2020-07-14 | 传神语联网网络科技股份有限公司 | Corpus authentication method and apparatus |
| CN112069288A (en) * | 2019-05-23 | 2020-12-11 | 中国移动通信集团河南有限公司 | Data processing method and device and electronic equipment |
Families Citing this family (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110020420B (en) * | 2018-01-10 | 2023-07-21 | 腾讯科技（深圳）有限公司 | Text processing method, device, computer equipment and storage medium |
| CN109522417A (en) * | 2018-10-26 | 2019-03-26 | 浪潮软件股份有限公司 | A kind of trading company's abstracting method of company name |
| CN109740152B (en) * | 2018-12-25 | 2023-02-17 | 腾讯科技（深圳）有限公司 | Text category determination method and device, storage medium and computer equipment |
| CN110516259B (en) * | 2019-08-30 | 2023-03-07 | 盈盛智创科技(广州)有限公司 | Method and device for identifying technical keywords, computer equipment and storage medium |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1536483A (en) * | 2003-04-04 | 2004-10-13 | 陈文中 | Method for extracting and processing network information and its system |
| CN101206673A (en) * | 2007-12-25 | 2008-06-25 | 北京科文书业信息技术有限公司 | Intelligent error correcting system and method in network searching process |
| CN101441663A (en) * | 2008-12-02 | 2009-05-27 | 西安交通大学 | Chinese text classification characteristic dictionary generating method based on LZW compression algorithm |
| CN101620608A (en) * | 2008-07-04 | 2010-01-06 | 全国组织机构代码管理中心 | Information collection method and system |
| CN102280106A (en) * | 2010-06-12 | 2011-12-14 | 三星电子株式会社 | VWS method and apparatus used for mobile communication terminal |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101206653A (en) * | 2006-12-22 | 2008-06-25 | 英业达股份有限公司 | System and method for automatically collecting network information |
| CN101763395A (en) * | 2009-12-31 | 2010-06-30 | 浙江大学 | Method for automatically generating webpage by adopting artificial intelligence technology |
| CN104008126A (en) * | 2014-03-31 | 2014-08-27 | 北京奇虎科技有限公司 | Method and device for segmentation on basis of webpage content classification |
-
2014
- 2014-03-31 CN CN201410126465.1A patent/CN104008126A/en active Pending
- 2014-12-09 WO PCT/CN2014/093396 patent/WO2015149533A1/en active Application Filing
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1536483A (en) * | 2003-04-04 | 2004-10-13 | 陈文中 | Method for extracting and processing network information and its system |
| CN101206673A (en) * | 2007-12-25 | 2008-06-25 | 北京科文书业信息技术有限公司 | Intelligent error correcting system and method in network searching process |
| CN101620608A (en) * | 2008-07-04 | 2010-01-06 | 全国组织机构代码管理中心 | Information collection method and system |
| CN101441663A (en) * | 2008-12-02 | 2009-05-27 | 西安交通大学 | Chinese text classification characteristic dictionary generating method based on LZW compression algorithm |
| CN102280106A (en) * | 2010-06-12 | 2011-12-14 | 三星电子株式会社 | VWS method and apparatus used for mobile communication terminal |
Cited By (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2015149533A1 (en) * | 2014-03-31 | 2015-10-08 | 北京奇虎科技有限公司 | Method and device for word segmentation processing on basis of webpage content classification |
| CN104636465B (en) * | 2015-02-10 | 2018-11-16 | 百度在线网络技术（北京）有限公司 | Web-page summarization generation method, methods of exhibiting and related device |
| CN104636465A (en) * | 2015-02-10 | 2015-05-20 | 百度在线网络技术（北京）有限公司 | Webpage abstract generating methods and displaying methods and corresponding devices |
| CN104881403A (en) * | 2015-06-04 | 2015-09-02 | 百度在线网络技术（北京）有限公司 | Word segmentation method and device |
| CN107368489A (en) * | 2016-05-12 | 2017-11-21 | 阿里巴巴集团控股有限公司 | A kind of information data processing method and device |
| CN107368489B (en) * | 2016-05-12 | 2020-07-03 | 阿里巴巴集团控股有限公司 | Information data processing method and device |
| WO2019024755A1 (en) * | 2017-08-01 | 2019-02-07 | 阿里巴巴集团控股有限公司 | Webpage information extraction method, apparatus and system, and electronic device |
| CN110096695A (en) * | 2018-01-30 | 2019-08-06 | 腾讯科技（深圳）有限公司 | Hyperlink label method and apparatus, file classification method and device |
| CN110096695B (en) * | 2018-01-30 | 2023-01-03 | 腾讯科技（深圳）有限公司 | Hyperlink marking method and device and text classification method and device |
| CN108334610A (en) * | 2018-02-06 | 2018-07-27 | 北京神州泰岳软件股份有限公司 | A kind of newsletter archive sorting technique, device and server |
| CN108874869A (en) * | 2018-04-24 | 2018-11-23 | 中国地质大学（武汉） | A kind of method for building up of the geological classes dictionary based on data collaborative |
| CN108763200A (en) * | 2018-05-15 | 2018-11-06 | 达而观信息科技（上海）有限公司 | Chinese word cutting method and device |
| CN109326279A (en) * | 2018-11-23 | 2019-02-12 | 北京羽扇智信息科技有限公司 | A kind of method, apparatus of text-to-speech, electronic equipment and storage medium |
| CN112069288A (en) * | 2019-05-23 | 2020-12-11 | 中国移动通信集团河南有限公司 | Data processing method and device and electronic equipment |
| CN111079428A (en) * | 2019-12-27 | 2020-04-28 | 出门问问信息科技有限公司 | Word segmentation and industry dictionary construction method and device and readable storage medium |
| CN111079428B (en) * | 2019-12-27 | 2023-09-19 | 北京羽扇智信息科技有限公司 | Word segmentation and industry dictionary construction method and device and readable storage medium |
| CN111414648A (en) * | 2020-03-04 | 2020-07-14 | 传神语联网网络科技股份有限公司 | Corpus authentication method and apparatus |
| CN111414648B (en) * | 2020-03-04 | 2023-05-12 | 传神语联网网络科技股份有限公司 | Corpus authentication method and device |
Also Published As
| Publication number | Publication date |
|---|---|
| WO2015149533A1 (en) | 2015-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN104008126A (en) | Method and device for segmentation on basis of webpage content classification | |
| CN111460787B (en) | Topic extraction method, topic extraction device, terminal equipment and storage medium | |
| US10025819B2 (en) | Generating a query statement based on unstructured input | |
| US5794177A (en) | Method and apparatus for morphological analysis and generation of natural language text | |
| CN103678684B (en) | A kind of Chinese word cutting method based on navigation information retrieval | |
| US9495358B2 (en) | Cross-language text clustering | |
| CN107818085B (en) | Answer selection method and system for reading understanding of reading robot | |
| CN106021572B (en) | The construction method and device of binary feature dictionary | |
| WO2019229769A1 (en) | An auto-disambiguation bot engine for dynamic corpus selection per query | |
| CN104077275A (en) | Method and device for performing word segmentation based on context | |
| US20130060769A1 (en) | System and method for identifying social media interactions | |
| JP2013502643A (en) | Structured data translation apparatus, system and method | |
| US20110295857A1 (en) | System and method for aligning and indexing multilingual documents | |
| CN102214189B (en) | Data mining-based word usage knowledge acquisition system and method | |
| KR101509727B1 (en) | Apparatus for creating alignment corpus based on unsupervised alignment and method thereof, and apparatus for performing morphological analysis of non-canonical text using the alignment corpus and method thereof | |
| CN101796508A (en) | Coreference resolution in an ambiguity-sensitive natural language processing system | |
| CN102253930A (en) | Method and device for translating text | |
| GB2555207A (en) | System and method for identifying passages in electronic documents | |
| Gencosman et al. | Character n-gram application for automatic new topic identification | |
| CN103544266A (en) | Method and device for generating search suggestion words | |
| Jabbar et al. | An improved Urdu stemming algorithm for text mining based on multi-step hybrid approach | |
| CN110532354A (en) | The search method and device of content | |
| Hamdi et al. | In-depth analysis of the impact of OCR errors on named entity recognition and linking | |
| US20110178792A1 (en) | Acquisition Of Out-Of-Vocabulary Translations By Dynamically Learning Extraction Rules | |
| Wijeratne et al. | Sinhala language corpora and stopwords from a decade of sri lankan facebook |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20140827 |