CN104916131A - Freeway incident detection data cleaning method - Google Patents

Abstract

The invention belongs to the technical field of road traffic detection. A method comprises the steps that a condition with delay time tolerance ta is used to determine whether freeway incident detection data are missing; a condition screening method and a screening method based on statistics and traffic flow theory are used to determine whether the freeway incident detection data are abnormal; if non morning time data in the missing data and the abnormal data are all zero, a moving average method is used for repairing; and if occupancy and average speed in the abnormal data are high at the same time and partial traffic parameters are zero, an inverse proportion method is used for repairing. According to the invention, the method is applicable to freeway incident detection system practical application; the screening methods can dynamically adapt to different flow changes and meet a traffic flow mechanism; a repairing algorithm can retain partial real information of the abnormal data of the current period; a repairing result is closer to real data; and the method has the advantages of low computational complexity and small system overhead.

Description

The Data Cleaning Method of freeway incident detection

Technical field

The invention belongs to road traffic detection technique field, be specifically related to a kind of Data Cleaning Method of freeway incident detection.

Background technology

In recent years, again and again there is the problem such as traffic congestion, traffic hazard in highway, affects freeway network operational efficiency, along with the development of intelligent transport technology, the application research and development that freeway incident detection system drops into successively, bring opportunity for solving freeway incident management.

Freeway incident detection system relies on a large amount of data obtained from sensor, the degree of reliability of these data directly can affect credibility and the reliability of freeway incident detection, but due to the restriction of technical merit and engineering specifications, the current data degree of reliability can not meet Practical Project demand.Because China's intelligent transportation system starting is relatively late, auxiliary facility (traffic information acquisition system, transmission system, hardware facility etc.) is built and imperfection, and the Vehicle Detection data reliability uploading to freeway incident detection system also exists very large deficiency.Engineering practice shows, various data unreliable factor causes the Detection results of freeway incident detection system unsatisfactory.

Therefore, data cleansing work is necessity work improving freeway incident detection system reliability.Data cleansing work mainly contains two contents: the first, and screening causes the data of system cisco unity malfunction (comprising shortage of data, data exception etc.); The second, these abnormal data are repaired.

Available data screening technique is mainly divided into Corpus--based Method and based on traffic flow theory two class methods.The data screening method (as time series method, exponential smoothing etc.) of Corpus--based Method, its advantage can evaluate current data by the historical information of data, and the historical data of each cycle data is not quite similar, therefore there is dynamic, can adapt to the change of different flow feature, shortcoming does not consider basic three parameters relationships of traffic flow theory; Based on the traffic data screening technique (as based on traffic flow three parameters relationship method, flow Conservation Method etc.) of traffic flow theory, its advantage considers traffic flow mechanism, standard for evaluating data must meet traffic flow mechanism, and shortcoming lacks dynamic.

The method of the relevant solution of data restore aspect mainly Corpus--based Method, comprises time series, regression model etc., mainly utilizes historical data to carry out predicting and repairing.

From existing Data Cleaning Method, on the one hand, two class data screening methods respectively have relative merits, not yet have good both methods combining advantage and evade the two shortcoming; On the other hand, data recovery method is repaired by means of only historical data, have ignored the part real information that current period data retain.Therefore, how in conjunction with the advantage of two class data screening methods, avoid shortcoming, and effectively hold the true telecommunication flow information that current period data retain, for the data reliability important in inhibiting improving freeway incident detection system at repairing phase.

Summary of the invention

In view of this, the object of the present invention is to provide a kind of Data Cleaning Method of freeway incident detection, to reduce the impact that shortage of data and data exception phenomenon cause freeway incident detection system works, credible with the work of elevator system.

For achieving the above object, the invention provides following technical scheme:

The Data Cleaning Method of freeway incident detection, comprises the following steps:

1) band tolerance t time delay is adopted _athe data of condition judgment freeway incident detection whether lack; Adopt conditional filtering method to judge the data whether exception of freeway incident detection, described condition comprise non-morning time data be 0 entirely, speed or occupation rate be 0 higher than threshold value, part traffic parameter;

2) missing data and abnormal data are repaired, for non-morning in shortage of data and abnormal data, time data is the situation of 0 entirely, moving average method is adopted to repair, for two kinds of situations that higher while of occupation rate in abnormal data and average speed, part traffic parameter are 0, inverse proportion method is adopted to repair.

Further, described step 1) specifically adopt following steps:

101) the tolerance t of setting data time delay _a;

102) t is postponed _atime reads current period data;

103) current period data are read;

104) if current period has data, step 105 is performed), if current period does not have data, be then labeled as shortage of data, and the screening process of end data disappearance;

105) judge read data whether be 0 entirely, if so, then redirect perform step 106), if not, then redirect perform step 107);

106) judging whether it is morning, if morning, is then normal data, if non-morning, is then abnormal data, terminates screening, indicates that non-morning, data were 0 and redirect performs step 2 entirely);

107) read n cycle data before current period and calculate front n cycle value:

$\∂=q\left(t\right)/[o\left(t\right)*v\left(t\right)];$

Wherein q (t) represent current period before n cycle data flow value, o (t) represent current period before n cycle occupation rate value, v (t) represent current period before n cycle average speed value, arrange from small to large simultaneously value;

108) gauge index p _i:

p _i＝(i/n)*100

Wherein i is the ascending arrangement sequence number of a front n cycle data;

109) if p _i>25, then i-th data value is then first quartile Q1, and if p _i>75, then i-th data value is then third quartile Q3;

110) calculate interquartile-range IQR IQR, the computing formula of IQR is IQR=Q3-Q1;

111): calculate span $[\∂\_\min ,\∂\_\max ]=[Q1-\mathrm{IQR},Q3+3*\mathrm{IQR}],$ Wherein with be respectively the value upper limit and value lower limit;

112) span [q (t) _ min, q (t) _ max] of q (t) is calculated, $[q\left(t\right)\_\min ,q\left(t\right)\_\max ]=[\∂\_\min *o\left(t\right)*v\left(t\right),\∂\_\max *o\left(t\right)*v\left(t\right)],$ Wherein q (t) _ min is the value lower limit of flow, q (t) _ max is the value upper limit of flow, q (t) represents current period data stream value, and o (t) represents current period occupation rate value, and v (t) represents current period average speed value;

113) judge whether q (t) is less than q (t) _ min or is greater than q (t) _ max, if so, then thinks abnormal data.

Further, described step 2) specifically comprise the steps:

201) judge whether current data belongs to shortage of data situation, if not then entering 202), if so, then adopt the moving average method of weighting carry out data restore and terminate repair process;

202) judge current data be whether non-morning time data be entirely 0 situation, if not then directly terminating repair process, if so, then adopt the moving average method of weighting carry out data restore and terminate repair process;

203) judging whether to belong to data exception and need to repair, if not then terminating repair process, if so, then performing step 204);

204) β value is calculated:

Judge that whether flow is higher than threshold value, if, then β=q (t)-q (t) _ max, if not, then β=Q_min-q (t), wherein q (t) is current period flow value, Q_min and Q_max is respectively flow value lower limit in data screening process and the value upper limit;

205) carry out data restore, repair formula as follows:

$\hat{y}\left(t\right)=\frac{1}{\mathrm{\β}}y\left(t\right)+\frac{\mathrm{\α}*\mathrm{\β}-\mathrm{\α}}{\mathrm{\β}}y(t-1)+\frac{\mathrm{\β}-\mathrm{\α}*\mathrm{\β}-1+\mathrm{\α}}{\mathrm{\β}}y(t-2);$

Wherein, for the magnitude of traffic flow, average speed or occupation rate after reparation, y (t-1) and y (t-2) is respectively the flow in previous cycle and the first two cycle, average speed or occupation rate, 1/ β is called inverse proportion coefficient, and α is weighting coefficient, generally gets 0.4-0.8.

Further, to carry out the formula of data restore as follows for the moving average method of described weighting:

$\hat{y}\left(t\right)=\mathrm{\αy}(t-1)+(1-\mathrm{\α})y(t-2);$

Wherein for the magnitude of traffic flow, average speed or occupation rate after repairing, y (t-1) and y (t-2) is respectively the flow in previous cycle and the first two cycle, average speed or occupation rate, and α is weighting coefficient, gets 0.4-0.8.

The present invention has the following advantages relative to prior art tool: the present invention is applicable to freeway incident detection system practical engineering application, screening technique can change by dynamically adapting different flow, and meet traffic flow mechanism, restore design can retain the part real information of current period abnormal data, repair result closer to True Data, and operand is lower, system overhead is little.

Accompanying drawing explanation

Fig. 1 shows the schematic flow sheet of data screening in the Data Cleaning Method of freeway incident detection;

Fig. 2 shows the schematic flow sheet of data restore in the Data Cleaning Method of freeway incident detection.

Embodiment

In order to make the object, technical solutions and advantages of the present invention clearly, will be described in further detail the specific embodiment of the present invention below.

The Data Cleaning Method of the freeway incident detection of the present embodiment, comprises the steps:

1) band tolerance t time delay is adopted _athe data of condition judgment freeway incident detection whether lack, the data of described freeway incident detection comprise the magnitude of traffic flow, average speed and occupation rate; Whether the data of freeway incident detection are abnormal to adopt conditional filtering method to judge; Described condition comprise non-morning time data be 0 entirely, speed or occupation rate be 0 higher than threshold value, part traffic parameter; Because freeway incident detection system is " in real time " detection system, due to the existence (each system clock asynchronous, transmission delay, computing relay etc.) of various factors, data transmission and detection system can not reach absolute real-time, therefore, for the data screening of disappearance, the degrees of tolerance t for data delay time first to be determined _a, namely time delay is at t _abe acceptable in scope, generally speaking, the tolerance of time delay is arranged according to sense cycle T, and general desirable scope is T/2 ~ 2*T, can according to user's sets itself of freeway incident detection system.

See Fig. 1, step 1) specifically comprise the steps:

101) the tolerance t of setting data time delay _a;

102) t is postponed _atime reads current period data;

103) current period data are read;

104) if current period has data, step 105 is performed), if current period does not have data, be then labeled as shortage of data, and the screening process of end data disappearance;

105) judge read data whether be 0 entirely, if so, then redirect perform step 106), if not, then redirect perform step 107);

106) judging whether it is morning, if morning, is then normal data, if non-morning, is then abnormal data, terminates screening, indicates that non-morning, data were 0 and redirect performs step 2 entirely);

107) read n cycle data before current period and calculate front n cycle value:

$\∂=q\left(t\right)/[o\left(t\right)*v\left(t\right)];$

Wherein q (t) represent current period before n cycle data flow value, o (t) represent current period before n cycle occupation rate value, v (t) represent current period before n cycle average speed value, arrange from small to large simultaneously value;

108) gauge index p _i:

p _i＝(i/n)*100

Wherein i is the ascending arrangement sequence number of a front n cycle data;

109) if p _i>25, then i-th data value is then first quartile Q1, and if p _i>75, then i-th data value is then third quartile Q3;

110) calculate interquartile-range IQR IQR, the computing formula of IQR is IQR=Q3-Q1;

111): calculate span $[\∂\_\min ,\∂\_\max ]=[Q1-\mathrm{IQR},Q3+3*\mathrm{IQR}],$ Wherein with be respectively the value upper limit and value lower limit;

112) span [q (t) _ min, q (t) _ max] of q (t) is calculated, $[q\left(t\right)\_\min ,q\left(t\right)\_\max ]=[\∂\_\min *o\left(t\right)*v\left(t\right),\∂\_\max *o\left(t\right)*v\left(t\right)],$ Wherein q (t) _ min is the value lower limit of flow, q (t) _ max is the value upper limit of flow, q (t) represents current period data stream value, and o (t) represents current period occupation rate value, and v (t) represents current period average speed value;

113) judge whether q (t) is less than q (t) _ min or is greater than q (t) _ max, if so, then thinks abnormal data.

2) missing data and abnormal data are repaired, for non-morning in shortage of data and abnormal data, time data is the situation of 0 entirely, moving average method is adopted to repair, for two kinds of situations that higher while of occupation rate in abnormal data and average speed, part traffic parameter are 0, inverse proportion method is adopted to repair.

See Fig. 2, described step 2) specifically comprise the steps:

201) judge whether current data belongs to shortage of data situation, if not then entering 202), if so, then adopt the moving average method of weighting carry out data restore and terminate repair process;

202) judge current data be whether non-morning time data be entirely 0 situation, if not then directly terminating repair process, if so, then adopt the moving average method of weighting carry out data restore and terminate repair process;

203) judging whether to belong to data exception and need to repair, if not then terminating repair process, if so, then performing step 204);

204) β value is calculated:

Judge that whether flow is higher than threshold value, if, then β=q (t)-q (t) _ max, if not, then β=Q_min-q (t), wherein q (t) is current period flow value, Q_min and Q_max is respectively flow value lower limit in data screening process and the value upper limit;

205) carry out data restore, repair formula as follows:

$\hat{y}\left(t\right)=\frac{1}{\mathrm{\β}}y\left(t\right)+\frac{\mathrm{\α}*\mathrm{\β}-\mathrm{\α}}{\mathrm{\β}}y(t-1)+\frac{\mathrm{\β}-\mathrm{\α}*\mathrm{\β}-1+\mathrm{\α}}{\mathrm{\β}}y(t-2);$

Wherein, for the magnitude of traffic flow, average speed or occupation rate after reparation, y (t-1) and y (t-2) is respectively the flow in previous cycle and the first two cycle, average speed or occupation rate, 1/ β is called inverse proportion coefficient, and α is weighting coefficient, generally gets 0.4-0.8.

The formula that the moving average method of described weighting carries out data restore is as follows:

$\hat{y}\left(t\right)=\mathrm{\αy}(t-1)+(1-\mathrm{\α})y(t-2);$

Wherein for the magnitude of traffic flow, average speed or occupation rate after repairing, y (t-1) and y (t-2) is respectively the flow in previous cycle and the first two cycle, average speed or occupation rate, and α is weighting coefficient, gets 0.4-0.8.

Current data recovery method is most popular is the data recovery method of Corpus--based Method, and the most typically utilize multiple cycle data to carry out the method for moving average, this restorative procedure mainly utilizes history cycle data to repair.

Existing moving average method mainly utilizes historical data reparation, and does not consider the part real information that current period data retain, and therefore, in order to retain the part real information of current period abnormal data, employing inverse proportion method is repaired.

Inverse proportion is repaired the physical significance of coefficient and is, when data, to depart from normal data span far away, then more unreliable, and information of its reservation is then fewer; When data, to depart from normal data span nearer, then illustrate that the credibility of these data is higher, more close to actual value.

The feature of this restorative procedure is: in the repair process of abnormal data, adopts current period measured data, retains the part real information of abnormal data.Although current period data are abnormal, but partial information is real, during as there is traffic events, occupation rate should continue higher, if now with only repairing with the first two cycle data, data will be made can not to retain Current traffic event conditions, thus cause freeway incident detection system alarm to be removed, or when traffic events continues to occur when date restoring is normal, cause repetition of alarms.

What finally illustrate is, above embodiment is only in order to illustrate technical scheme of the present invention and unrestricted, although with reference to preferred embodiment to invention has been detailed description, those of ordinary skill in the art is to be understood that, can modify to technical scheme of the present invention or equivalent replacement, and not departing from aim and the scope of technical solution of the present invention, it all should be encompassed in the middle of right of the present invention.

Claims (4)

1. the Data Cleaning Method of freeway incident detection, is characterized in that: comprise the following steps:

1) band tolerance t time delay is adopted _athe data of condition judgment freeway incident detection whether lack; Adopt conditional filtering method to judge the data whether exception of freeway incident detection, described condition comprise non-morning time data be 0 entirely, speed or occupation rate be 0 higher than threshold value, part traffic parameter;

2) missing data and abnormal data are repaired, for non-morning in shortage of data and abnormal data, time data is the situation of 0 entirely, moving average method is adopted to repair, for two kinds of situations that higher while of occupation rate in abnormal data and average speed, part traffic parameter are 0, inverse proportion method is adopted to repair.

2. the Data Cleaning Method of freeway incident detection as claimed in claim 1, is characterized in that: described step 1) specifically adopt following steps:

101) the tolerance t of setting data time delay _a;

102) t is postponed _atime reads current period data;

103) current period data are read;

104) if current period has data, step 105 is performed), if current period does not have data, be then labeled as shortage of data, and the screening process of end data disappearance;

105) judge read data whether be 0 entirely, if so, then redirect perform step 106), if not, then redirect perform step 107);

106) judging whether it is morning, if morning, is then normal data, if non-morning, is then abnormal data, terminates screening, indicates that non-morning, data were 0 and redirect performs step 2 entirely);

107) read n cycle data before current period and calculate front n cycle value:

$\∂=q\left(t\right)/[o\left(t\right)*v\left(t\right)];$

Wherein q (t) represent current period before n cycle data flow value, o (t) represent current period before n cycle occupation rate value, v (t) represent current period before n cycle average speed value, arrange from small to large simultaneously value;

108) gauge index pi:

p _i＝(i/n)*100

Wherein i is the ascending arrangement sequence number of a front n cycle data;

109) if p _i>25, then i-th data value is then first quartile Q1, and if p _i>75, then i-th data value is then third quartile Q3;

110) calculate interquartile-range IQR IQR, the computing formula of IQR is IQR=Q3-Q1;

111): calculate span $[\∂\_\min ,\∂\_\max ]=[Q1-\mathrm{IQR},Q3+3*\mathrm{IQR}],$ Wherein with be respectively the value upper limit and value lower limit;

112) span [q (t) _ min, q (t) _ max] of q (t) is calculated, $[q\left(t\right)\_\min ,q\left(t\right)\_\max ]=[\∂\_\min *o\left(t\right)*v\left(t\right),\∂\_\max *o\left(t\right)*v\left(t\right)],$ Wherein q (t) _ min is the value lower limit of flow, q (t) _ max is the value upper limit of flow, q (t) represents current period data stream value, and o (t) represents current period occupation rate value, and v (t) represents current period average speed value;

113) judge whether q (t) is less than q (t) _ min or is greater than q (t) _ max, if so, then thinks abnormal data.

3. the Data Cleaning Method of freeway incident detection as claimed in claim 2, is characterized in that: described step 2) specifically comprise the steps:

201) judge whether current data belongs to shortage of data situation, if not then entering 202), if so, then adopt the moving average method of weighting carry out data restore and terminate repair process;

202) judge current data be whether non-morning time data be entirely 0 situation, if not then directly terminating repair process, if so, then adopt the moving average method of weighting carry out data restore and terminate repair process;

203) judging whether to belong to data exception and need to repair, if not then terminating repair process, if so, then performing step 204);

204) β value is calculated:

Judge that whether flow is higher than threshold value, if, then β=q (t)-q (t) _ max, if not, then β=Q_min-q (t), wherein q (t) is current period flow value, Q_min and Q_max is respectively flow value lower limit in data screening process and the value upper limit;

205) carry out data restore, repair formula as follows:

$\hat{y}\left(t\right)=\frac{1}{\mathrm{\β}}y\left(t\right)+\frac{\mathrm{\α}*\mathrm{\β}-\mathrm{\α}}{\mathrm{\β}}y(t-1)+\frac{\mathrm{\β}-\mathrm{\α}*\mathrm{\β}-1+\mathrm{\α}}{\mathrm{\β}}y(t-2);$

Wherein, for the magnitude of traffic flow, average speed or occupation rate after reparation, y (t-1) and y (t-2) is respectively the flow in previous cycle and the first two cycle, average speed or occupation rate, 1/ β is called inverse proportion coefficient, and α is weighting coefficient, generally gets 0.4-0.8.

4. the Data Cleaning Method of freeway incident detection as claimed in claim 3, is characterized in that: the formula that the moving average method of described weighting carries out data restore is as follows:

$\hat{y}\left(t\right)=\mathrm{\αy}(t-1)+(1-\mathrm{\α})y(t-2);$

Wherein for the magnitude of traffic flow, average speed or occupation rate after repairing, y (t-1) and y (t-2) is respectively the flow in previous cycle and the first two cycle, average speed or occupation rate, and α is weighting coefficient, gets 0.4-0.8.