CN102819406A - Front-end data storage method and device - Google Patents
Front-end data storage method and device Download PDFInfo
- Publication number
- CN102819406A CN102819406A CN2012102311006A CN201210231100A CN102819406A CN 102819406 A CN102819406 A CN 102819406A CN 2012102311006 A CN2012102311006 A CN 2012102311006A CN 201210231100 A CN201210231100 A CN 201210231100A CN 102819406 A CN102819406 A CN 102819406A
- Authority
- CN
- China
- Prior art keywords
- data
- index
- write
- index data
- network storage
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Abstract
The invention provides a front-end data storage device applied to front end equipment. The front end equipment is connected to a network storage device through a network. The device comprises a writing judging unit and a data filling unit, wherein the writing judging unit is used for judging whether index data written at present can cover the overall index area when the index data are written into corresponding index area, if so, writing is carried out, otherwise, index data are transferred to the data filling unit for processing; the data filling unit is used for filling the index data by invalid data, so that the filled index data can cover corresponding index area, and submitting the filled index data into the data writing unit. Compared with the prior art, as the index data are filled in the front end, an operation of reading additionally disk data caused by reconstruction writing when the index data are written is avoided, and the writing performance and efficiency are improved.
Description
Technical field
The present invention relates to the data processing technique of front end terminal device, relate in particular to a kind of front end data storage means and device.
Background technology
(Redundant Array of Independent Disks, RAID) technology is widely used in field of storage to independent disk RAID array, and is especially popular in the network storage.The RAID technology mainly is that the redundancy through disk guarantees to store the availability of data above that.Along with the continuous increase of single disk size, the size of main flow disk has reached the TB magnitude at present, and the probability that mistake or magnetic track fault appear in same disk also increases to some extent.
When the supervisory system application network was stored, the front end encoding device all was the index area of writing video flowing earlier, writes video stream data again, and for increasing the reliability of storage data, memory device generally can dispose the RAID5 array and do data protection.Because storage all was video data (or even video data of high definition) during monitoring was used; Therefore data volume is extremely huge; This has caused the user relatively more responsive to storage aspect operation cost, so becomes the focus that the client pays close attention to the serviceable life of disk tolerance technology and prolongation disk.
There is following problem in the network storage at present in monitoring is used: the front end encoding device can write some monitoring index datas usually in the continuous video data stream of needs storage; The monitoring index data is generally little than preallocated space, and causing the memory device of supporting raid5 can't be that unit carries out data and writes with whole band.Memory device can adopt the reconstruct WriteMode or read Overtype Mode this moment.The processing procedure that (Reconstruct Write) write in reconstruct comprises: read original data in the band piece (Segment) that from current band, need not revise; Again with this band in new data on all Segment that need revise calculate parity values, and new Segment data and the Segment data of more not correcting one's mistakes and new parity values are written to current band in the lump.Obviously reconstruct is write and will be involved more I/O operation, so the whole band of efficiency ratio writes much lower together.
It is following to read the Overtype Mode processing procedure: the Segment that revises from needs earlier reads old data, reads old parity values from band again; New data according on the Segment of legacy data, old proof test value and needs modification calculates the new proof test value on this band; Write new data and new parity values at last.
Please refer to Fig. 1, the rank of supposing the RAID array is RAID5.At first it should be noted that: wherein second of the form row is the character of expression disk space store data, representes it is the index area such as Index, and data representes the data field, is a kind of diagram performance of image, is not data in magnetic disk.Carry out data at the front end encoding device and write fashionablely, if the current data that will write comprise index data, it may occur, the exhaustless situation in the space of pre-assigned index area.It is the band piece that tentation data is stored in the 64KB size, on the RAID5 array of 9 dishes.Suppose that the index area size of distributing is 128KB.Current actual index data only need write 64KB, and the band piece that Disk2 shown in Figure 1 then will occur need not write the situation of data for empty.This moment memory device can to find to use reconstruct to write be the preferable mode of performance.Promptly read data (invalid) the back calculation check data Disk9 of Disk2, then the data of one-time write Disk1～9 to business.Suppose this moment read error to take place when reading the data on the Disk2, need read Disk1 this moment earlier, and Disk3～Disk9 calculates the data of Disk2, and the process read data number of times that whole reconstruct is write increases to 8 times from 1 time, and treatment effeciency is had a greatly reduced quality.Further, if suppose that this moment, Disk8 owing to a variety of causes read error took place, because RAID5 only supports just to calculate the data of Disk2 this moment, reconstruct is write and can't be accomplished.
Summary of the invention
In view of this, the present invention provides a kind of front end data memory storage, is applied on the front-end equipment, and this front-end equipment is connected to the network storage equipment through network, and this device comprises:
Write judging unit, be used for when writing index data and write the manipulative indexing district, judging whether the current index data that writes can cover whole index area, if revolution handles according to writing unit, otherwise revolution is handled according to filler cells; Wherein said index area is the pre-assigned storage space of the network storage equipment, and it comprises the band piece that one or more network storage equipment distributes;
The data filler cells be used to use invalid data index data to be filled so that the index data after filling can cover corresponding index area, and the index data after will filling is submitted to data write unit;
Data write unit is used for index data is write corresponding index area.
Also a kind of front end data storage means of the present invention is applied on the front-end equipment, and this front-end equipment is connected to the network storage equipment through network, it is characterized in that, this method comprises:
A, when writing index data and write the manipulative indexing district, judge whether the current index data that writes can cover whole index area, if revolution handles according to writing unit, otherwise revolution is handled according to filler cells; Wherein said index area is the pre-assigned storage space of the network storage equipment, and it comprises the band piece that one or more network storage equipment distributes;
B, use invalid data are filled index data so that the index data after filling can cover corresponding index area, and the index data after will filling is submitted to data write unit;
C, index data is write corresponding index area.
With respect to prior art, because the present invention has carried out the filling of index data at front end, adopt reconstruct to write the extra operation of reading data in magnetic disk of the needs that caused when having avoided writing index data, promoted the performance and the efficient that write; Also avoided simultaneously reconstruct write in two band piece read errors caused can't write problem.
Description of drawings
Fig. 1 is that prior art index data and video data are organized synoptic diagram on memory device.
Fig. 2 is the building-block of logic of front end data memory storage in one embodiment of the present invention.
Fig. 3 is a front end data storage means process flow diagram in one embodiment of the present invention.
Fig. 4 is that index and video data are organized synoptic diagram in one embodiment of the present invention on the network storage equipment.
Embodiment
Please refer to Fig. 2, the present invention provides a kind of front end data memory storage, is applied on the front-end equipment, and this front-end equipment is connected to the network storage equipment through network.Be embodied as example with computer program, this front end data memory storage comprises and writes judging unit, data filler cells and data write unit.With the front-end equipment in the video monitoring system is that front end encoding device (such as scrambler EC) is that example describes.The present invention also is confined to the front end encoding device of video monitoring system.When in other system or network, using, different is just professional own, and business datum is a video data in supervisory system, and possibly be the business datum of other types in other system or the network.Said front end data memory storage is carried out following steps when operation:
Step 101 writes judging unit when writing index data and write the manipulative indexing district, judges whether the current index data that writes can cover whole index area, if revolution handles according to writing unit, otherwise revolution is handled according to filler cells; Wherein said index area is the pre-assigned storage space of the network storage equipment, and it comprises the band piece that one or more network storage equipment distributes;
EC sends live video stream except needs to decoding end in video monitoring system, also need usually to send a store video stream to the network storage equipment, can carry out playback by follow-up video to storage in order to the user.Be stored as example with IP, EC can set up iSCIS with the network storage equipment and be connected, and through this iSCIS the data that needs write is sent to the network storage equipment then.EC can be decomposed into the storage operation of two kinds of data according to data type in the operation of the said video flowing of storage, a kind of is to write index data to the index area, and another kind is to write video data to the data field.Prior art does not have to consider to write index data and write video data characteristics separately, video data continuous data normally in fact, and index data then is not continuous data usually.The present invention requires to write judging unit need jump out normal data writing process earlier at EC when memory device writes index data, change step 102 over to and carry out the data filler cells and handle.
Step 102, data filler cells use invalid data that index data is filled so that the index data after filling can cover corresponding index area, and the index data after will filling is submitted to data write unit;
As previously mentioned, because video data continuous data normally, do not have generally speaking therefore that a band can't be by the full situation of video data write on the memory device.Please refer to Fig. 4, suppose that the network storage equipment is is the band piece with the 64KB size, it comprises the RAID5 array of 9 dishes.Suppose that it is 128KB (2 band block sizes) that the network storage equipment is distributed to the index area size of this EC.If the current actual index data of EC has only 64KB, this moment, the data filler cells can be filled the invalid data of other 64KB.To fill the index data of back formation then and submit to data write unit.
Step 103, data write unit writes corresponding index area with index data.
Step 104, the network storage equipment receive the index data of EC transmission and this index data are left in the buffer memory, and wait for the arrival of the video data that index data is pointed;
Step 105, data write unit continue the video data that index data is corresponding and write the data field;
Step 106, the network storage equipment will receive the video data of EC transmission and the data of a whole piece band of the composition of the index data in the aforementioned buffer memory are written in the RAID array.
In preferred embodiment; EC can set up two-stage index on the network storage equipment; It is preferable that the present invention writes effect for the index data in secondary index district, because the secondary index district can constantly write index data, the index data in secondary index district points to the data field of a correspondence.The variation of one-level index area is less relatively, and the secondary index district has more variation relatively, and index data wherein can be followed business datum (such as video data) and upgraded synchronously; Such as in the storage of video data; The storage space of user's planning possibly be limited, such as only can store one month space size, after arriving in one month; Video data in the past can progressively be covered by new video data, and therefore corresponding index data also can upgrade synchronously.
The index data in secondary index district points to a data field as previously mentioned, and wherein the size of index data may change because of the number change of the data block in the data field.If the user is distinguished into the more data piece with current data, index data can increase undoubtedly so, if be divided into data block still less, index data can diminish undoubtedly so, so the size of index data depends on the use of user to the data field.So in fact in planning secondary index district size, can take into account the use of user usually, so being provided with of index area size relatively can be bigger, guarantees to be large enough to hold index data to be written to the data field.
Please refer to Fig. 4, the index area that EC will write comprises two band pieces on the band, is user's index data on the band, and another is the invalid data of arbitrarily filling.After follow-up video data arrived, the network storage equipment just can obtain the data of a complete band, carries out the write operation of whole strip data then.And need not go to carry out the operation that reconstruct is write to the prior art that kind.
With respect to prior art,, make the network storage equipment always write data like this with the mode of whole piece band because EC of the present invention has carried out the filling of index data at front end.Adopt reconstruct to write the extra operation of reading data in magnetic disk of the needs that caused when at first having avoided writing index data, promoted the performance and the efficient that write.Secondly; Because whole band writes and does not relate to read operation; Therefore avoided reconstruct write in two band piece read errors caused can't write problem; Because if current band piece goes wrong, for example, the magnetic track fault cause the band piece occur can't reading of data problem, have only through write operation this moment just can address this problem.When the network storage equipment when writing data, if the physical space generation magnetic track fault at current band piece place, the disk of the network storage equipment can support that usually the band piece is substituted into another physical space at bottom to get on.And this characteristic can't be used during read data, because there are not needed data in another space.The present invention has utilized these data of network storage equipment bottom to write characteristics just and has gone to solve the problem that is stored in the prior art.In addition, consider in the video monitoring system that the existence of a large amount of EC is normally arranged, this moment, EC can increase substantially the performance and the efficient of the network storage equipment in the little change of software view.
The above is merely preferred embodiment of the present invention, and is in order to restriction the present invention, not all within spirit of the present invention and principle, any modification of being made, is equal to replacement, improvement etc., all should be included within the scope that the present invention protects.
Claims (8)
1. a front end data memory storage is applied on the front-end equipment, and this front-end equipment is connected to the network storage equipment through network, it is characterized in that, this device comprises:
Write judging unit, be used for when writing index data and write the manipulative indexing district, judging whether the current index data that writes can cover whole index area, if revolution handles according to writing unit, otherwise revolution is handled according to filler cells; Wherein said index area is the pre-assigned storage space of the network storage equipment, and it comprises the band piece that one or more network storage equipment distributes;
The data filler cells be used to use invalid data index data to be filled so that the index data after filling can cover corresponding index area, and the index data after will filling is submitted to data write unit;
Data write unit is used for index data is write corresponding index area.
2. device as claimed in claim 1 is characterized in that, said index data is the secondary index data, and said index area is the secondary index district on the network storage equipment.
3. device as claimed in claim 2 is characterized in that, said data write unit is further used for business datum is write corresponding data field, and said index data points to corresponding data field.
4. device as claimed in claim 3 is characterized in that, said front-end equipment is the front end encoding device, and said business datum is a video data, and wherein said index data is along with video data upgrades synchronously.
5. a front end data storage means is applied on the front-end equipment, and this front-end equipment is connected to the network storage equipment through network, it is characterized in that, this method comprises:
A, when writing index data and write the manipulative indexing district, judge whether the current index data that writes can cover whole index area, if revolution handles according to writing unit, otherwise revolution is handled according to filler cells; Wherein said index area is the pre-assigned storage space of the network storage equipment, and it comprises the band piece that one or more network storage equipment distributes;
B, use invalid data are filled index data so that the index data after filling can cover corresponding index area, and the index data after will filling is submitted to data write unit;
C, index data is write corresponding index area.
6. method as claimed in claim 5 is characterized in that, said index data is the secondary index data, and said index area is the secondary index district on the network storage equipment.
7. method as claimed in claim 5 is characterized in that, said step C further comprises: business datum is write corresponding data field, and said index data points to corresponding data field.
8. method as claimed in claim 7 is characterized in that, said front-end equipment is the front end encoding device, and said business datum is a video data, and wherein said index data is along with video data upgrades synchronously.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012102311006A CN102819406A (en) | 2012-07-04 | 2012-07-04 | Front-end data storage method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012102311006A CN102819406A (en) | 2012-07-04 | 2012-07-04 | Front-end data storage method and device |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102819406A true CN102819406A (en) | 2012-12-12 |
Family
ID=47303537
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012102311006A Pending CN102819406A (en) | 2012-07-04 | 2012-07-04 | Front-end data storage method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102819406A (en) |
Cited By (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902232A (en) * | 2012-12-28 | 2014-07-02 | 联想(北京)有限公司 | Data write-in method and device |
| CN104750773A (en) * | 2013-12-31 | 2015-07-01 | 国际商业机器公司 | Index maintenance based on a comparison of rebuild vs. update |
| CN111143117A (en) * | 2018-11-06 | 2020-05-12 | 浙江宇视科技有限公司 | Data storage method and device |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020040413A1 (en) * | 1995-01-13 | 2002-04-04 | Yoshiyuki Okada | Storage controlling apparatus, method of controlling disk storage device and method of managing compressed data |
| CN101415024A (en) * | 2007-09-28 | 2009-04-22 | 赛门铁克公司 | Storing and searching data by using cooperation storing device redundantly |
| CN101556564A (en) * | 2008-04-11 | 2009-10-14 | 联芯科技有限公司 | Data receiving/sending method and device |
| CN101727299A (en) * | 2010-02-08 | 2010-06-09 | 北京同有飞骥科技有限公司 | RAID5-orientated optimal design method for writing operation in continuous data storage |
| CN102063270A (en) * | 2010-12-28 | 2011-05-18 | 成都市华为赛门铁克科技有限公司 | Write operation method and device |
| CN102185929A (en) * | 2011-06-02 | 2011-09-14 | 杭州华三通信技术有限公司 | Storage area network (SAN)-resource-based video monitoring data storage method and device |
| CN102223510A (en) * | 2011-06-03 | 2011-10-19 | 杭州华三通信技术有限公司 | Method and device for scheduling cache |
| CN102291256A (en) * | 2011-07-25 | 2011-12-21 | 杭州华三通信技术有限公司 | Monitoring data query method and equipment |
| CN102520883A (en) * | 2011-12-12 | 2012-06-27 | 杭州华三通信技术有限公司 | Data access method and device thereof |
-
2012
- 2012-07-04 CN CN2012102311006A patent/CN102819406A/en active Pending
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20020040413A1 (en) * | 1995-01-13 | 2002-04-04 | Yoshiyuki Okada | Storage controlling apparatus, method of controlling disk storage device and method of managing compressed data |
| CN101415024A (en) * | 2007-09-28 | 2009-04-22 | 赛门铁克公司 | Storing and searching data by using cooperation storing device redundantly |
| CN101556564A (en) * | 2008-04-11 | 2009-10-14 | 联芯科技有限公司 | Data receiving/sending method and device |
| CN101727299A (en) * | 2010-02-08 | 2010-06-09 | 北京同有飞骥科技有限公司 | RAID5-orientated optimal design method for writing operation in continuous data storage |
| CN102063270A (en) * | 2010-12-28 | 2011-05-18 | 成都市华为赛门铁克科技有限公司 | Write operation method and device |
| CN102185929A (en) * | 2011-06-02 | 2011-09-14 | 杭州华三通信技术有限公司 | Storage area network (SAN)-resource-based video monitoring data storage method and device |
| CN102223510A (en) * | 2011-06-03 | 2011-10-19 | 杭州华三通信技术有限公司 | Method and device for scheduling cache |
| CN102291256A (en) * | 2011-07-25 | 2011-12-21 | 杭州华三通信技术有限公司 | Monitoring data query method and equipment |
| CN102520883A (en) * | 2011-12-12 | 2012-06-27 | 杭州华三通信技术有限公司 | Data access method and device thereof |
Cited By (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103902232A (en) * | 2012-12-28 | 2014-07-02 | 联想(北京)有限公司 | Data write-in method and device |
| CN104750773A (en) * | 2013-12-31 | 2015-07-01 | 国际商业机器公司 | Index maintenance based on a comparison of rebuild vs. update |
| US9996568B2 (en) | 2013-12-31 | 2018-06-12 | International Business Machines Corporation | Index maintenance based on a comparison of rebuild vs. update |
| CN104750773B (en) * | 2013-12-31 | 2018-11-27 | 国际商业机器公司 | The method and system of index maintenance |
| US10579608B2 (en) | 2013-12-31 | 2020-03-03 | International Business Machines Corporation | Index maintenance based on a comparison of rebuild vs. update |
| US11226948B2 (en) | 2013-12-31 | 2022-01-18 | International Business Machines Corporation | Index maintenance based on a comparison of rebuild vs. update |
| CN111143117A (en) * | 2018-11-06 | 2020-05-12 | 浙江宇视科技有限公司 | Data storage method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101727299B (en) | RAID5-orientated optimal design method for writing operation in continuous data storage | |
| CN103136074B (en) | Data storage method and data storage system of multiple disk array systems | |
| US7206899B2 (en) | Method, system, and program for managing data transfer and construction | |
| US11074129B2 (en) | Erasure coded data shards containing multiple data objects | |
| US9405625B2 (en) | Optimizing and enhancing performance for parity based storage | |
| TWI716416B (en) | Storage system, method and system for managing storage media, method of operating storage system, and non-transitory computer-readable medium | |
| US20030084397A1 (en) | Apparatus and method for a distributed raid | |
| CN105339907A (en) | Synchronous mirroring in non-volatile memory systems | |
| CN103793182A (en) | Scalable storage protection | |
| CN109814807B (en) | Data storage method and device | |
| CN102147713B (en) | Method and device for managing network storage system | |
| CN102667738A (en) | Memory system with multiple striping of RAID groups and method for performing the same | |
| US11340986B1 (en) | Host-assisted storage device error correction | |
| CN102520890B (en) | RS (Reed-Solomon) - DRAID( D redundant array of independent disk) system based on GPUs (graphic processing units) and method for controlling data of memory devices | |
| US11416166B2 (en) | Distributed function processing with estimate-based scheduler | |
| CN102164165B (en) | Management method and device for network storage system | |
| CN101984400A (en) | RAID control method, device and system | |
| CN105302492A (en) | RAID reconstruction method and device | |
| CN107729536A (en) | A kind of date storage method and device | |
| CN103605582B (en) | Erasure code storage and reconfiguration optimization method based on redirect-on-write | |
| CN102246135A (en) | Thin import for a data storage system | |
| US8954668B2 (en) | Writing of data of a first block size in a raid array that stores and mirrors data in a second block size | |
| CN102819406A (en) | Front-end data storage method and device | |
| CN109196458A (en) | Storage system active volume calculation method and device | |
| CN101073066B (en) | Method, system, and program for generating parity data |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20121212 |