WO2011105892A1

WO2011105892A1 - Wideband analog recording method an circuitry for wideband analog data recorder

Info

Publication number: WO2011105892A1
Application number: PCT/NL2010/050092
Authority: WO
Inventors: Laurens Bierens
Original assignee: Eonic B.V.
Priority date: 2010-02-24
Filing date: 2010-02-24
Publication date: 2011-09-01

Abstract

Wideband analog recording method comprising transferring high volumes of data from one or more sources (2) to one or more destinations (4). The method comprises composing data packets (5), each having a data packet header (6), a data part (7) and a data packet trailer (8). Each data packet (5) comprises data in the data part (7) originating from one single source (2) out of the one or more sources (2), and the data packet header (6) of a data packet (5) comprises an identification of the associated single source (2).

Description

Wideband analog recording method and circuitry for wideband analog data recorder

Field of the invention

The present invention relates to a wideband analog recording method comprising transferring high volumes of data from one or more sources (such as A/D converters) to one or more destinations (such as storage media). In a further aspect, the present invention relates to an analog-to-digital converter unit, a storage controller, or a wideband analog data recorder. Prior art

Wideband analog recording methods are known, in which one or more A/D converters and one or more storage mediums are present to store sampled data coming from the A/D converters.

In some prior art systems, the high speed, high data volume storage mediums are tape drives with magnetic tapes like S-VHS and DCRsi. The data streams from one or more A/D converters are combined, together with narrowband auxiliary analog and digital data like time code data and audio data.

Later, instead of magnetic tapes, computer memory have been used as storage medium. An example of a low-end wideband analog recorder is an A/D converter PCI- board in a PC, which stream sampled wideband data directly into the PC's work memory through direct memory access. When the memory is full, the data is stored in a file on the PC's hard disk. Auxiliary narrowband data is stored directly on the PC's hard disk, either in separate files or included in the data file. Such PC-based wideband analog recorders have limitations in recording time due to the limited amount of work memory, and limitations in data rates and/or number of A/D converters due to internal (PCI-bus) bandwidth limitations.

An example of a high-end wideband analog recorder is a VME-rack with one or more A/D converter VME-boards, and a memory controller VME -board. The data transfer between A/D converters and storage controllers could be through high-speed switch fabric interconnects like RACEway (ANSI/VITA 5.1-1999 (R2004)) or SKYchannel (ANSI/VITA 10-1995 (R2002)).

The storage medium could have been a disk array based on standards like Front- Panel Data Port (FPDP), Fibre Channel, SCSi. Data streams of the A/D converters are combined in one file, and auxiliary narrowband data is captured through the VME-bus and stored either in separate files or included in the data file. Such VME based recorders overcome the limitations of PC-based systems, but are expensive, large, and complex.

Current prior art wideband analog data recorders make use of integrated FPGA technology and PC-technology to overcome the above disadvantages. Multiple A/D converter boards, often with a PCI-interface, connects to an integrated switch fabric in FPGA. In this FPGA data streams are combined and formatted and transferred to one or more (often PCI-based) storage controllers, and disk arrays (often based on SATA or SAS hard disks). Still most of the auxiliary narrowband data is recorded separately and either stored as a separate file or included in the data file afterwards.

Architectures and data structures of wideband analog recorders (such as the prior art systems described above) allow more flexibility and higher data transfer rates, yet they are optimized for streaming of high volume data at high speed onto the storage medium. Streaming of data imposes certain limitations to the entire system, in view of bandwidth or throughput restrictions of the separate components in the system. Also, once data samples have been recorded, it is difficult to retrieve the data in a flexible and useable manner, which may result in restrictions on hardware that can be used to read the data.

Summary of the invention

The present invention seeks to provide an improved data transfer system and method, which is particularly suitable for high speed, high volume data transfer such as in wideband analog data recorders.

According to the present invention, a wideband analog recording method comprising transferring high volumes of data from one or more sources (e.g. channels of an A/D converter, or a storage controller) to one or more destinations (e.g. storage media, D/A card), the method further comprising: composing data packets, each having a data packet header, a data part and a data packet trailer, wherein each data packet comprises data in the data part originating from one single source out of the one or more sources, and the data packet header of a data packet comprises an identification of the associated single source. As the source identification is recorded in the data packet header, both storage and retrieval can take place very efficient. The present invention embodiments provide for a very reliable data transfer to and from one or more storage media without hick-ups due to interruptions in receiving or sending data. Furthermore, in actual implementations, data transfer systems may transfer large amounts of data using the present method, originating from various sources (i.e. different types of data, with different data speeds (high speed/low speed)). Data may be received from various sources in an asynchronous manner, and the precise time of reception may be unpredictable. The present method allows to transfer and store all these data independently in packets. Furthermore for future growth the system employing such methods is easily scalable, e.g. to accommodate future data

connections with even higher speeds.

In a further embodiment, the data packet header comprises h x 64 bits, h being an integer number, and the data packet trailer comprises 64 bits. Furthermore, the data packet header, data part and data packet trailer are all 64 bit aligned in a further embodiment. This allows to use an efficient firmware architecture, and provides very good performance.

The data packet comprises a sub- formatting structure in a further embodiment, the sub- formatting structure having a plurality of sets in the data field, each set comprising a header size field, a stuffed bit count field, a data size field, a source offset field and a data field. This allows to include snap shots of analogue signals in the data to be recorded, as well as discontinuous data blocks.

In a further embodiment, the method further comprises storing data on the destination as a file comprising a sequence of data packets, composing a data packet table data packet wherein the data part comprises a data packet table, which for each data packet in the file, comprises the data packet header and an offset in the file for the specific data packet, storing the data packet table data packet at the end of a file. This prevents the need to browse through entire file to find a specific data packet of a specific source, as all the information needed is present in the file.

The method may further comprise composing a file header data packet wherein the data part comprises a file header, wherein the file header comprises an offset value in the file in bytes of the data packet table data packet, storing the file header data packet at the start of a file. This is possible as the size of such a packet is known, and can be reserved. Furthermore it allows to find the data packet table, and from that a first data packet with actual data. In a further embodiment, the method further comprises recording data in a ring buffer mode. This allows to e.g. monitor data continuously, and start recording of data just before an event. A ring buffer may be part of one of the sources, e.g. an A/D converter or a storage controller.

In a further aspect, the present invention relates to an analog-to-digital converter unit comprising one or more channel inputs associated with a source of data, comprising data circuitry arranged to compose data packets, each having a data packet header, a data part and a data packet trailer, wherein each data packet comprises data in the data part originating from the a single source, and the data packet header of a data packet comprises an identification of the associated single source. The data circuitry may be further arranged to implement the present method embodiments.

In an even further aspect, the present invention relates to a storage controller for transferring high volumes of data from one or more sources to one or more

destinations, the storage controller comprising controller circuitry arranged to store data on the destination as a file comprising a sequence of data packets, to compose a data packet table data packet wherein the data part comprises a data packet table, which for each data packet in the file, comprises the data packet header and an offset in the file for the specific data packet, and to store the data packet table data packet at the end of a file. Furthermore, the controller circuitry may be arranged to implement the present method embodiments.

Furthermore, in an even further aspect, the present invention relates to a wide band analog data recorder for transferring high volumes of data from one or more sources to one or more destinations, comprising one or more analog-to-digital converter units connected to one or more sources of data, one or more destinations for the data, and one or more storage controllers, each of the one or more analog-to-digital converter units having one or two output channels, each of the output channels being connected to a different one of the one or more storage controllers, and each storage controller being connected to an associated destination, the one or more analog-to-digital converter units being arranged to execute the associated method embodiments, and each of the one or more storage controllers being arranged to execute the associated method embodiments. The wide band analog data recorder may further comprise a global storage controller in communication with each of the storage controllers, the global storage controller being arranged to manage control data of the wide band analog data recorder. Furthermore, the wide band analog data recorder may comprise two or more storage controllers each having an associated storage medium, wherein one of the one or more sources of data is connected to the two or more storage controllers, and is arranged to send data packets to the two or more storage controllers in a round robin manner.

Short description of drawings

The present invention will be discussed in more detail below, using a number of exemplary embodiments, with reference to the attached drawings, in which

Fig. la shows a schematic diagram of a typical set-up of a part of a wideband analog data recorder;

Fig. lb shows a schematic diagram of a further part of the wideband analog data recorder of Fig. la;

Fig. 2 shows a structure of a data packet as used in embodiments of the present invention;

Fig. 3 shows a schematic diagram of an analog-to-digital converter according to an embodiment of the present invention;

Fig. 4 shows a structural view of the contents of a data packet header of the data packet of Fig. 2;

Fig. 5 shows a typical application of an embodiment of the present invention, wherein parts of a signal of a source are processed into a data packet using sub- formatting;

Fig. 6 shows a hardware setup of a data transfer system according to an embodiment of the present invention;

Fig. 7 shows the structure of a file having data packets which is composed using an embodiment of the present invention;

Fig. 8 shows a schematic view of a further embodiment of the wideband analog data recorder according to the present invention;

Fig. 9 shows a schematic view of an even further embodiment of the wideband analog data recorder of Fig. 8 and

Fig. 10 shows a schematic view of a generalized embodiment of the present invention. Detailed description of exemplary embodiments

The present application relates to providing a packet based architecture for transferring large amounts (or high volumes) of data in a data transfer system from one or more sources to one or more destinations, e.g. for wideband analog recording applications but also for (radar) image processing applications. The present invention embodiments can be implemented in various computer based systems, e.g. the data processing unit as described in patent application PCT/NL2008/050314 of the same applicant, which is incorporated herein by reference.

In Fig. la a schematic diagram is shown of a typical set-up of part of a wideband analog data recorder, in which (digital) data is generated by a plurality of analog-to- digital converter units 2 (e.g. in the form of dedicated A/D cards) connected to a storage controller 1, and data can be sent to one or more digital-to-analog converter units 3 (e.g. in the form of dedicated D/A cards, for (real-time) playback of data). As shown in Fig. lb, the storage controller 1 interfaces with the A/D and D/A converter units 2, 3 (indicated by arrows to and from the storage controller 1, e.g. in the form of a high speed serial data connections), and also with one or more storage media 4 (read and write to retrieve or store data). The storage medium or media 4 is e.g. a large disk array (Just a Bunch Of Disks, JBOD, or other configurations as customary in the field).

In specific embodiments, the connections between storage controller 1 and the

A D or D/A converter units 2, 3 are implemented using high speed serial data connections, which in a processing configuration may use a cPCI (compact peripheral component interconnect) backplane to connect the A D and D/A converter units 2, 3 to the storage controller 1.

In actual present day implementations, the hardware as used may impose certain limitations, e.g. the bandwidth on a single high speed serial data connection is limited to approximately 200 MB/Sec. Also, the used storage medium 4 may impose a limit on the bandwidth the storage controller 1 is able to handle, e.g. up to 200MB/sec.

Furthermore, in present day data transfer systems, the data is handled in streams of data, which results in a number of disadvantages for the entire data transfer system. Data originating from different A D converter units 2 are merged or split depending on the hardware connections. Data compression may be applied, which also influences the data streaming process. Further user settings may be applied, and eventually, the data is stored in a particular manner, necessitating that for retrieval of the data the same hardware (storage controller 1 and associated settings) needs to be used. While the individual A/D converter units 2 may remain generic, the result for the D/A converter unit 3 is that the firmware of the D/A convertor unit 3 needs to be equipped with sufficient un-splitters and un-mergers to be able to make all recorded channels visible on one of the D/A converter output channels. The result is that the firmware of the D/A convertor unit 3 has become dependent of the chosen architecture. All the settings of these un-splitters etc. need to be software controlled since it impossible for the D/A convertor unit 3 firmware to "understand" what data is packed and merged in the stream it receives.

The stream based systems as presently used result in a number of disadvantages, including a difficulty to expand or scale the system when needed. Furthermore, in general, the data is not self-describing, i.e. the data can only be read by the application that recorded the data files. Adding sources, like more A/D converters, generating additional data streams requires in many cases architectural adaptation and significant software and firmware modifications of the system. Adding storage media to increase the storage data rate or adding D/A converters requires major hardware revisions and update of the data structure.

According to the present invention, embodiments of a data transfer method and system are proposed wherein the data is transferred from one or more sources (e.g. an input channel of an A/D convertor unit 2) to one or more destinations (e.g. a storage medium 4, or a D/A convertor unit 3), the method comprising composing data packets 5, each having a data packet header 6, a data part 7 and a data packet trailer 8, wherein each data packet 5 comprises data in the data part 7 originating from one single source out of the one or more sources, and the data packet header 6 of a data packet 5 comprises an identification of the associated single source. Functionally similar structures and connections as used in present day streaming data systems, as discussed above with reference to Fig. la and lb, can be utilized for transferring data packets 5, with certain modifications as discussed below.

The present invention embodiments allow a data transfer system to be expanded without undue effort, and also, such a data transfer system can be scaled as needed.

The structure of the data packet as used in the present invention embodiments is shown schematically in Fig. 2. A data packet 5 comprises three parts, i.e. a data packet header 6 (all information needed to interpret the data etc.), the data 7 itself, and a data packet trailer 8, used for CRC purposes to monitor the transmission channel. The data packet header 6 precedes the data 7. After the data 7 itself, the data packet trailer 8 is sent.

The idea of this packet based architecture is that the data will not be split and merged as in prior art stream based architectures. Instead, the data will be transferred as a complete packet 5 of data. This packet 5 of data will ONLY contain samples of the same source. The packet structure will remain intact when it is transferred from a source 2 (e.g. A/D card 2 or storage medium 4 via storage controller 1) to a destination 3, 4 (e.g. storage medium 4 or D/A card 3 via storage controller 1), similar to the data paths described with reference to Fig. la and lb.

The method described above may e.g. be implemented in an A/D converter unit 2, a schematic diagram of which is shown in Fig. 3. The A/D converter unit comprises one or more channel inputs (an input interface 41 for channel 0 and a second input interface 42 for channel 1 which connect to the actual analog signal sources) and further data circuitry arranged to compose data packets as discussed above. Under control of a control unit 40, incoming data is stored in a buffer 43 for channel 0 or a second buffer 44 for channel 1. As soon as a buffer 43, 44 reaches a certain limit, the data is composed into a data packet 5 and sent out via output interface 45. In the schematic diagram of Fig. 3, the control unit 40 is furthermore connected to a memory unit 46, e.g. comprising a look-up table, which is used in composing the data packet 5 as discussed below. In further embodiments, the A/D converter unit may comprise more than one output interface 45.

To be able to do something with the data, some means to interpret the data packets 5 must be supplied. These means are provided by the data structure of the data packets 5.

The data structure defines a minimum set of information needed to correctly interpret the data in the data packet 5, or at least provide information on how to find out how it can be interpreted.

A storage controller 1 (as depicted in Fig. la or lb) will receive lots of data packets 5 from various sources, such as A/D converter unit 2 (or its input channels), or storage medium 4. To be able to handle the data packet 5 and understand to which data source the data 7 belongs to, some sort of identification of the data is needed. The packet header 6 of the data packet 5 defines this identification amongst other things.

The composition of an embodiment of a complete header 6 as provided by the A/D converter unit 2 can be found in Fig. 4, and in the following table including descriptions of the individual fields:

010 = not allowed

01 1 = record active, first packet

100 = not allowed

101 = record active, last packet

1 10 = not allowed

1 1 1 = record active, first AND last packet

Bit 3:Safe flag.

0 = this data packet can be deleted in case of a ring buffer recording.

1 = this data packet MAY NOT BE deleted in case of a ring buffer recording.

Bit 4: Overflow bit,

0 = no overflow,

1 = data overflow occurred during duration of this packet.

Bits 15....5: Reserved for future use.

1 1 CRC of header 32 bit 4 CRC of the header

The individual fields in the data packet header 6 will be discussed below, the reference numerals in parenthesis refer to Fig. 4:

Field 0: header size (11)

The header size defines the size of the header defined in bytes.

Field 1 : HP version (12)

The header version defines the version of the header and trailer.

Field 2: Sub- formatting (13)

The sub- formatting field defines the type of sub- formatting that is used. A sub- format defines the format of the data part 7 of the data packet 5.

Field 3: Data Size (14)

The data size defines the size of the data part 7 in the data packet 5 in bytes. This does not include the size of the header 6 and the trailer 8.

Field 4: Packet order number (15)

Order number of the data packet 5. Per source data ID (next field) and auto- incremented.

Field 5: trailer size (16)

The trailer size defines the size of the trailer 8 defined in bytes. See field 0, data packet size.

Field 6: StuffedBitCount (17)

For firmware architecture/performance reasons all data in the data packet 5 is 64 bit aligned. Both the header 6 and the trailer 8 are also 64 bit aligned. Since a data packet 5 can also be used to transfer a single sample (if needed), the data size could possibly be less than 64 bits. The stuffed bit count field defines the number of bits inserted after the real data in the data part 7 to let the entire data part 7 be 64 bit aligned. Note that the data size (Field 3) describes the size of the data part 7 including the stuffed bits.

Field 7: Source data ID (18)

The source data ID, defines a unique number that identifies the source of the data.

This unique number corresponds to a data ID structure where the 64-bit ID is divided into 2 parts, a subscriber ID (32-bit) and a data sub ID (32-bit).

In the data ID structure (also named Eonic Generic Library (EGL)) a list of data objects (CGiiData) is available. Through these data objects, information is available regarding the layout of the data. The data objects provide enough information to interpret the data. As shown in the table above, some special values of the Data Source ID field in the data packet header 6 were reserved for a file header 5B and a data packet table 5C (to be discussed below with reference to Fig. 7. If for some reason the data packet table 5C was not properly saved at the end of the recording, the table 5C can be rebuild when the storage controller 1 explicitly knows this information.

Field 8: Source offset = SO (19)

The source offset field the sample offset of the first sample in the data part 7 of the data packet 5. The source offset is illustrated in an example where a source sends out a total of 2500 samples in 3 data packets 5. The first packet 5 has source offset 0. The second packet 5 has source offset 1000, and the third packet 5 has source offset 2000.

Field 9: Data format (20)

The data format field defines in what format the data in the data part 7 is given. Specific bits in this field tell if the data is signed or unsigned, complex or real valued together with the size of each sample in bits (8, 10, 12 etc. ). If the data is not sampled data coming from an A/D converter, this field is set to 0x0000 and some other means of interpretation is needed.

Field 10: special (21)

The special field holds the record flags but provides space for future updates. The record flags are discussed below:

Bits 2..0: Record flags (RF)

The record flags define which packets 5 need to be stored. In a first example only one packet 5 is marked. This packet 5 is the first and the last packet (short recording, only consisting of one packet). A second example contains two marked packets 5. Packet2 is the first packet; record active is 1 and the first packet bit is 1. A third example has three marked packets. Packet2 is again the first packet with record-active 1 and first packet bit 1. Packet3 is the second packet, and only has the record active bit set to 1. Packet4 is the last packet for the recording, and has its record-active bit and the last packet bit set to 1.

Bit 3: Safe marker.

The storage controller 1 , in case of a ring buffer recording (to be discussed below), may not delete data packets 5 that have the safe marker set.

Bit 4: Overflow bit.

This bit provides information about possible bandwidth overflows occurred during the duration of the packet 5 : 0 = no overflow; 1 = data overflow occurred during duration of this packet 5.

Field 11 : CRC of the header (22)

This is the 32-bit result of a CRC algorithm performed on the data packet header. See REF01. The CRC-32-IEEE 802.3 algorithm with the standard 0x04Cl 1DB7 polynomial is used.

The trailer 8 of a data packet 5 provides space for additional information about the packet 5. The trailer 8 is, in this embodiment, used only for a CRC field but the data structure is flexible enough to enable future additions to the trailer 8 since the length of the trailer 8 is defined in the header 6.

The CRC calculation algorithm is the same as the Serial data packet CRC. The polynomial used is the de facto standard for all CRC calculations in for instance compression algorithms and Ethernet namely: 0x04Cl 1DB7.

Field 0: CRC value

This is the 32-bit result of the CRC algorithm performed on the data itself. For this CRC value, the same algorithm is used as for the header. See REFOl .

Field 1 : CRC result When a data packet 5 is transferred from source to destination, the source (e.g. A/D converter unit 2) can calculate the CRC value and insert it in the trailer 6. The destination receives the packet 5 and can do the calculation again, and verify the results against the value passed in the trailer 6. If the check failed (values not equal) the CRC bit failed bit must be set.

As is clear from the above, the data packet header 6 comprises h x 64 bits, h being an integer number (h=5 in the above example), and the data packet trailer 8 comprises 64 bits. These sizes allow a sufficiently flexible method and system, and at the same time also sufficient efficiency is provided.

In further embodiments, the fields of the data packet header 6 and trailer 8 may be defined in a different order, specific fields may be left out, and other field may be added. However, in all embodiments the 64-bit alignment will be maintained. A 64-bit alignment ensures compatibility with the associated hardware implementations in combination with high speed data transfer and throughput capabilities. In a further embodiment, the elements of data packet 5 may also be 128-bit aligned.

When additional information must be transferred to correctly describe the data transferred in a data packet 5, sub- formatting can be used, as illustrated in an exemplary form in Fig. 5. At the top of Fig. 5 a signal is represented as input to an A/D converter unit 2, and only the blocks as indicated (with size 1 and size 2) are sampled and included in the data packet 5. As discussed above, the Source Offset field in the packet header 6 defines the sample number of the first sample in the data part 7 of the data packet 5. When multiple non continuous blocks of data must be transferred, all individual blocks should be transferred in their own data packet 5. This will increase the number of data packets 5 transferred during a recording.

Two variations of non continuous data blocks can be identified, i.e. snapshots

(e.g. x samples every y seconds) and A/D converters 2 that only transfer data when data is "detected" (e.g. the signal indicated in the top of Fig. 5). By making use of a sub-format, multiple of these blocks can be transferred within a single data packet 5, i.e. the sub- formatting structure has a plurality of sets in the data field 7, each set comprising a header size field, a stuffed bit count field, a data size field, a source offset field and a data field. This format provides the additional SO fields and size fields. Below the signal graph in Fig. 5, the data packet 5 is shown, and in the original data part 7, additional sub-header data is included, comprising sub-header size HS (7a; 7f); Stuffed Bit count (7b; 7g); sub data size DS (7c; 7h); source offset (7d; 7i), as well as the data itself (7e; 7j). Note that while the data blocks for detected data have variable sizes, the same format can be used as in the case of equally sized data blocks

(snapshots).

The resulting effect of the sub-format is that during read back of a data packet 5 the main header (the packet header 6, not the headers 7a-7d; 7f-7i of the data block 7) does not completely describe the time-window described by the data in the data part 7 of the data packet 5.

In further embodiments, the data transfer method further comprises storing data from the one or more sources on the destination 4 as a file 25 comprising a sequence of data packets 5 (5A, 5B, 5C). These embodiments are e.g. implemented in controlling circuitry of the storage controller 1. In Fig. 6, a hardware setup of a data transfer system according to the present invention is shown, which is built up from the following components: an A/D converter 2 having two input ports 9 and an output channel connected to a storage controller 1, which in its turn interfaces with a storage medium 4.

The A/D converter 2 does the sampling of the data and generates data packets 5.

The data packets 5 have the form as discussed above with reference to the figures 2-5.

These data packets 5 are sent out to the storage controller 1. The storage controller is able to receive the data packets 5 that were generated by the A/D converter 2 and to store them on the storage medium 4.

In Fig. 7, the structure of a data file 25 that is saved to the storage medium 4 is shown graphically, comprising a number of 'normal' data packets 5A, a file header data packet 5B and a data packet table data packet 5C.

To be able to retrieve individual data packets 5 A that were saved to file 25, some formatting must take place when writing the file 25. The storage controller 1 takes care of the formatting of the file 25. The format in which the data on the medium 4 is stored is called the file format.

When all data packets 5 are saved to a file 25 in the order in which they are received by the storage controller 1, one big file 25 of data packets 5 is created. If for whatever reason a packet 5 is needed that was saved somewhere in the middle of the file 25, the storage controller 1 would have to open the file 25 and start reading from the beginning and search through all saved packets 5 in the search for the packet 5 that is needed. This is a slow process since all data packet headers 6 should be read, parsed and evaluated. To overcome this problem, a list of data packets 5 is created inside the storage controller 1. The entries in the list contain the data packet header together with the offset in the file where the data packet can be found. This table is called the data packet table 26, and is also structured into a data packet table data packet 5C.

The entire data packet table 26 is stored as the data packet table data packet 5C at the end of a recording. The data packet table 26 should contain references to all data packets 5 in the file 25. Since the table 26 can only be completed when all data packets 5 are stored, this data packet table 26 can only be stored at the end of the file 25.

As discussed, the data packet table 26 will be used to quickly browse through the entire file 25; indexing in a table is much faster than browsing through a file 25 with all disk accesses involved. If a recording is analyzed and a packet 5 in the middle of the file 25 is needed, the data packet table 26 is needed to quickly locate the packet 5 in the file 25. To find the data packet table 26 one would again need to browse through the complete file 25. To overcome this problem, another header is used, called the File Header 27. In this header 27, the location of the data segment header table 26, i.e. the last data packet 5, is stored, as part of the file header data packet 5C.

Since the data packet table 26 and the file header 27 are also data blocks, they can be handled as any other data packet 5 and even be saved as such. The data part 7 of the packet 5B; 5C consists of the file header 27 and data segment header table 26, respectively.

As discussed, for every data packet 5 that is stored in the file 25 an entry is created in the data packet table 26. Entries in this table 26 comprise a complete copy of the data packet header 6 of the associated data packet 5 and the file offset. This file offset describes an offset of where the packet 5 can be found within the file 25 on the storage medium 4.

Field 0: File offset

This is the offset of the data segment within the file 25 of on the storage unit Field 1 : Data segment header

This is a complete copy of the data packet header 6 as discussed above. To be able to quickly start reading the correct data packets 5, it should be simple to also quickly start reading the data packet table 26. The location of the data packet table 26, amongst other things, is put in the file header 27.

The way the data is written to the storage unit 4 or other media is defined by the file format. The data structure of a data packet 5 contains the data packet header 6 (DPH), the data 7 itself and a data packet trailer 8 (DPT) as shown in Fig. 2. Again it is noted that the file 25 can accommodate data packets 5 of varying sizes.

A commonly known hard disk and other similar storage media 4 can only store data in blocks. These blocks have a certain minimum size. This is called the alignment size. This means that when for instance 64 bytes should be written to disk, in fact alignment bytes are saved. In most cases the alignment is 512 bytes. This number is defined by the storage medium 4 used during the recording. If a data packet 5 is smaller than 512 bytes, the packet 5 must be aligned to the 512 bytes. For this, field 7 of the packet header 6 (bit stuff count) can be used to increase the number of stuffed bits stuffed in the data part 7. Of course, the CRC in the trailer 6 must be re-calculated since the data field 7 is modified.

In Fig. 8, an embodiment is shown of a general structure for a very flexible data transfer system. The system comprises two A/D converters 2, two local storage controllers 1 (which correspond to the storage controller 1 embodiments as discussed above), two storage media 4, and a global storage controller 10. The storage media 4 are connected with an associated local storage controller 1. The A/D converters 2 have two output channels, which are connected to different ones of the local storage controllers 1. In this system, each A D converter 2 sends out data packets 5 to both output channels. Which storage controller 1 stores which packet 5, is completely depending on the algorithm used on the A/D converter 2 that decides which output channel is used. In one embodiment, the global storage controller 10 is left out from the schematic view of Fig. 8. If the top A/D converter 2 has two data sources with ID 1 and 2, and the second A/D converter 2 has two data sources with ID 3 and 4, and a recording is done where each ID has two packets 5 sent during the recording, the files 25 will look like:

FH - 2i - 4i - 2₂ - 4₂ - DPT

where the suffix indicates the order number of the data from the specific source. To be able to retrieve a specific part of a specific packet 5 that is needed to be analyzed, some sort of global storage controller is present that somehow "knows" where all packets reside in the system, i.e. which is able to manage control data of the data storage system. For this a global storage controller 10 is used as shown in Fig. 8. The task of the global storage controller 10 is to administer all data packets 5 in the entire system and provide generic information to the local storage controllers 1. In this embodiment, a global storage controller 10 will be present, even if the system is equipped with only one storage controller 1.

If one A/D converter 2, one of the storage controllers 1 and its associated storage medium 4 are removed from the system as shown in Fig. 8, the system as shown in and discussed with reference to Fig. 6 is obtained again.

If the bandwidth needed by the A/D converter 2 is larger than either the output channel or the storage controller 1 can handle, the following setup can be used, wherein two local storage controllers 1 with associated storage medium 4 are present, and each of two output channels of the A/D converter 2 are connected to different storage controllers 1. In this setup, the A/D converter 2 divides the generated bandwidth over the two outgoing connections. In general, an A/D converter 2 can divide it's output over x storage controllers 1, x being an integer number. A simple algorithm would be to send packets 5 out to alternating storage controllers 1. Both local storage controllers 1 will create a file at the start of the recording, create the file header packet 5B, store all incoming packets 5, write the data packet table 5C and close the file. Except in this case, only half the number of the data packets 5 are stored on the top storage controller 1 and the other half is stored on the bottom storage controller 1. A total list of data packets 5 is now formed by the combined list of the two storage controllers 1.

As discussed above the global storage controller 10 has knowledge about all local storage controllers 1 in the system. If one of the data packets 5 of the recording is needed, the global storage controller 10 can retrieve the data packet tables 26 from both storage controllers 1 , possibly sort the list by Source data ID to speed up the lookup process, and lookup which storage controller 1 has the packet 5 saved and retrieve it from there.

If the A/D converter 2 generates the same number of data packets 5 and the same number of Source data IDs as in the embodiment discussed above, a block diagram of the two files 25 would look like:

FH - 11 - 1₂ - DPT

FH - 2i - 2₂ - DPT

Two files 25 were created by two different storage controllers 1. In the example, data packets 5 were sent out to alternating storage controllers 1. The first data packet 5 (l i) was send to the top storage controller 1, the next (2i) to the bottom storage controller 1 etc. The algorithm of sending packets 5 to alternating storage controllers 1 does take the changing packet sizes into account. So in order to gradually fill the storage media 4 the size of packets 5 is also considered in a further embodiment.

If the bandwidth needed by the A/D converter 2 is much smaller than either the output channels bandwidth or the bandwidth the storage controller 1 can handle, a setup can be used where two A/D converter 2 are connected to a single local storage controller 1. Again, at the start of a recording the local storage controller 1 creates a file 25 and writes the first data packet 5 containing the initial file header 27. Then both A/D converters 2 start generating their own data packets 5. All packets 5 are stored and at the end the packet table 26 is added. If the top A/D converter 2 has two data sources with ID 1 and 2, and the second A/D converter 2 has two data sources with ID 3 and 4 the file 25 will look like:

In this recording, four data sources were saved with only one packet 5 per data source.

Signal recorders are commonly used for signal monitoring. When a recorder is inactive, and something interesting happens that should be recorded, the recorder can be started. Chances are the signal has already disappeared. To overcome this problem recorders are equipped with a ring buffer recording mode. This ring buffer mode, specifically the compatibility with packet based recorders utilizing embodiments of the present invention, is discussed below. A ring buffer recording in the prior art stream based systems is started with a <duration> setting. At the moment this recording is started, a buffer is allocated on the storage medium 4. The size of the allocated buffer is large enough to hold all samples of the stream for the <duration> time. If the recording is kept running for a longer time, the buffer is overwritten with newer data, the oldest data is overwritten first.

The resulting effect is, that when a recording is stopped at some time (t_e) an operator can analyze the signals from

• t_e - duration until t_e if it is a complete ring buffer recording

• t_e - recording time until t_e if it is an incomplete ring buffer recording

In the prior art stream based system, a model of the entire system is available that

"knows" the data rates of the streams. The storage controllers need this information to calculate the ring buffer size.

In the packet based system according to the present invention embodiments, also the storage media 4 are used, and such a model of at least the estimated bandwidth that each storage controller 1 can expect is available. It is the task of the global storage controller 10 to let all local storage controllers 1 know what bandwidth they can expect during the recording. The model must take into account what type of algorithm is used by the A/D converters 2 to divide the bandwidth on the outgoing connections to be able to make this bandwidth assessment.

One of the nice features of the packet based system according to the present invention embodiments is that the packet structure of the data packets 5 is always kept intact. This means that the order of the packet 5 is always defined by first a data packet header 6, then a data part 7 and finally a data packet trailer 8 (see Fig. 2).

During a ring buffer recording, it is possible that at some time, the remaining disk space within the allocated buffer is not sufficient to store an incoming packet 5. The simplest algorithm could start writing the data packet 5 at the start of the remaining space and continue writing the data packet 5 at the start of the allocated space.

However, this breaking up of the data packet 5 is not allowed.

In a normal recording, there is a chronological order of all packets on the storage medium 4 and during the recording no packets 5 are removed. During a ring buffer recording this chronological order of data packets 5 cannot be guaranteed. This imposes no problem since the information in the data packet header 6 (specifically the source offset 19, see Fig. 4) can be used to order the data packets 5. Furthermore, if the allocated ring buffer space is full, some of the oldest data packets 5 must be removed to be able to store a new data packet 5 inside the ring buffer space.

This removing of data packets 5 introduces some issues:

1. If a data packet 5 is removed to be able to store a new packet 5 it is possible that this results in a file 25 on the storage medium 4 with a gap inside. This gap is created when a large data packet 5 is removed to store a smaller data packet 5. When the recorder suffers from a power loss just before the data packet 5C with the packet table 26 (see Fig. 7) is saved, it should be possible to reconstruct the data packet table 26 to still be able to use the file 25. If a file 25 is opened, the first packets 5 which are adjacent can be read correctly. The problem is to be able to a further packet 5 which is located after the gap created by removing the large data packet 5.

2. Some of a the data packets 5 should never be deleted. Data packets like the file header data packet 5B or the packets 5 containing information about the data itself are examples of this.

3. It should not be possible that the algorithm that deletes the oldest data packets

5, deletes packets 5 that still (partly) belong to the ring buffer time interval.

To be able to determine if it is allowed to delete a packet 5 in the ring buffer recording mode, the SAFE flag is used in the special field 21 of the data packet header 6 (see Fig. 4 and the associated table above). During a ring buffer recording there is the data packet table 26 that is created in the storage controller 1 just as any other recording. The entries in the data packet table 26 contain the full copies of the data packet header 6 of the associated data packets 5 (see Fig. 7 and associated table above). In the data packet header 6 the SAFE flag is available for the algorithm to determine if the packet 5 can be deleted.

To be able to determine if a packet contains data that lays within the ring buffer duration, each storage controller 1 needs the following information:

• Data rate of a data source.

• Information from the data packet header 6 (source offset 19)

• Current time

· Ring buffer duration

With this information, the algorithm is able to decide if a packet 5 (partly) does or does not contains data that still belongs to the ring buffer interval. A ring buffer implementation using the data packets 5 of the present

embodiments provides a simple solution, as the storage controllers 1 can determine which of the older data packets 5 can be removed. Advanced algorithms can be used to determine in what manner data packets 5 are written to the associated storage medium 4 to obtain a very efficient solution (e.g. by determining how data packets 5 can be written most efficiently to the physical discs of a storage medium 4, such as minimizing magnetic head movement).

Another embodiment of an implementation of the ring buffer is the one where a user defines which parts of the recording should be kept. This can be implemented using the diagram representation as shown in Fig. 9. In a graphical user interface (GUI) 28 plots are available that show the signals over longer periods of time (at least the ring buffer duration). In this plot, the user is enabled to make selections that define which parts of the recording should be kept. The selection that is made in the plot is done in time. For a single selection this results in a start time stamp and a stop time stamp. These timestamps are converted to sample offsets in a conversion unit 29. The sample offsets are provided to the global storage controller 10 that provides them to the local storage controllers 1. The local storage controllers 1 must then check all data packets 5 listed in the table to see if they should be kept. If so, they must be marked safe.

During the recording, if multiple selections are kept, the remaining space in the allocated ring buffer space becomes smaller with every new selection that is kept. This will resolve in faster cycling through the ring buffer and thus the time the interval the user can select specific parts of the recording becomes ever smaller. Some feedback to the GUI 28 must be made available for the user. This is an estimation based on the average bandwidth the storage controller 1 could expect at the start of the recording (on which the size of the allocated buffer was based in the first place) and the number and total size of the packets 5 that are now marked safe.

The implementation of this ring buffer method is similar as the embodiment described above, the only difference is that the local storage controllers 1 receive the SAFE information from the global storage controller 10. Each local storage controller 1 must provide the data availability marker to the global storage controller 10 so it can provide the minimum value to the GUI 28.

It is noted that the local storage controller 1 provides more than just recording capabilities. It also facilitates read back of data during analysis, importing and exporting functions, and on - and offline playback functions. To coordinate the routing of packets inside the local storage controllers 1 , a routing table is needed. The proposed routing table is generic enough that it can be used inside ADC and DAC carriers. This routing table is shown in the following table.

Based on entries in this table, the routing of a packet 5 is done by the local storage controller 1. The Source ID of the packet 5 is checked against entries in the table and the corresponding output interface is used to send out the data.

A more generalized schematic view of a wideband data recorder according to an embodiment of the present invention is shown in Fig. 10. This embodiment is a generalization of the embodiment shown in Fig. 8, and comprises M sources of data 2, N local storage controllers 1 (and N storage media 4). Each source 2 is connected to all N local storage controllers 1, enabling the source 1 (e.g. an A/D conversion unit) to determine to which local storage controller 1 and associated storage medium 4 a finished data packet from that source 2 is sent.

As described in some of the embodiments above, an A/D converter 2 generates a high-speed high volume stream of digital data and stores them in a buffer 43, 44. When the buffer 43, 44 is full, the data is packed and sent out to a storage controller 1 as a "packet" 5. This process is generally referred to as "recording".

Multiple A/D converters 2 can connect to a storage controller 1. Data from these A/D converters 2 are stored in separate buffers 43, 44 and when either one is full then this data is packed and sent out to the storage controller 1 as a packet.

The data structure of a packet consists of three parts: Data packet header, including all information to interpret the data; Data itself; Data packet trailer, used for CRC purposes to monitor the transmission channel.

The storage controller 1 collects packets of all A/D converters 2 that are connected to it, and stores them in one file 25 on the storage medium 4, as discussed in the embodiments above. The storage controller 1 keeps a "data packet table" 26, with among others the location of each specific packet in the file 25. The entire data packet table 26 is the last packet to be stored at the end of the file 25 upon termination of the recording, as discussed in detail above. The first packet of the file contains the "file header" 27, in which file information (such as the location of the data packet table 26) is kept.

In the generalized embodiment shown in Fig. 10, apart from a number of storage controllers 1 who administer the storage medium, a "global storage controller" 10 is provided who administers the division of the packets 5 over multiple storage media 4.

Upon termination of a recording using multiple storage media 4, the data packets 5 are divided over these media 4. The global storage controller 10 is able to reconstruct the recording from the individual data packet tables 26 of the files 25 that belong to this specific recording.

The main advantages of the "packet" data structure is that it is easy to define auxiliary data sources, like metric data, text messages, audio files, time code signals, etc., in conjunction with (and often related to) the A/D converters 2 and treat them similar to the A/D converter data. In general the A/D converter source 2 generates many large packets 5 per second, whereas other "slow" sources generate small packets 5 at a low rate. Sources 2 send the packets 5 to the storage controllers 1 in a "round robin" fashion in an embodiment, so that all storage media 4 are equally filled with packets of all sources 2. This makes it very easy to add new sources 2 and additional storage media 4 without bothering about the architecture and data structure.

All information to reconstruct the data streams from the files 25, as well as their relation, is kept in the header of the packets 5, and this is fully self describing. To speed up the reconstruction upon opening and reading the files 25, the data packet table 26 kept by the global storage controller 10 is also stored in the file 25. The entries of the data packet table 26 comprise the packet header content and the location of the packet 5 in the file 25 as discussed above.

The method of reconstruction of one or more data streams from sources 2 can be used, for example, to playback a signal generated by an A/D converter 2 on a D/A converter 3. Specific packets 5 corresponding to this specific A/D converter 2 are read by the storage controllers 10, 1 and send in the right order (indicated by the data packet table 26) to the D/A converter 3.

Claims

1. Wideband analog recording method comprising transferring high volumes of data from one or more sources (2) to one or more destinations (4), the method further comprising:

composing data packets (5), each having a data packet header (6), a data part (7) and a data packet trailer (8), wherein each data packet (5) comprises data in the data part (7) originating from one single source (2) out of the one or more sources (2), and the data packet header (6) of a data packet (5) comprises an identification of the associated single source (2).

2. Wideband analog recording method according to claim 1, wherein the data packet header (6) comprises h x 64 bits, h being an integer number, and the data packet trailer (8) comprises 64 bits.

3. Wideband analog recording method according to claim 1 or 2, wherein the data packet header (6), data part (7) and data packet trailer (8) are all 64 bit aligned.

4. Wideband analog recording method according to claim 1, 2 or 3, wherein the data packet (5) comprises a sub- formatting structure (7a- 7j), the sub- formatting structure

(7a- 7j) having a plurality of sets in the data field (7), each set comprising a header size field (7a; 7f), a stuffed bit count field (7b; 7g), a data size field (7c; 7h), a source offset field (7d; 7i) and a data field (7e; 7j).

5. Wideband analog recording method according to any one of claims 1-4, further comprising

storing data on the destination (4) as a file (25) comprising a sequence of data packets (5A), composing a data packet table data packet (5C) wherein the data part (7) comprises a data packet table (26), which for each data packet (5) in the file (25), comprises the data packet header (6) and an offset in the file (25) for the specific data packet (5),

storing the data packet table data packet (5C) at the end of a file (25).

6. Wideband analog recording method according to any one of claim 1-5, further comprising:

composing a file header data packet (5B) wherein the data part (7) comprises a file header (27), wherein the file header (27) comprises an offset value in the file (25) in bytes of the data packet table data packet (5C),

storing the file header data packet (5B) at the start of the file (25).

7 Wideband analog recording method according to any one of claims 1-6, further comprising recording data in a ring buffer mode.

8. Analog-to-digital converter unit comprising one or more channel inputs associated with a source of data, comprising data circuitry (40-46) arranged to compose data packets (5), each having a data packet header (6), a data part (7) and a data packet trailer (8), wherein each data packet (5) comprises data in the data part (7) originating from the a single source (2), and the data packet header (6) of a data packet (5) comprises an identification of the associated single source (2).

9. Analog-to-digital converter unit according to claim 8, wherein the data circuitry (40-46) is further arranged to implement the method according to claim 2, 3 or 4.

10. Storage controller for transferring high volumes of data from one or more sources (2) to one or more destinations (4), the storage controller (1) comprising controller circuitry arranged to store data on the destination as a file (25) comprising a sequence of data packets (5), to compose a data packet table data packet (5C) wherein the data part (7) comprises a data packet table (26), which for each data packet (5) in the file (25), comprises the data packet header (6) and an offset in the file (25) for the specific data packet (5), and to store the data packet table data packet (5C) at the end of the file (25).

11. Storage controller according to claim 10, wherein the controller circuitry is further arranged to implement the method according to any one of claims 5-7.

12. Wide band analog data recorder for transferring high volumes of data from one or more sources (2) to one or more destinations (4),

comprising one or more analog-to-digital converter units (2) connected to one or more sources of data, one or more destinations (4) for the data, and one or more storage controllers (1),

each of the one or more analog-to-digital converter units (2) having one or two output channels,

each of the output channels being connected to a different one of the one or more storage controllers (1),

and each storage controller (1) being connected to an associated destination (4), the one or more analog-to-digital converter units (2) being arranged to execute the method according to any one of claims 1-4, and each of the one or more storage controllers (1) being arranged to execute the method according to any one of claims 5- 7.

13. Wide band analog data recorder according to claim 12, further comprising a global storage controller (10) in communication with each of the storage controllers (1), the global storage controller (10) being arranged to manage control data of the wide band analog data recorder.

14. Wide band analog data recorder according to claim 12 or 13, comprising two or more storage controllers (1) each having an associated storage medium (4), wherein one of the one or more sources of data (2) is connected to the two or more storage controllers (1), and is arranged to send data packets (5) to the two or more storage controllers (1) in a round robin manner.