CN102662639A - Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method - Google Patents
Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method Download PDFInfo
- Publication number
- CN102662639A CN102662639A CN2012101028344A CN201210102834A CN102662639A CN 102662639 A CN102662639 A CN 102662639A CN 2012101028344 A CN2012101028344 A CN 2012101028344A CN 201210102834 A CN201210102834 A CN 201210102834A CN 102662639 A CN102662639 A CN 102662639A
- Authority
- CN
- China
- Prior art keywords
- gpu
- task
- mapreduce
- cpu
- computing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method, which belongs to the application field of computer software. Corresponding to single-layer parallel architecture of common high-performance GPU computing and MapReduce parallel computing, a programming model adopts a double-layer GPU and MapReduce parallel architecture to help a developer simplify the program model and multiplex the existing concurrent codes through a MapReduce program model with cloud computing concurrent computation by combining the structure characteristic of a GPU plus CPU (Central Processing Unit) heterogeneous system, thus reducing the programming complexity, having certain system disaster tolerance capacity and reducing the dependency of equipment. According to the computing method provided by the invention, the GPU plus MapReduce double concurrent mode can be used in a cloud computing platform or a common distributive computing system so as to realize concurrent processing of MapReduce tasks on a plurality of GPU cards.
Description
Technical field
The present invention relates to the collaborative computing method of a kind of many GPU, affiliated computer software application field based on Mapreduce.
Background technology
In recent years under hardware technology promotes; The computing power and the programmability of graphic process unit (GPU) are developed fast; Characteristics with highly-parallel calculating make GPU no longer be confined to daily graphics process task, and beginning to relate to more widely, high performance universal calculates (GPGPU) field.Because GPU has high performance multi-processor array and high bandwidth, hides the video memory system that postpones, this makes in the computing application of a large amount of repeating data set operations and intensive memory access, and GPU has than traditional C PU and has more advantage.And the calculating of frequently-used data and program all need accomplish by CPU, as far as the user, if the too much time that takies CPU can make the user move large program or when comparing big data computation the sensation computing machine very slow, reduce system performance; GPU is mainly used in the calculating of recreation or figure in domestic consumer, free time, therefore appropriateness used GPU can bring good effect, has promptly reduced the holding time of CPU, and the GPU that is in the excessive free time is fully used obviously more than CPU.
The same focus that obtains paying close attention in the high performance parallel computation field is a MapReduce mass data processing framework; Our the large-scale data computing power that can be had only expensive large server just to have in the past of common computer cluster through cheapness, and better at aspects such as stability and extendabilities than the latter.The MapReduce model is applied to aspects such as astronomical information calculations processing, the inventory analysis of magnanimity case, virus base storage, network retrieval service now, solves the contradiction between the growth of data explosion formula and Computer Storage ability and the computing power deficiency.
Up to the present much the research of this two aspect is all limited in some aspects; As assist CPU to algorithm with program is quickened and carry out Distributed Calculation by the cluster that many computing machine GPU form to single computer GPU; Undeniable these aspects all obtain sizable progress, but deficiency also exists.In the face of the increasing recreation of scale during with program single computer GPU acceleration can not bring how big change, the contradiction between the growth of data magnanimity and the computing power of computing machine can not be resolved; Though same common distributed computer GPU cluster is all well and good aspect computing power; But in case node failure or other problem occur; The performance of whole cluster can receive very big influence: need frequent CPU to calculate when also having the MapReduce model to carry out Map and Reduce operation; Sometimes in addition CPU usage be absolutely, the therefore also very necessary participation of using GPU comes the computing power of balanced system.
Summary of the invention
Because the MapReduce model needs frequent CPU to calculate when carrying out Map and Reduce operation, during in the face of a large amount of parallel computation task, CPU usage even reach a hundred per cent.And GPU has than better data width of CPU and computation capability, and appropriateness is used GPU, has promptly reduced the holding time of CPU, can use the participation of GPU to come the computing power of balanced system again.
The objective of the invention is to combine the different advantages of GPU technology and MapReduce technology; On the basis of MapReduce multiple programming; The participation of use GPU comes the computing power of balanced system, and a kind of method of supporting large-scale distributed parallel computation through GPU calculating and MapReduce technological incorporation programming model finally is provided.
The collaborative computing method of a kind of many GPU based on Mapreduce of the present invention comprise following step:
1) at first, client is to the request of management phase transmission tasks.
2) then, the name node NameNode in the management phase is in charge of NameSpace, the calculation stages cluster configuration information of file system, the information such as position of storage block; Work shadow device JobTracker is responsible for calculation task is started and dispatches, and can realize the state of the implementation status and the calculation stages of tracing task.
3) in calculation stages:
1. after back end DataNode receives the read-write requests from name node NameNode; Call CPU; Mass data is read scanning, and horizontal division again is divided into the data subset splits fragment of M fixed size; M is a natural number, and its size is that number and data results according to computing node in the computing system determines usually;
2. the task tracker TaskTracker of idle CPU formats M data subclass split to work shadow device JobTracker request task and after meeting with a response, and further resolves into a collection of key/value to < key1, value1 >;
3. the task tracker TaskTracke of idle GPU is to work shadow device JobTracker request task and after meeting with a response, and each the data subset split to input creates a Map task; With each the record < key1 among the corresponding split; Value1>to the line scanning of going forward side by side as input, and it is formatd to the GPU special algorithm, use the CUDA storehouse of GPU to realize a local combiner Combiner; < key2, value2>key/value is right in the middle of producing and exporting;
4. utilize middle key/value that subregion function hash (key) mod R produces the Map function to being divided into R different partition areas, the R here is a natural number less than M.Then GPU carries out relevance ranking with intermediate result according to key2; And the value2 data aggregate that the key2 value is identical forms a new tabulation together; It is right to form < key2, list (value2) >, and the list here (value2) is the array that value2 formed by identical key2 value.Again with these key/values to being divided into R different partition areas, each division is fitted on the Reduce task of appointment
5. the workstation that has been assigned with the Reduce task calls idle CPU, and the task tracker TaskTracker that starts CPU goes to read the data < key2, list (value2)>that the Map function is submitted to.Behind the intermediate data after the traversal ordering; The task tracker TaskTracker of CPU passes to each subregion the task tracker TaskTracker of idle GPU; Format by it; Use the GPU concurrent technique to do the handled operation, obtain a plurality of output results of Reduce task, start union operation and obtain final output valve.
6. the task tracker TaskTracker of GPU gives CPU with net result and calls part, and so far, a MapReduce technical process is accomplished.
The invention has the beneficial effects as follows
Programming model of the present invention combines the two advantage of GPU general-purpose computations and MapReduce model; Research and realize a complete high performance parallel computation system is that hardware cooperates and to carry out large-scale data based on MapReduce parallel computational model software and handle with GPU.
Nowadays, aspect high-performance calculation, the independent utility of GPU technology and cloud computing technology is ripe relatively, and the instantiation of plurality of applications is arranged.Realize using GPU to cooperate the generic structure of CPU calculating, utilize the data structure of mass data, make the program part of data and parallel computation on GPU, store and move.This framework forms a level of abstraction between graphic hardware and application program, have very strong versatility, reduces communicating by letter of GPU and CPU to greatest extent, improves the computing power of entire system and improves system performance.
But this two big technology is merged, enable to carry out the parallel computation of dual level, further promote efficient.Through to research based on the distributed computing framework of MapReduce, proposed the Distributed Calculation based on GPU is improved, can utilize existing computing equipment to reach higher parallel speed through this improvement.
Description of drawings
Fig. 1 is based on the MapReduce parallel computational model hardware topology structural drawing of GPGPU;
Fig. 2 is based on the MapReduce parallel computation building-block of logic of CPU;
Fig. 3 is based on the MapReduce parallel computational model building-block of logic of GPGPU;
Fig. 4 is based on the MapReduce parallel computational model TaskTracker calculation task module map of GPGPU.
Embodiment
Below in conjunction with accompanying drawing and embodiment the programming flow process that has added GPU concurrent technique MapReduce framework afterwards of the present invention is elaborated:
The visual description that the described model hardware topology diagram of Fig. 1 is the real hardware platform mainly is made up of common computer, 100 m ethernet switch and the link of the circuit between them.Usually other ripe MapReduce models also have by rack-mount server replacement common computer; Perhaps extract file system module and carry out the storage of data, make the computing node cluster only be used for multiple parallel computing platform patterns such as calculating by independent equipment.
Shown in Figure 2 is realization Hadoop platform based on the MapReduce parallel computational model system of CPU, and Hadoop is not one and is used to the distributed file system of storing merely. but a framework that is designed with bundle execution Distributed Application on the large-scale cluster of not forming fully by common hardware with.Hadoop comprises two parts: the big formula file system of a branch HDFS (Hadoop Distributed File System) and real the sloughing off of MapReduce.Therefore, the target of Hadoop is that the MapReduce module that we will improve Hadoop realizes our MapReduce parallel computational model based on GPGPU for the exploitation Distributed Application provides a framework.
The Hadoop platform logically is divided into three layers, and soil will be through carrying out communication based on ICP/IP protocol between layer and layer.Ground floor is a client tier, and client is transmitted the calculation task request to management node, after calculation task is accomplished by management node to the client return results.The second layer is the management node layer, and management node has two parts: name node NameNode is the supvr in the model system, mainly is in charge of NameSpace, the computing node cluster configuration information of file system, the information such as position of storage block; Work shadow device JobTracker is responsible for calculation task is started and dispatch, and the state of ability real-time follow-up task executions situation and computing node.The 3rd layer is the computing node layer, and this layer also is divided into two parts: back end DataNode is responsible for handling the read-write requests from name node NameNode, can also carry out simultaneously data block establishment, delete and duplicate; The operation that task tracker TaskTracker then is responsible for to work shadow device JobTracker request task and starts this calculation task after obtaining task at computing node.
Fig. 3 will provide our the MapReduce parallel computational model building-block of logic based on GPGPU; As can be seen from the figure first and second layers with as broad as long based on the model of CPU; And difference is that mainly task tracker TaskTracker partly is divided into two in the 3rd layer of computing node; One is the CPU calling module, and another piece is the GPU computing module, can the modular design in Fig. 4 provide detailed explanation.
Though name node NameNode and work shadow device JobTracker module can adopt GPU to calculate the performance with balance The model system among the management node Master, we improve the maximum task tracker TaskTracker calculation task module section of load earlier.
As explaining among Fig. 4; With task tracker TaskTracker module disassemble by CPU call the part and by the GPU calculating section; GPU can not replace other non-computing functions of CPU; Like access hard disk, obtain physical address, reading and writing of files etc. and still do by CPU, calculate this heavy task and then can let GPU go to accomplish.Here, at first CPU calls part and will call the GPU calculating section then and give a plurality of Map operations among the GPU with each deblocking, and start these Map tasks among the GPU according to option read data files and with its piecemeal from hard disk or internal memory.Can produce after Map task run among the GPU is accomplished that a plurality of middle key/value is right, and then GPU start again sorting operation with these middle key/values to according to keywords carrying out sorting operation.Next call the part catcher by CPU; To carrying out piecemeal again, and give a plurality of Reduce operations in the GPU calculating section middle key/value that sorts by key word again, start these Reduce tasks among the GPU then each piecemeal; Obtain a plurality of output results of Reduce task at last; The final output valve of startup union operation acquisition is given CPU and is called part, so far, even if preliminary completion of MapReduce computation process.
The collaborative computing method concrete steps of many GPU based on Mapreduce of the present invention are following:
1. after back end DataNode received the read-write requests from NameNode, horizontal division was carried out with large-scale dataset in the MapReduce storehouse, was divided into the data subset splits fragment of M fixed size, and this part work has CPU to handle.M is a natural number, and its size is to come and the data results determines according to the number of computing node in the computing system usually.
2. M data subclass split formatd, further resolve into a collection of key/value < key1, value1 >; Specifically be formatted as which kind of data layout, key value and value value can be set according to the characteristics of concrete data set, for example can be in the things database with < key1; Value1>be set to < Tid; List >, Tid representes the things identifier in the things database, list is the corresponding list value of things in the things database.Because operations such as this part comprises access hard disk, obtains physical address, reading and writing of files, this part is accomplished by CPU, calls the relevant Map function of GPU then and carries out calculating operation.
3.Map the task of function is to each data subset split of input, creates a Map task, with each the record < key1 among the corresponding split; Value1>to the line scanning of going forward side by side as input, and it is formatd to the GPU algorithm, use the GPU algorithm to realize a local combiner Combiner; Produce and the middle < key2 of output; Value2>key/value is right, and it is right for example in the things database, to may be defined as < itemsets, sup>key/value; Sup representes the support counting of itemsets in data subset, and itemsets representes candidate k item collection.
4. utilize middle key/value that subregion function hash (key) mod R produces the Map function to being divided into R different partition areas, the R here is a natural number less than M.Then GPU sorts intermediate result according to key2; And the value2 data aggregate that the key2 value is identical forms a new tabulation together; It is right to form < key2, list (value2)>key/value, and list (value2) is the array that value2 formed by identical key2 value; Again with these key/values to being divided into R different partition areas, each division is fitted on the Reduce task of appointment.
5. the workstation that has been assigned with the Reduce task calls CPU, and data < key2, list (value2)>key/value that reads the submission of Map function is right.Intermediate data after the traversal ordering, CPU passes to GPU with each subregion, is formatd by it, uses the GPU concurrent technique to do the handled operation, obtains a plurality of output results of Reduce task, starts union operation and obtains final output valve.
Call part 6.GPU give CPU with net result, so far, a MapReduce technical process is accomplished.
Claims (1)
1. the many GPU based on Mapreduce work in coordination with computing method, it is characterized in that comprising following step:
1) at first, client is to the request of management phase transmission tasks;
2) then, the name node NameNode in the management phase is in charge of NameSpace, the calculation stages cluster configuration information of file system, the positional information of storage block; Work shadow device JobTracker is responsible for calculation task is started and dispatch, and the implementation status of realization tracing task and the state of calculation stages;
3) in calculation stages:
1. after back end DataNode receives the read-write requests from name node NameNode; Call CPU; Mass data is read scanning, and horizontal division again is divided into the data subset splits fragment of M fixed size; M is a natural number, and its size determines according to the number and the data results of computing node in the computing system;
2. the task tracker TaskTracker of idle CPU formats M data subclass split to work shadow device JobTracker request task and after meeting with a response, and further resolves into a collection of key/value to < key1, value1 >;
3. the task tracker TaskTracke of idle GPU is to work shadow device JobTracker request task and after meeting with a response, and each the data subset split to input creates a Map task; With each the record < key1 among the corresponding split; Value1>to the line scanning of going forward side by side as input, and it is gone up the operation special algorithm to GPU format, use the CUDA storehouse of GPU to realize a local combiner Combiner; < key2, value2>key/value is right in the middle of producing and exporting;
4. utilize middle key/value that subregion function hash (key) mod R produces the Map function to being divided into R different partition areas; The R here is a natural number less than M; Then GPU carries out relevance ranking with intermediate result according to key2, and the value2 data aggregate that the key2 value is identical forms a new tabulation, formation < key2 together; List (value2)>right; List (value2) is for by the array that value2 formed of identical key2 value, again with these key/values to being divided into R different partition areas, each division is fitted on the Reduce task of appointment;
5. the workstation that has been assigned with the Reduce task calls idle CPU; The task tracker TaskTracker that starts CPU removes to read the data < key2 that the Map function is submitted to; List (value2) >, behind the intermediate data after the traversal ordering, the task tracker TaskTracker of CPU passes to each subregion the task tracker TaskTracker of idle GPU; Format by it; Use the GPU concurrent technique to do the handled operation, obtain a plurality of output results of Reduce task, start union operation and obtain final output valve;
6. the task tracker TaskTracker of GPU gives CPU with net result and calls part.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012101028344A CN102662639A (en) | 2012-04-10 | 2012-04-10 | Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2012101028344A CN102662639A (en) | 2012-04-10 | 2012-04-10 | Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN102662639A true CN102662639A (en) | 2012-09-12 |
Family
ID=46772140
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2012101028344A Pending CN102662639A (en) | 2012-04-10 | 2012-04-10 | Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN102662639A (en) |
Cited By (34)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103020320A (en) * | 2013-01-11 | 2013-04-03 | 西安交通大学 | Method for reusing and optimizing video-memory-level data of GPU (graphic processing unit) on basis of dynamic search during running of GPU |
| CN103279330A (en) * | 2013-05-14 | 2013-09-04 | 江苏名通信息科技有限公司 | MapReduce multiple programming model based on virtual machine GPU computation |
| CN103336959A (en) * | 2013-07-19 | 2013-10-02 | 西安电子科技大学 | Vehicle detection method based on GPU (ground power unit) multi-core parallel acceleration |
| CN103684754A (en) * | 2013-12-03 | 2014-03-26 | 中国电子科技集团公司第三十研究所 | WPA shared key cracking system based on GPU cluster |
| CN103699656A (en) * | 2013-12-27 | 2014-04-02 | 同济大学 | GPU-based mass-multimedia-data-oriented MapReduce platform |
| WO2014108768A1 (en) * | 2013-01-11 | 2014-07-17 | International Business Machines Corporation | Computing regression models |
| CN103955400A (en) * | 2014-04-17 | 2014-07-30 | 国网宁夏电力公司 | Online checking method of parallel computing in electrical power system |
| CN104133661A (en) * | 2014-07-30 | 2014-11-05 | 西安电子科技大学 | Multi-core parallel hash partitioning optimizing method based on column storage |
| WO2014206233A1 (en) * | 2013-06-25 | 2014-12-31 | 华为技术有限公司 | Data processing method and device |
| CN104270437A (en) * | 2014-09-25 | 2015-01-07 | 中国科学院大学 | Mass data processing and visualizing system and method of distributed mixed architecture |
| CN104536937A (en) * | 2014-12-30 | 2015-04-22 | 深圳先进技术研究院 | Big data appliance realizing method based on CPU-GPU heterogeneous cluster |
| CN104731569A (en) * | 2013-12-23 | 2015-06-24 | 华为技术有限公司 | Data processing method and relevant equipment |
| CN104965689A (en) * | 2015-05-22 | 2015-10-07 | 浪潮电子信息产业股份有限公司 | Hybrid parallel computing method and device for CPUs/GPUs |
| CN104978228A (en) * | 2014-04-09 | 2015-10-14 | 腾讯科技（深圳）有限公司 | Scheduling method and scheduling device of distributed computing system |
| CN105094981A (en) * | 2014-05-23 | 2015-11-25 | 华为技术有限公司 | Method and device for processing data |
| CN105574649A (en) * | 2015-12-10 | 2016-05-11 | 西安交通大学 | Taxpayer tax evasion suspicion group detection method based on multi-stage MapReduce model |
| CN105608046A (en) * | 2015-12-17 | 2016-05-25 | 南京航空航天大学 | Multi-core processor architecture based on MapReduce programming model |
| CN105677486A (en) * | 2016-01-08 | 2016-06-15 | 上海交通大学 | Data parallel processing method and system |
| CN105740332A (en) * | 2016-01-22 | 2016-07-06 | 北京京东尚科信息技术有限公司 | Data sorting method and device |
| CN105786938A (en) * | 2014-12-26 | 2016-07-20 | 华为技术有限公司 | Big data processing method and apparatus |
| WO2017113277A1 (en) * | 2015-12-31 | 2017-07-06 | 华为技术有限公司 | Data processing method, device, and system |
| CN107391250A (en) * | 2017-08-11 | 2017-11-24 | 成都优易数据有限公司 | A kind of controller of raising Mapreduce task Shuffle performances |
| CN107515860A (en) * | 2017-08-07 | 2017-12-26 | 中译语通科技（青岛）有限公司 | A kind of machine translation method based on neuron |
| CN107729138A (en) * | 2017-09-14 | 2018-02-23 | 北京天耀宏图科技有限公司 | A kind of analysis method and device of high-performance distributed Vector spatial data |
| CN107885599A (en) * | 2016-09-30 | 2018-04-06 | 达索系统公司 | Method, program and the system of 3D scenes are simulated with one group of computing resource run parallel |
| CN109388428A (en) * | 2017-08-11 | 2019-02-26 | 华为技术有限公司 | Figure layer traversal method, control device and data processing system |
| CN109743453A (en) * | 2018-12-29 | 2019-05-10 | 出门问问信息科技有限公司 | A kind of multi-screen display method and device |
| CN109992575A (en) * | 2019-02-12 | 2019-07-09 | 哈尔滨学院 | The distributed memory system of big data |
| CN109992372A (en) * | 2017-12-29 | 2019-07-09 | 中国移动通信集团陕西有限公司 | A kind of data processing method and device based on mapping reduction |
| CN110187970A (en) * | 2019-05-30 | 2019-08-30 | 北京理工大学 | A kind of distributed big data parallel calculating method based on Hadoop MapReduce |
| CN110222105A (en) * | 2019-05-14 | 2019-09-10 | 联动优势科技有限公司 | Data summarization processing method and processing device |
| CN112307008A (en) * | 2020-12-14 | 2021-02-02 | 湖南蚁坊软件股份有限公司 | Druid compaction method |
| CN112444851A (en) * | 2019-08-30 | 2021-03-05 | 中国石油化工股份有限公司 | Reverse time migration imaging method based on MapReduce parallel framework and storage medium |
| CN116389485A (en) * | 2023-06-05 | 2023-07-04 | 上海朗力半导体有限公司 | Data center system and method, map node, reduce node, equipment and chip |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090158248A1 (en) * | 2007-12-17 | 2009-06-18 | Linderman Michael D | Compiler and Runtime for Heterogeneous Multiprocessor Systems |
| CN102262557A (en) * | 2010-05-25 | 2011-11-30 | 运软网络科技(上海)有限公司 | Method for constructing virtual machine monitor by bus architecture and performance service framework |
-
2012
- 2012-04-10 CN CN2012101028344A patent/CN102662639A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090158248A1 (en) * | 2007-12-17 | 2009-06-18 | Linderman Michael D | Compiler and Runtime for Heterogeneous Multiprocessor Systems |
| CN102262557A (en) * | 2010-05-25 | 2011-11-30 | 运软网络科技(上海)有限公司 | Method for constructing virtual machine monitor by bus architecture and performance service framework |
Non-Patent Citations (2)
| Title |
|---|
| BINGSHENG HE等: "Mars: A MapReduce Framework on Graphics Processors", 《PROC. 17TH INTERNATIONAL CONFERENCE ON PARALLEL ARCHITECTURES AND COMPILATION TECHNIQUES(PACT)》 * |
| JEFFREY DEAN等: "MapReduce: Simplified Data Processing on Large Clusters", 《OSDI》 * |
Cited By (60)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US9159028B2 (en) | 2013-01-11 | 2015-10-13 | International Business Machines Corporation | Computing regression models |
| CN103020320B (en) * | 2013-01-11 | 2016-01-13 | 西安交通大学 | GPU video memory DBMS multiplex optimization method during a kind of operation based on News Search |
| US9152921B2 (en) | 2013-01-11 | 2015-10-06 | International Business Machines Corporation | Computing regression models |
| CN103020320A (en) * | 2013-01-11 | 2013-04-03 | 西安交通大学 | Method for reusing and optimizing video-memory-level data of GPU (graphic processing unit) on basis of dynamic search during running of GPU |
| WO2014108768A1 (en) * | 2013-01-11 | 2014-07-17 | International Business Machines Corporation | Computing regression models |
| CN104937544A (en) * | 2013-01-11 | 2015-09-23 | 国际商业机器公司 | Computing regression models |
| CN104937544B (en) * | 2013-01-11 | 2017-06-13 | 国际商业机器公司 | Method, computer-readable medium and computer system for calculating task result |
| CN103279330A (en) * | 2013-05-14 | 2013-09-04 | 江苏名通信息科技有限公司 | MapReduce multiple programming model based on virtual machine GPU computation |
| WO2014206233A1 (en) * | 2013-06-25 | 2014-12-31 | 华为技术有限公司 | Data processing method and device |
| CN103336959B (en) * | 2013-07-19 | 2016-09-28 | 西安电子科技大学 | A kind of vehicle checking method accelerated based on GPU multi-core parallel concurrent |
| CN103336959A (en) * | 2013-07-19 | 2013-10-02 | 西安电子科技大学 | Vehicle detection method based on GPU (ground power unit) multi-core parallel acceleration |
| CN103684754B (en) * | 2013-12-03 | 2016-11-23 | 中国电子科技集团公司第三十研究所 | A kind of WPA shared key based on GPU cluster cracks system |
| CN103684754A (en) * | 2013-12-03 | 2014-03-26 | 中国电子科技集团公司第三十研究所 | WPA shared key cracking system based on GPU cluster |
| WO2015096649A1 (en) * | 2013-12-23 | 2015-07-02 | 华为技术有限公司 | Data processing method and related device |
| CN104731569A (en) * | 2013-12-23 | 2015-06-24 | 华为技术有限公司 | Data processing method and relevant equipment |
| CN104731569B (en) * | 2013-12-23 | 2018-04-10 | 华为技术有限公司 | A kind of data processing method and relevant device |
| CN103699656A (en) * | 2013-12-27 | 2014-04-02 | 同济大学 | GPU-based mass-multimedia-data-oriented MapReduce platform |
| CN104978228B (en) * | 2014-04-09 | 2019-08-30 | 腾讯科技（深圳）有限公司 | A kind of dispatching method and device of distributed computing system |
| CN104978228A (en) * | 2014-04-09 | 2015-10-14 | 腾讯科技（深圳）有限公司 | Scheduling method and scheduling device of distributed computing system |
| CN103955400A (en) * | 2014-04-17 | 2014-07-30 | 国网宁夏电力公司 | Online checking method of parallel computing in electrical power system |
| WO2015176689A1 (en) * | 2014-05-23 | 2015-11-26 | 华为技术有限公司 | Data processing method and device |
| CN105094981A (en) * | 2014-05-23 | 2015-11-25 | 华为技术有限公司 | Method and device for processing data |
| CN105094981B (en) * | 2014-05-23 | 2019-02-12 | 华为技术有限公司 | A kind of method and device of data processing |
| CN104133661B (en) * | 2014-07-30 | 2017-01-18 | 西安电子科技大学 | Multi-core parallel hash partitioning optimizing method based on column storage |
| CN104133661A (en) * | 2014-07-30 | 2014-11-05 | 西安电子科技大学 | Multi-core parallel hash partitioning optimizing method based on column storage |
| CN104270437B (en) * | 2014-09-25 | 2017-08-25 | 中国科学院大学 | The mass data processing of distributed mixed architecture and visualization system and method |
| CN104270437A (en) * | 2014-09-25 | 2015-01-07 | 中国科学院大学 | Mass data processing and visualizing system and method of distributed mixed architecture |
| US10691669B2 (en) | 2014-12-26 | 2020-06-23 | Huawei Technologies Co., Ltd. | Big-data processing method and apparatus |
| CN105786938A (en) * | 2014-12-26 | 2016-07-20 | 华为技术有限公司 | Big data processing method and apparatus |
| CN104536937B (en) * | 2014-12-30 | 2017-10-31 | 深圳先进技术研究院 | Big data all-in-one machine realization method based on CPU GPU isomeric groups |
| CN104536937A (en) * | 2014-12-30 | 2015-04-22 | 深圳先进技术研究院 | Big data appliance realizing method based on CPU-GPU heterogeneous cluster |
| CN104965689A (en) * | 2015-05-22 | 2015-10-07 | 浪潮电子信息产业股份有限公司 | Hybrid parallel computing method and device for CPUs/GPUs |
| CN105574649B (en) * | 2015-12-10 | 2021-05-28 | 西安交通大学 | Tax payer tax evasion suspicion group detection method based on multi-stage MapReduce model |
| CN105574649A (en) * | 2015-12-10 | 2016-05-11 | 西安交通大学 | Taxpayer tax evasion suspicion group detection method based on multi-stage MapReduce model |
| CN105608046A (en) * | 2015-12-17 | 2016-05-25 | 南京航空航天大学 | Multi-core processor architecture based on MapReduce programming model |
| WO2017113277A1 (en) * | 2015-12-31 | 2017-07-06 | 华为技术有限公司 | Data processing method, device, and system |
| US10599436B2 (en) | 2015-12-31 | 2020-03-24 | Huawei Technologies Co., Ltd. | Data processing method and apparatus, and system |
| CN105677486B (en) * | 2016-01-08 | 2019-03-22 | 上海交通大学 | Data parallel processing method and system |
| CN105677486A (en) * | 2016-01-08 | 2016-06-15 | 上海交通大学 | Data parallel processing method and system |
| CN105740332A (en) * | 2016-01-22 | 2016-07-06 | 北京京东尚科信息技术有限公司 | Data sorting method and device |
| CN107885599B (en) * | 2016-09-30 | 2023-07-28 | 达索系统公司 | Method, program and system for simulating 3D scene by using a group of parallel running computing resources |
| CN107885599A (en) * | 2016-09-30 | 2018-04-06 | 达索系统公司 | Method, program and the system of 3D scenes are simulated with one group of computing resource run parallel |
| CN107515860A (en) * | 2017-08-07 | 2017-12-26 | 中译语通科技（青岛）有限公司 | A kind of machine translation method based on neuron |
| CN109388428A (en) * | 2017-08-11 | 2019-02-26 | 华为技术有限公司 | Figure layer traversal method, control device and data processing system |
| CN109388428B (en) * | 2017-08-11 | 2021-05-04 | 华为技术有限公司 | Layer traversal method, control device and data processing system |
| CN107391250A (en) * | 2017-08-11 | 2017-11-24 | 成都优易数据有限公司 | A kind of controller of raising Mapreduce task Shuffle performances |
| CN107729138B (en) * | 2017-09-14 | 2020-11-20 | 北京天耀宏图科技有限公司 | Method and device for analyzing high-performance distributed vector space data |
| CN107729138A (en) * | 2017-09-14 | 2018-02-23 | 北京天耀宏图科技有限公司 | A kind of analysis method and device of high-performance distributed Vector spatial data |
| CN109992372A (en) * | 2017-12-29 | 2019-07-09 | 中国移动通信集团陕西有限公司 | A kind of data processing method and device based on mapping reduction |
| CN109743453A (en) * | 2018-12-29 | 2019-05-10 | 出门问问信息科技有限公司 | A kind of multi-screen display method and device |
| CN109992575A (en) * | 2019-02-12 | 2019-07-09 | 哈尔滨学院 | The distributed memory system of big data |
| CN109992575B (en) * | 2019-02-12 | 2020-02-14 | 哈尔滨学院 | Distributed storage system for big data |
| CN110222105B (en) * | 2019-05-14 | 2021-06-29 | 联动优势科技有限公司 | Data summarization processing method and device |
| CN110222105A (en) * | 2019-05-14 | 2019-09-10 | 联动优势科技有限公司 | Data summarization processing method and processing device |
| CN110187970A (en) * | 2019-05-30 | 2019-08-30 | 北京理工大学 | A kind of distributed big data parallel calculating method based on Hadoop MapReduce |
| CN112444851A (en) * | 2019-08-30 | 2021-03-05 | 中国石油化工股份有限公司 | Reverse time migration imaging method based on MapReduce parallel framework and storage medium |
| CN112307008A (en) * | 2020-12-14 | 2021-02-02 | 湖南蚁坊软件股份有限公司 | Druid compaction method |
| CN112307008B (en) * | 2020-12-14 | 2023-12-08 | 湖南蚁坊软件股份有限公司 | Druid compacting method |
| CN116389485A (en) * | 2023-06-05 | 2023-07-04 | 上海朗力半导体有限公司 | Data center system and method, map node, reduce node, equipment and chip |
| CN116389485B (en) * | 2023-06-05 | 2023-08-15 | 上海朗力半导体有限公司 | Data center system and method, map node, reduce node, equipment and chip |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN102662639A (en) | Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method | |
| US10769148B1 (en) | Relocating data sharing operations for query processing | |
| CN102663117B (en) | OLAP (On Line Analytical Processing) inquiry processing method facing database and Hadoop mixing platform | |
| Ji et al. | Big data processing: Big challenges and opportunities | |
| CN102567495B (en) | Mass information storage system and implementation method | |
| CN107515878B (en) | Data index management method and device | |
| CN102609446B (en) | Distributed Bloom filter system and application method thereof | |
| CN108280522A (en) | A kind of plug-in type distributed machines study Computational frame and its data processing method | |
| CN103678520A (en) | Multi-dimensional interval query method and system based on cloud computing | |
| Wang et al. | Research and implementation on spatial data storage and operation based on Hadoop platform | |
| CN109933631A (en) | Distributed parallel database system and data processing method based on Infiniband network | |
| CN104375824A (en) | Data processing method | |
| CN107220310A (en) | A kind of database data management system, method and device | |
| CN110147377B (en) | General query method based on secondary index under large-scale spatial data environment | |
| CN102937964B (en) | Intelligent data service method based on distributed system | |
| Ngu et al. | B+-tree construction on massive data with Hadoop | |
| CN106569896B (en) | A kind of data distribution and method for parallel processing and system | |
| CN104731925A (en) | MapReduce-based FP-Growth load balance parallel computing method | |
| CN104111924A (en) | Database system | |
| CN106055678A (en) | Hadoop-based panoramic big data distributed storage method | |
| Koh et al. | MapReduce skyline query processing with partitioning and distributed dominance tests | |
| CN110677461A (en) | Graph calculation method based on key value pair storage | |
| US10642520B1 (en) | Memory optimized data shuffle | |
| CN104063501A (en) | Copy balancing method based HDFS | |
| CN107908713B (en) | Distributed dynamic rhododendron filtering system based on Redis cluster and filtering method thereof |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| EE01 | Entry into force of recordation of patent licensing contract |
Application publication date: 20120912 Assignee: Jiangsu Wisedu Information Technology Co., Ltd. Assignor: Nanjing University of Aeronautics and Astronautics Contract record no.: 2013320000314 Denomination of invention: Mapreduce-based multi-GPU (Graphic Processing Unit) cooperative computing method License type: Exclusive License Record date: 20130410 |
|
| LICC | Enforcement, change and cancellation of record of contracts on the licence for exploitation of a patent or utility model | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20120912 |
|
| EC01 | Cancellation of recordation of patent licensing contract |
Assignee: JIANGSU WISEDU EDUCATION INFORMATION TECHNOLOGY CO., LTD. Assignor: Nanjing University of Aeronautics and Astronautics Contract record no.: 2013320000314 Date of cancellation: 20150421 |
|
| LICC | Enforcement, change and cancellation of record of contracts on the licence for exploitation of a patent or utility model |