CA1302584C - Parallel processing computer in which memory access priorities are varied - Google Patents
Parallel processing computer in which memory access priorities are variedInfo
- Publication number
- CA1302584C CA1302584C CA000573400A CA573400A CA1302584C CA 1302584 C CA1302584 C CA 1302584C CA 000573400 A CA000573400 A CA 000573400A CA 573400 A CA573400 A CA 573400A CA 1302584 C CA1302584 C CA 1302584C
- Authority
- CA
- Canada
- Prior art keywords
- processors
- priority
- access
- offset
- memory elements
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Lifetime
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F12/00—Accessing, addressing or allocating within memory systems or architectures
- G06F12/02—Addressing or allocation; Relocation
- G06F12/08—Addressing or allocation; Relocation in hierarchically structured memory systems, e.g. virtual memory systems
- G06F12/0802—Addressing of a memory level in which the access to the desired data or data block requires associative addressing means, e.g. caches
- G06F12/0806—Multiuser, multiprocessor or multiprocessing cache systems
- G06F12/084—Multiuser, multiprocessor or multiprocessing cache systems with a shared cache
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F13/00—Interconnection of, or transfer of information or other signals between, memories, input/output devices or central processing units
- G06F13/14—Handling requests for interconnection or transfer
- G06F13/16—Handling requests for interconnection or transfer for access to memory bus
- G06F13/18—Handling requests for interconnection or transfer for access to memory bus based on priority control
Abstract
Abstract of the Disclosure A parallel processing computer is disclosed in which a plurality of memory elements (e.g., caches) are accessable by a plurality of processors, and in which a fixed access priority for the processors is varied periodically to reduce differences in processing times between the processors in applications where memory access conflicts occur. The variation in priority is done infrequently enough so as not to disturb the ability of the system to avoid memory access conflicts by falling into a "lockstep" condition, in which the fixed priority combined with a selected interleaving of the memory elements produces a memory access pattern that, for certain memory strides, produces no memory access conflicts.
Description
~3~12S~3~
COMPUTER WITH MULTI~E PROC~SSORS HAVING VARYING
PRIORITIES FOR ACCESS TO A MULTI-EL_MENT NEMORY
Backqround of the Invention 1. Field of the Invention This invention relates to digital computers capable of parallel processing, and particularly to assigning memory access priorities to ~he processors in such computers.
COMPUTER WITH MULTI~E PROC~SSORS HAVING VARYING
PRIORITIES FOR ACCESS TO A MULTI-EL_MENT NEMORY
Backqround of the Invention 1. Field of the Invention This invention relates to digital computers capable of parallel processing, and particularly to assigning memory access priorities to ~he processors in such computers.
2. Description of the Related Art In engineering and scientific applications, it is typical to find repetitive accesses to memory at fixed address intervals known as strides. Each new access ini~iated by a processor is for a memory location separated from the last access by the length of the stride. A stride of one means that the processor accesses every word (whose length may vary) in sequence.
A stride of two means that every other word is accessed. When interleaved memory elements (i.e., ones in which memory addresses are assigned on the basis of a low-order portion of the memory address) are accessed by the processors, the stride determines a unique sequence of memory accesses known as the access pattern.
Caches ha~e long been used in digital computers, and have been applied to parallel processing, with one cache assigned to each processor. A cache is a high-speed memory containing copies of selected data from the main memory. Memory accesses from a processor come to the cache, which determines whether it currently has a copy of the accessed memory location. If not, a cache "miss" has occurred, and tlle cache customarily stops accepting new accesses while it performs a main memory access for the data needed by the processor.
Conventionally each cache serves a single processor, but a cache configured so that it may serve multiple processors is disclosed in Canadian Patent Application Serial No. 511,838, filed June 18, 1986, now Canadian Ratent No. 1,270,064.
;~
~31~25i~14 Where several caches are serving multiple processors, it is desirable to minimize simultaneous accesses by ~ifferent processors to the same cache. One mechanism ~or doing so is disclosed in the referencetl application. ~ach processor is assigned a fixed priority level, and the caches are interleaved in such a manner that, ~or selected memory strides, the processors fall into an access pattern in which no simultaneous access requests occur (known as "lockstep").
Summary of the Invention We have come to realize that, while this interleaving mechanism produces efficient operation when lockstep can be achieved, it may lead to inefficiency in some situations where lockstep does not occur ~e.g., because of frequent cache misses, or memory accesses that do not follow a stride patternr or that follow an unusual stride that will not force lockstep to occur).
In these situations, if the parallel processors are called upon to process tasks of comparable size (e.g., when a vector operation is evenly split among the processors) and the softwaxe allows the first-to-finish processor to remain idle until the last-to-finish processor has completed its task, the fixed priority can produce inefficient operation. A processor with a low priority will take a longer time to complete the task than a processor with a high priority, because of the greater number o~ instances when the low priority processor will wait for access to the cache. Higher priority processors will finish sooner than those with low priority and will remain idle waiting for the others to finish.
~a3~
We have discovered a technique for improving performance in those situations where lockstep i6 not achieved, without subætantially degrading performance when it is. The memory elements are interleaved and processors are assigned fixed priority levels, all as described in the cited Canadian Patent 1,270,064. But performance is improved in the absence of locks~ep by periodically rotating the fixed priority levels among the processors, so that delays (e.g., from cache access conflicts and cache misses) are more equally shared by the processors.
Preferably, the period between priority rotation is made long enough not to interfere with esta~lishment of lockstep (for those cases where lockstep can be achieved), yet not so long as to unduly reduce the delay-sharing benefit of the priority rotatlon (in those cases when lockstep is not achieved and when early ~inishing processors must remain idle until other processors are finished). The invention is extremely simple to lmplement (e.g., using a counter to initiate the priority ro~ation after a fixed number of cache cycles) and achieves a significant improvament in processing speed in the non-lockstep situation, without appreciable loss of speed in the lockstep case.
In preferred embodiments, the prlori-ty rotation is such that the lowest priority pair oE processors are moved to the highest priority position at each priority rotation and the other pairs of processors are moved down a notch in priority. In the case where there are eight processors, Eour cache quadrants, and the cache operates at twice the clock speed of the processors (so .~
~3~
that the quadrants are capable of acceptin~ two accesses during each processor cycle), priority rotation of processor pairs is preferably done every 14 cache cycles.
According to a broad aspect of the invention there is provided a digital processing system, comprising: a plurality of memory elements, a plurality of parallel processors, s~id processors eacll having means for initiating an access of data from said memory ele~ents simultaneously with accesses of other processors, said memory elements beiny interleaved (i.e., each is assigned memory addresses on the basis of a low order portion of the memory address) so that the access patterns generated by said processors when accessing data from said memory elements at a predetermined stride permits said processors to reach a phase relationship known as "lockstep") with other said processors in which each said processor is able to access a differant said memory element simul~aneously without creating access conflic~s, means for deciding, based on fixed priority levels, which said processor is granted access to a yiven memory element when there are simultaneous access requests to the given memory element, and priority-altering means for altering at a predetermined periodic interval said fixed priority levels to vary the hiyhest priority among said processors, said predetermined periodic interval being selected so as not to lengthen substantially the time required for said proces~ors to fall into lockstep and said prede~ermined periodic interval being selected so as to reduce differences in the times required by each of said processors to complete comparable tasks not amenable to lockstep execution.
3a B
~3C~ 8~
Other features and advan~ages of the invention will be apparent from the followiny description of a preferred embodiment and from the claims.
Brief DescriPtion of the Drawinqs Fig. 1 is a system block diagram.
Fig. 2 is a block diagram showing the computational elements (CEs), a backplane switch, and cache ~uadrants, and the connections therebetween.
Fig. 3 is a hlock diagram showing a portion of the digital logic on the CP cache boards and cache quadrants of Fig. 1.
DescriPtion of the Preferred Embodim~nt A system block diagram is shown in Fig. l. Eight high-speed processors or computational elements ~CEs) 10 are connected to two central processing cache boards 12 by a switch 14 which resides on a backplane (not shown) into which the CE and cache boards are plugged. Each cache board 12 comprises two cache quadrants 15. The switch permits any four C'Es to be concurrently accessing the four cache quadrants. The CEs each have a concurrency control unit ~CCU) 24 for controlling concurrent processing. The CCUs communicate with other CCUs across a concurrency control bus 26. Memory bus 16 connects the cache quadrants to eiyht memory modules 17 ~each 8 or 32 megabytes).
Also connected to the memory bus are two interacti~re processor caches 18, each of which is connected to three interactiv0 processors ~IPs) 20. Each IP serves a multibus 22, to which peripheral devices ~not shown) are connected.
The system is a shared-global-memory, symmetric (i.e.
not master-slave~ multiprocessing computer particularly useful for general scientific and engineering computation. The CEs concurrently process different iterations of the same iterative construct ~3~
(but they may also operate independently to provide a high-performance multi-tasking system). The IPs are moderate-speed interactive processors that are used for handling input/output traffic, text editing, and similar operations Data types supported include 8, 16, and 32 bit integer/logical data as well as IEEE standard 32 and 64 bit floating-point data on the CEs only. Memory is virtually addressed. CEs access global memory through cache boards 12, which the CEs communicate with via switch 14. Each CE has its own 16K byte virtually-addressed instruction cache (not shown). The IPs access global memory through interactive processor caches 18.
The CEs are intended for high-speed computations. The CEs are identical, and as few as one may be installed in a system. Any number of CEs may participate in concurrent processing. Those that do are said to be in the concurrency complex (hereinater "the complex"). Those that do not are said to be detached.
CEs can be detached in the event that a job mix being executed on the system includes jobs that cannot make use of concurrent processing (e.g. compilation and debugging) as well as jobs that can (e.g. production jobs). A detached CE acts as if it were a system with 25 only one CE present. CEs have a 170 nanosecond clock.
The system has a global central processing cache that serves up to eight CEs. The cache is divided into four interleaved quadrants 15 (denoted W,X,Y,Z), two of which reside on each of two CP cache boards 12;
A stride of two means that every other word is accessed. When interleaved memory elements (i.e., ones in which memory addresses are assigned on the basis of a low-order portion of the memory address) are accessed by the processors, the stride determines a unique sequence of memory accesses known as the access pattern.
Caches ha~e long been used in digital computers, and have been applied to parallel processing, with one cache assigned to each processor. A cache is a high-speed memory containing copies of selected data from the main memory. Memory accesses from a processor come to the cache, which determines whether it currently has a copy of the accessed memory location. If not, a cache "miss" has occurred, and tlle cache customarily stops accepting new accesses while it performs a main memory access for the data needed by the processor.
Conventionally each cache serves a single processor, but a cache configured so that it may serve multiple processors is disclosed in Canadian Patent Application Serial No. 511,838, filed June 18, 1986, now Canadian Ratent No. 1,270,064.
;~
~31~25i~14 Where several caches are serving multiple processors, it is desirable to minimize simultaneous accesses by ~ifferent processors to the same cache. One mechanism ~or doing so is disclosed in the referencetl application. ~ach processor is assigned a fixed priority level, and the caches are interleaved in such a manner that, ~or selected memory strides, the processors fall into an access pattern in which no simultaneous access requests occur (known as "lockstep").
Summary of the Invention We have come to realize that, while this interleaving mechanism produces efficient operation when lockstep can be achieved, it may lead to inefficiency in some situations where lockstep does not occur ~e.g., because of frequent cache misses, or memory accesses that do not follow a stride patternr or that follow an unusual stride that will not force lockstep to occur).
In these situations, if the parallel processors are called upon to process tasks of comparable size (e.g., when a vector operation is evenly split among the processors) and the softwaxe allows the first-to-finish processor to remain idle until the last-to-finish processor has completed its task, the fixed priority can produce inefficient operation. A processor with a low priority will take a longer time to complete the task than a processor with a high priority, because of the greater number o~ instances when the low priority processor will wait for access to the cache. Higher priority processors will finish sooner than those with low priority and will remain idle waiting for the others to finish.
~a3~
We have discovered a technique for improving performance in those situations where lockstep i6 not achieved, without subætantially degrading performance when it is. The memory elements are interleaved and processors are assigned fixed priority levels, all as described in the cited Canadian Patent 1,270,064. But performance is improved in the absence of locks~ep by periodically rotating the fixed priority levels among the processors, so that delays (e.g., from cache access conflicts and cache misses) are more equally shared by the processors.
Preferably, the period between priority rotation is made long enough not to interfere with esta~lishment of lockstep (for those cases where lockstep can be achieved), yet not so long as to unduly reduce the delay-sharing benefit of the priority rotatlon (in those cases when lockstep is not achieved and when early ~inishing processors must remain idle until other processors are finished). The invention is extremely simple to lmplement (e.g., using a counter to initiate the priority ro~ation after a fixed number of cache cycles) and achieves a significant improvament in processing speed in the non-lockstep situation, without appreciable loss of speed in the lockstep case.
In preferred embodiments, the prlori-ty rotation is such that the lowest priority pair oE processors are moved to the highest priority position at each priority rotation and the other pairs of processors are moved down a notch in priority. In the case where there are eight processors, Eour cache quadrants, and the cache operates at twice the clock speed of the processors (so .~
~3~
that the quadrants are capable of acceptin~ two accesses during each processor cycle), priority rotation of processor pairs is preferably done every 14 cache cycles.
According to a broad aspect of the invention there is provided a digital processing system, comprising: a plurality of memory elements, a plurality of parallel processors, s~id processors eacll having means for initiating an access of data from said memory ele~ents simultaneously with accesses of other processors, said memory elements beiny interleaved (i.e., each is assigned memory addresses on the basis of a low order portion of the memory address) so that the access patterns generated by said processors when accessing data from said memory elements at a predetermined stride permits said processors to reach a phase relationship known as "lockstep") with other said processors in which each said processor is able to access a differant said memory element simul~aneously without creating access conflic~s, means for deciding, based on fixed priority levels, which said processor is granted access to a yiven memory element when there are simultaneous access requests to the given memory element, and priority-altering means for altering at a predetermined periodic interval said fixed priority levels to vary the hiyhest priority among said processors, said predetermined periodic interval being selected so as not to lengthen substantially the time required for said proces~ors to fall into lockstep and said prede~ermined periodic interval being selected so as to reduce differences in the times required by each of said processors to complete comparable tasks not amenable to lockstep execution.
3a B
~3C~ 8~
Other features and advan~ages of the invention will be apparent from the followiny description of a preferred embodiment and from the claims.
Brief DescriPtion of the Drawinqs Fig. 1 is a system block diagram.
Fig. 2 is a block diagram showing the computational elements (CEs), a backplane switch, and cache ~uadrants, and the connections therebetween.
Fig. 3 is a hlock diagram showing a portion of the digital logic on the CP cache boards and cache quadrants of Fig. 1.
DescriPtion of the Preferred Embodim~nt A system block diagram is shown in Fig. l. Eight high-speed processors or computational elements ~CEs) 10 are connected to two central processing cache boards 12 by a switch 14 which resides on a backplane (not shown) into which the CE and cache boards are plugged. Each cache board 12 comprises two cache quadrants 15. The switch permits any four C'Es to be concurrently accessing the four cache quadrants. The CEs each have a concurrency control unit ~CCU) 24 for controlling concurrent processing. The CCUs communicate with other CCUs across a concurrency control bus 26. Memory bus 16 connects the cache quadrants to eiyht memory modules 17 ~each 8 or 32 megabytes).
Also connected to the memory bus are two interacti~re processor caches 18, each of which is connected to three interactiv0 processors ~IPs) 20. Each IP serves a multibus 22, to which peripheral devices ~not shown) are connected.
The system is a shared-global-memory, symmetric (i.e.
not master-slave~ multiprocessing computer particularly useful for general scientific and engineering computation. The CEs concurrently process different iterations of the same iterative construct ~3~
(but they may also operate independently to provide a high-performance multi-tasking system). The IPs are moderate-speed interactive processors that are used for handling input/output traffic, text editing, and similar operations Data types supported include 8, 16, and 32 bit integer/logical data as well as IEEE standard 32 and 64 bit floating-point data on the CEs only. Memory is virtually addressed. CEs access global memory through cache boards 12, which the CEs communicate with via switch 14. Each CE has its own 16K byte virtually-addressed instruction cache (not shown). The IPs access global memory through interactive processor caches 18.
The CEs are intended for high-speed computations. The CEs are identical, and as few as one may be installed in a system. Any number of CEs may participate in concurrent processing. Those that do are said to be in the concurrency complex (hereinater "the complex"). Those that do not are said to be detached.
CEs can be detached in the event that a job mix being executed on the system includes jobs that cannot make use of concurrent processing (e.g. compilation and debugging) as well as jobs that can (e.g. production jobs). A detached CE acts as if it were a system with 25 only one CE present. CEs have a 170 nanosecond clock.
The system has a global central processing cache that serves up to eight CEs. The cache is divided into four interleaved quadrants 15 (denoted W,X,Y,Z), two of which reside on each of two CP cache boards 12;
3~ together, the four cache quadrants comprise a four-way interleaved processing cache. Cache quadrants have a 85 nanosecond clock, which is twice the frequency of the CE
clock. Each cache quadrant can ordinarily complete one ~3~2S~L
quadword (64-bit) read or write access in each 85 nanosecond cycle, resulting in a total peak throughput in the vicinity of 375 megabytes per second.
The block size of the cache (the smallest amount of data moved between the cache and memory) is 32 bytes. The data of each block is interleaved between two quadrants W,X on the same cache board so that for any 32-byte block quadwords 0 and 3 (quadwords are 8-byte words) are contained in quadrant W and quadwords 1 and 2 in quadrant X. Contiguous 32-b~te blocks are interleaved between cache boards so that quadwords 4 and 7 are located in quadrant ~ and quadwords 5 and 6 in quadrant Z.
The cache quadrants receive a 28-bit physical 15 byte address from the CEs. Two bits (bits 3 and 4) of the address specify the desired quadword within the cache block. Because quadwords 0 and 3 are located in one quadrant and quadwords 1 and 2 are located in the other, a CE-supplied CADR4X3 signal ~see below), which is the Exclusive OR of bits 3 and 4, is used to determine which cache quadrant on a CP cache board contains the desired data. The CE also supplies a normal address bit 4 so that the selected cache quadrant can determine which of its two words is required.
Bit 5 of the address (supplied as CADR5 by the requesting CE; see below) indicates which of the two CP
cache boards in a four-way interleaved system contains the desired block of data.
The CEs are connected to the cache quadrants by ~,t~` 30 backplane switch 14, as shown in Fig. 2. Forty-eight ` control lines ~4~} extend across the backplane directly from the CEs to the cache guadrants, without passing through the backplane switch. Six control lines are connected to each CE, for a total of forty-eight control ~3~25~3~
lines, all of which are are connected to each quadrant.
The si~ control lines for each CE are: CSTRTL, CBUSYL, CWAITL, CADR5, CADR4X3, and CDTPAR. Collectively, the six control lines are denoted AC for CE0, BC for CEl, CC
for CE2, and so on. CBUSYL, CWAITL, and CDTPAR are implemented as open-collector signals.
Ninety-six lines extend to the backplane switch from each CE and each cache quadrant: sixty-four bidirectional data lines CDT(63:0), twenty-six address lines CADR(27:6,4,2:0), an address parity line CADRPAR, three CE operation lines CEOP(2:0), and two data-length lines CDLN(1 0). Each CE and cache quadrant also provides the backplane switch with enable lines, e.g. AE
and WE, for enabling data and addresses to be read from the ninety-six bidirectional lines connected to the switch. The cache quadrants provide the backplane switch with direction signals WD, XD, YD, ZD, which define the direction of data being transferred through the switch, and two sets of three-bit CE selection signals WS(2:0), XS(2:0), YS(2:0), ZS(2:0), which specify the CE to which, or from which, a transfer is being made. Only one set of selection signals is shown in the drawings, but separate 3-bit selection signals are provided for addresses and data.
Referring to Fig. 3, control lines CSTRTL, CADR5, and CADR4X3 (one of each from each CE) enter cache board 12 and, within board 12, cache quadrant W
where they serve as inputs to a CE-to-central-processing-cache ~CPC) decoder 300. Decoder 300 outputs eight CE
request lines CEREQ(7:0) which indicate which of the eight CEs are currently attempting to access quadrant W. For each CE requesting access, a corresponding CEREQ
line is asserted. CEREQ(7 0) is one of the inputs into CBUSYL generation logic 310; the other input to CBUSYL
~L3~5i9~
generation logic 310 is the priority mode signal PM(1:0~, which indicates which pair of CEs ~e.g. pair CEO, CEl) currently has the highest priority level.
PM(1:0) is generated by priority mode logic 312 on cache 5 board 12. Priority mode logic 312 is shared by the two cache quadrants 15 on each cache board 12, and is replicated on each cache board. Logic 312 includes a divide-by-14 counter 314 which supplies, once every fourteen clock cycles, a pulse for clocking a mode 10 counter 316. Counter 316 generates PM(1:0), cycling continuously through the sequence "00", "01", "10", "11". Using CEREQ(7:0) and PM(1:0), CBUSYL Generation logic 310 generates CBUSYL (7:0), which goes to the CEs, and CEGRANT (7:0), which connects to another section of 15 the internal logic of the cache quadrant. CEREQ(7:0) and PM(1:0) are also inputs to CE Select Encoder 318, which generates three bit selection signal WS(2:0).
Signals CSTRTL(7:0) are also inputs to an eight-inpu~ OR
gate 302 whose output is the reset input to divide-by-32 20 counter 304. Divide-by-32 counter 304 is connected to the RESET inputs of divide-by-14 counter 314 and mode counter 316. Decoder 300, logic 310, logic 318, OR gate 302, counter 304, and accompanying signal lines are replicated in each of the cache quadrants 15.
In operation, a CE initiates a cache operation by asserting its CSTRTL line and supplying the two address bits CADR5 and CADR4X3 specifying which of the four interleaved ~uadrants of the cache are involved in the operation. CADR5 is the fifth bit of the memory address involved in the operation being initiated by the CE. CADR4X3 is the exclusive-OR combination of bits three and four of the memory address. The reason that these two bits are combined in addressing the cache quadrants has to do with the way in which the cache ~3~)25~3~
quadrants are interleaved (see discussion below). If only two cache quadrants are in use (e.g. if only four CEs are in use), only address bit CADR4X3 is used to address a particular quadrant, and CADR5 is saved by the 5 addressed cache quadrant for use with the remainder of the address transferred through the backplane switch.
As many as eight CEs may simultaneously bid for access to any of the four cache quadrants. Access conflicts are resolved by the cache quadrants. In the 10 same 85 nanosecond cycle as the CSTRTL is asserted (the first cache cycle), the cache quadrant addressed resolves which of the contending CEs has priority.
Referring to Fig. 3, for each CE that bids for access to a cache quadrant, one of the eight CSTRTL lines entering 15 cache quadrant W will be asserted. For each CSTRTL line which is asserted, if the corresponding CADR5 and CADR4X3 lines indicate that it is cache quadrant W which the CE wishes to access, the corresponding CEREQ line will be asserted by decoder 300. As few as zero and as 20 many as eight of the CEREQ lines may thus be asserted.
The eight CEREQ lines indicate which of the CEs wish to access quadrant W.
When more than one CE is contending for access to the same quadrant, priority is given to the CE pair 25 specified by the priority signal PM(l:0). (See below for discussion of why priority changes are made in pairs.) The priority mode signal PM(l:0) indicates which of the four pairs of CEs ("00" signifying CE0, CEl; "11", CE2, CE3; "10", CE4, CE5, "01", CE6, CE7) 30 currently has the highes-t priority level. Within each pair of CEs, the CE having the lowest numerical designation prevails (thus CE0 prevails over CEl).
The priority mode signal is initialized on startup and incremented every 14 cache cycles. At startup, prior to the assertion of one of the CSTRTL
lines, divide-by-14 counter 314 and mode counter 316 are reset so that PM(l:G) is set to "00", which designates the pair CE0, CEl as the highest priority pair, the pair 5 CE2, CE3 as the next highest, and so on. Every fourteen cache clock cycles, divide-by-14 counter 314 supplies a pulse on line 326 to mode counter 316, causing it to increment PM(1:0) by one ~e.g. from 00 to 01). So long as counter 316 is not reset, it will continuously cycle PM(1:0) through the four priority modes (i.e. 00, 01, 10, 11), changing the mode once every fourteen cache clock cycles.
The assertion of a CSTRTL line by one of the CEs also resets, via OR gate 302, divide-by-32 counter 304. After being reset, if counter 304 reaches a count of thirty-two, indicating that no new requests for access have been received by cache quadrant W in 32 cache cycles, the counter asser-ts a RESET signal on line 317, causing counters 314 and 316 to be reset. (This assures that the system is always in the same priority state at startup, making it possible to reliably repeat an execution sequence which caused an abnormal cessation of system activity.) In the second cache cycle, CE Select Encoder 318 uses the PM and CEREQ signals to select the CE whose access request is to be granted, and asserts the appropriate WS select lines going to backplane switch 14. Similarly, CBUSYL Generation Logic 310 asserts the CEGRANT line of the successful CE (signalling the cache 3a q~ladrant to accept an address from that CE3 and the CBUSYL lines of the losing CEs. CEs receiving CBUSYL
continue to assert CSTRTL in subsequent cycles ~if they continue to desire access to that quadrant). The cache quadrant continues to assert CBUSYL in subsequent cycles until it is able to accept an address from the CE in question or until the CE removes CSTRTL. When CBUSYL is removed, the address transfer takes place in the same cycle. A CE must remove its CSTRTL in the cycle following transfer of an address, or the cache quadrant will interpret its presence as a new cache access request. A cache access may be aborted by a CE by removing the CSTRTL assertion in the second cycle--the cycle in which the address transfer is to take place.
The preferred interleaving is shown in Table 1:
Table 1 Cache Cache ADR(5:3) ADR~X3 Quadrant Board O o o O W O
0 0 1 l X 0 The interleaving scheme provides excellent cache accessing efficiency (i.e. a minimum of wasted cache access cycles due to more than one CE attempting to access the same cache) when a plurality of CEs are concurrently processing a program that accesses memory in strides of one and two quadwords (as well as larger strides that are divisible by two but not by four; e.g.
strides of six and ten quadwords). A memory stride is the address spacing between successive memory accesses by software operating on the system. Scientific and engineering software, for which the present system is i8~
best suited, typically accesses memory with strides of both one and two (a stride of ten is also relatively common).
What is advantageous about the interleaving scheme of Table l is that it produces a stride-of-one access pattern (WXXWYZZY) that forces the CEs into a phase relationship (termed "lockstep") in which there are a minimum of (and ideally no) wasted cache accesses. Lockstep is also achieved in a similar way for strides of two (and larger strides divisible by two but not by four). Table 2 shows the initial cache Table _ CE
W X X W Y Z Z Y W X X W Y Z Z Y
2 # W # X X W ~ Z Z Y W X X W Y Z
3 # # W # # X X W Y Z Z Y W X X W
clock. Each cache quadrant can ordinarily complete one ~3~2S~L
quadword (64-bit) read or write access in each 85 nanosecond cycle, resulting in a total peak throughput in the vicinity of 375 megabytes per second.
The block size of the cache (the smallest amount of data moved between the cache and memory) is 32 bytes. The data of each block is interleaved between two quadrants W,X on the same cache board so that for any 32-byte block quadwords 0 and 3 (quadwords are 8-byte words) are contained in quadrant W and quadwords 1 and 2 in quadrant X. Contiguous 32-b~te blocks are interleaved between cache boards so that quadwords 4 and 7 are located in quadrant ~ and quadwords 5 and 6 in quadrant Z.
The cache quadrants receive a 28-bit physical 15 byte address from the CEs. Two bits (bits 3 and 4) of the address specify the desired quadword within the cache block. Because quadwords 0 and 3 are located in one quadrant and quadwords 1 and 2 are located in the other, a CE-supplied CADR4X3 signal ~see below), which is the Exclusive OR of bits 3 and 4, is used to determine which cache quadrant on a CP cache board contains the desired data. The CE also supplies a normal address bit 4 so that the selected cache quadrant can determine which of its two words is required.
Bit 5 of the address (supplied as CADR5 by the requesting CE; see below) indicates which of the two CP
cache boards in a four-way interleaved system contains the desired block of data.
The CEs are connected to the cache quadrants by ~,t~` 30 backplane switch 14, as shown in Fig. 2. Forty-eight ` control lines ~4~} extend across the backplane directly from the CEs to the cache guadrants, without passing through the backplane switch. Six control lines are connected to each CE, for a total of forty-eight control ~3~25~3~
lines, all of which are are connected to each quadrant.
The si~ control lines for each CE are: CSTRTL, CBUSYL, CWAITL, CADR5, CADR4X3, and CDTPAR. Collectively, the six control lines are denoted AC for CE0, BC for CEl, CC
for CE2, and so on. CBUSYL, CWAITL, and CDTPAR are implemented as open-collector signals.
Ninety-six lines extend to the backplane switch from each CE and each cache quadrant: sixty-four bidirectional data lines CDT(63:0), twenty-six address lines CADR(27:6,4,2:0), an address parity line CADRPAR, three CE operation lines CEOP(2:0), and two data-length lines CDLN(1 0). Each CE and cache quadrant also provides the backplane switch with enable lines, e.g. AE
and WE, for enabling data and addresses to be read from the ninety-six bidirectional lines connected to the switch. The cache quadrants provide the backplane switch with direction signals WD, XD, YD, ZD, which define the direction of data being transferred through the switch, and two sets of three-bit CE selection signals WS(2:0), XS(2:0), YS(2:0), ZS(2:0), which specify the CE to which, or from which, a transfer is being made. Only one set of selection signals is shown in the drawings, but separate 3-bit selection signals are provided for addresses and data.
Referring to Fig. 3, control lines CSTRTL, CADR5, and CADR4X3 (one of each from each CE) enter cache board 12 and, within board 12, cache quadrant W
where they serve as inputs to a CE-to-central-processing-cache ~CPC) decoder 300. Decoder 300 outputs eight CE
request lines CEREQ(7:0) which indicate which of the eight CEs are currently attempting to access quadrant W. For each CE requesting access, a corresponding CEREQ
line is asserted. CEREQ(7 0) is one of the inputs into CBUSYL generation logic 310; the other input to CBUSYL
~L3~5i9~
generation logic 310 is the priority mode signal PM(1:0~, which indicates which pair of CEs ~e.g. pair CEO, CEl) currently has the highest priority level.
PM(1:0) is generated by priority mode logic 312 on cache 5 board 12. Priority mode logic 312 is shared by the two cache quadrants 15 on each cache board 12, and is replicated on each cache board. Logic 312 includes a divide-by-14 counter 314 which supplies, once every fourteen clock cycles, a pulse for clocking a mode 10 counter 316. Counter 316 generates PM(1:0), cycling continuously through the sequence "00", "01", "10", "11". Using CEREQ(7:0) and PM(1:0), CBUSYL Generation logic 310 generates CBUSYL (7:0), which goes to the CEs, and CEGRANT (7:0), which connects to another section of 15 the internal logic of the cache quadrant. CEREQ(7:0) and PM(1:0) are also inputs to CE Select Encoder 318, which generates three bit selection signal WS(2:0).
Signals CSTRTL(7:0) are also inputs to an eight-inpu~ OR
gate 302 whose output is the reset input to divide-by-32 20 counter 304. Divide-by-32 counter 304 is connected to the RESET inputs of divide-by-14 counter 314 and mode counter 316. Decoder 300, logic 310, logic 318, OR gate 302, counter 304, and accompanying signal lines are replicated in each of the cache quadrants 15.
In operation, a CE initiates a cache operation by asserting its CSTRTL line and supplying the two address bits CADR5 and CADR4X3 specifying which of the four interleaved ~uadrants of the cache are involved in the operation. CADR5 is the fifth bit of the memory address involved in the operation being initiated by the CE. CADR4X3 is the exclusive-OR combination of bits three and four of the memory address. The reason that these two bits are combined in addressing the cache quadrants has to do with the way in which the cache ~3~)25~3~
quadrants are interleaved (see discussion below). If only two cache quadrants are in use (e.g. if only four CEs are in use), only address bit CADR4X3 is used to address a particular quadrant, and CADR5 is saved by the 5 addressed cache quadrant for use with the remainder of the address transferred through the backplane switch.
As many as eight CEs may simultaneously bid for access to any of the four cache quadrants. Access conflicts are resolved by the cache quadrants. In the 10 same 85 nanosecond cycle as the CSTRTL is asserted (the first cache cycle), the cache quadrant addressed resolves which of the contending CEs has priority.
Referring to Fig. 3, for each CE that bids for access to a cache quadrant, one of the eight CSTRTL lines entering 15 cache quadrant W will be asserted. For each CSTRTL line which is asserted, if the corresponding CADR5 and CADR4X3 lines indicate that it is cache quadrant W which the CE wishes to access, the corresponding CEREQ line will be asserted by decoder 300. As few as zero and as 20 many as eight of the CEREQ lines may thus be asserted.
The eight CEREQ lines indicate which of the CEs wish to access quadrant W.
When more than one CE is contending for access to the same quadrant, priority is given to the CE pair 25 specified by the priority signal PM(l:0). (See below for discussion of why priority changes are made in pairs.) The priority mode signal PM(l:0) indicates which of the four pairs of CEs ("00" signifying CE0, CEl; "11", CE2, CE3; "10", CE4, CE5, "01", CE6, CE7) 30 currently has the highes-t priority level. Within each pair of CEs, the CE having the lowest numerical designation prevails (thus CE0 prevails over CEl).
The priority mode signal is initialized on startup and incremented every 14 cache cycles. At startup, prior to the assertion of one of the CSTRTL
lines, divide-by-14 counter 314 and mode counter 316 are reset so that PM(l:G) is set to "00", which designates the pair CE0, CEl as the highest priority pair, the pair 5 CE2, CE3 as the next highest, and so on. Every fourteen cache clock cycles, divide-by-14 counter 314 supplies a pulse on line 326 to mode counter 316, causing it to increment PM(1:0) by one ~e.g. from 00 to 01). So long as counter 316 is not reset, it will continuously cycle PM(1:0) through the four priority modes (i.e. 00, 01, 10, 11), changing the mode once every fourteen cache clock cycles.
The assertion of a CSTRTL line by one of the CEs also resets, via OR gate 302, divide-by-32 counter 304. After being reset, if counter 304 reaches a count of thirty-two, indicating that no new requests for access have been received by cache quadrant W in 32 cache cycles, the counter asser-ts a RESET signal on line 317, causing counters 314 and 316 to be reset. (This assures that the system is always in the same priority state at startup, making it possible to reliably repeat an execution sequence which caused an abnormal cessation of system activity.) In the second cache cycle, CE Select Encoder 318 uses the PM and CEREQ signals to select the CE whose access request is to be granted, and asserts the appropriate WS select lines going to backplane switch 14. Similarly, CBUSYL Generation Logic 310 asserts the CEGRANT line of the successful CE (signalling the cache 3a q~ladrant to accept an address from that CE3 and the CBUSYL lines of the losing CEs. CEs receiving CBUSYL
continue to assert CSTRTL in subsequent cycles ~if they continue to desire access to that quadrant). The cache quadrant continues to assert CBUSYL in subsequent cycles until it is able to accept an address from the CE in question or until the CE removes CSTRTL. When CBUSYL is removed, the address transfer takes place in the same cycle. A CE must remove its CSTRTL in the cycle following transfer of an address, or the cache quadrant will interpret its presence as a new cache access request. A cache access may be aborted by a CE by removing the CSTRTL assertion in the second cycle--the cycle in which the address transfer is to take place.
The preferred interleaving is shown in Table 1:
Table 1 Cache Cache ADR(5:3) ADR~X3 Quadrant Board O o o O W O
0 0 1 l X 0 The interleaving scheme provides excellent cache accessing efficiency (i.e. a minimum of wasted cache access cycles due to more than one CE attempting to access the same cache) when a plurality of CEs are concurrently processing a program that accesses memory in strides of one and two quadwords (as well as larger strides that are divisible by two but not by four; e.g.
strides of six and ten quadwords). A memory stride is the address spacing between successive memory accesses by software operating on the system. Scientific and engineering software, for which the present system is i8~
best suited, typically accesses memory with strides of both one and two (a stride of ten is also relatively common).
What is advantageous about the interleaving scheme of Table l is that it produces a stride-of-one access pattern (WXXWYZZY) that forces the CEs into a phase relationship (termed "lockstep") in which there are a minimum of (and ideally no) wasted cache accesses. Lockstep is also achieved in a similar way for strides of two (and larger strides divisible by two but not by four). Table 2 shows the initial cache Table _ CE
W X X W Y Z Z Y W X X W Y Z Z Y
2 # W # X X W ~ Z Z Y W X X W Y Z
3 # # W # # X X W Y Z Z Y W X X W
4 # # # # W # # X X W ~ Z Z Y W X
accesses in the four CEs for a stride of one. In the hypothetical shown, the four CEs initially attempt to access the same cache quadrant in the first cycle (something approximating what could occur at initiation o~ vector concurrent processing). Access conflicts force CE2, CE3, and CE4 (which, at startup, have lower priority levels than CEl) to to be locked out of some accesses (indicated by a #) until each reaches a phase relationship in which no further conflicts arise, wherein each CE's access pattern is offset by an even number of cycles from that of the CE with the next highest priority. This phase relationship will be reached by all the CEs no matter what their initial phase relationship.
The cache accessing sequence shown in Table 2 actually shows only half of the accesses that occur during a typical initiation of vector-concurrent ~3~258g~
processing on an eight-CE system (with all eight CEs participating in the concurrency complex). The sequence of Table 2 depicts the accessing occurring every other 85 nanosecond cache cycle for four of the eight CEs, the same accessing pattern occurs during the other cycles for the other four CEs. ~ssuming for simplicity that all eight CEs initially contend for the same cache quadrant, those that lose on the first cycle will contend on the very next cache cycle (not shown in the Table 2 sequence). One CE will win during that cycle, and the remaining ones will contend during the third cache cycle (the second cycle shown in Table 2). One by one the eight CEs will find a niche on one or the other of the alternating cache cycles, with the result that l~ four will access the cache on odd cache cycles, and another four on even cycles (with the resulting assignment to odd or even cycles being random).
The superior performance of the preferred interleaving is most pronounced when seven to eight CEs are part of the concurrency complex.
This superior perf~or~ance assumes, of course, that lockstep is achieved. This is not always the case. Frequenk cache misses and cache access conflicts (as the result of software that does not access memory at the expected strides) will prevent lockstep from occurring. When this happéns, the lowest priority processors (CE6, CE7 in the absence of any priority rotation) have the greatest difficulty accessing the cache quadrants, and thus take longer to complete tasks. The resulting difference in processing time can result in substan~ial inefficiency when the processors have been assigned comparable tasks and each is required to wait until all the others have completed their tasks before proceeding with a new task. In the absence of ~3~
priority rotation, the highest priority processors (CE0, CEl) would finish their tasks much earlier than the lowest priority ones (CE6, CE7) and would unproductively remain idle until the lowest priority processors are finished. Examples of such situations are: (1) vector concurrent operations (i.e., operations in which the elements of a vector are divided evenly between the eight CEs for concurrent processing~ where lockstep does not occur; (2) scalar concurrent operations where the 10 number of iterations equals the number of CEs in the concurrency complex; (3) situations where all eight CEs are assigned different (but e~ual size) jobs and the operating software waits for all to be completed before proceeding.
An illustration of how cache misses can result in greater processing delays for lower priority processors is given in Table 3. The lockstep phase relationship is disturbed when there is a cache miss because each such miss has a ripple down effect, such that delays are introduced in the access patterns of all CEs having a lower priority level until the patterns are forced back into locks-tep. An example of a s~ngle cache miss and the resulting pattern of delays is i~ ustrated (for the simplified use of processors) in Tab~lë 3 (a "*"
indicates a cache miss):
Table 3 CE
W * X X W Y Z
2 Z Y W # X X W
3 y Z Z Y # W X
4 X W Y Z Z # Y
The illustrated miss causes a single clock cycle delay for each of the CEs having a lower priority level. The lower priority CEs experience cumulative delays from - ~ 3~2~ii8~
misses occurring in higher priority CEs. Thus, for example, the pair of CEs having the lowest priority level (CE6, CE7) accumulates delays whenever a cache miss was experienced by any other CE, whereas the pair of CEs having the highest priority level (CE0, CEl) experiences only those delays associated with their own mlsses.
The rotation of priority levels achieved by the invention is intended to prevent the burden of the delays caused by cache misses and cache access ccnflicts from being disproportionately borne by the eight CEs.
An uneven distribution of the burden will cause an undesirable lopsidedness in the time required for the eight CEs to finish jobs of equivalent length. This condition, in which the CEs having lower priority levels accumulate proportionately more delays, is ter~ed the "tail effect". Where cache misses or cache access conflicts are occurring frequently the tail effec-t becomes significant (e.g., the same task might take 165 clock cycles on CEO but 231 cycles on CE7).
Rotation of the priority levels causes delays to be distributed more evenly among the CEs and -thus there is a significant reduction in the tail effect (e.g. in the example just given, the difference in processing time is reduced from sixty-six to thirty-seven clock cycles). While the CE having the lowest priority level will still accumulate the most delays, any given CE retains the lowest priority level for only fourteen cycles, after which it acquires the highest priority level.
Competing considerations govern selection of the length of the period between priority rotations.
The period should be long enough so as not to impact substantially on achievement of lockstep in applications ~.3~25~
where lockstep is possible (if a priority rotation occurs in the transition period before lockstep has ~een fully achieved, there will likely be delay in reaching lockstep), but short enough to reduce the tail effect in 5 applications where lockstep cannot be achieved (too long a period will provide little or no averaging of delay between processors for tasks comparable in length to the period length). The lockstep phase relationship is not disturbed by the rotation of the priority levels after lockstep has been initially achieved, because priority levels only have an effect when more than one CE seeks access to the same cache quadrant.
For the disclosed embodiment, in which there are eight CEs and four cache quadrants (clocked at twice 15 the frequency of the CEs, and thus capable of handling two accesses in one processor cycle), a fourteen cache cycle period between priority rotation is optimal.
The priority rotation is done in pairs (i.e., the highest priority pair of processors become the lowest priority pair, and so on) because adjacent processors (e.g., CE0 and CEl) do not significantly compete with one another for cache access, and~thus effectively have the same priority. This happens because the caches, which operate at twice the speed of 25 the processors, are able to service two processors "simultaneously", i.e., permit two processors to access the cache in one processor cycle. Thus, for example, if both CE0 and OE l were simultaneously to access the same cache quadrant, the fact that CEl has lower priority 30 than CE0 would not ordinarily prevent CEl from achieving access.
Other embodiments are within the following claims.
accesses in the four CEs for a stride of one. In the hypothetical shown, the four CEs initially attempt to access the same cache quadrant in the first cycle (something approximating what could occur at initiation o~ vector concurrent processing). Access conflicts force CE2, CE3, and CE4 (which, at startup, have lower priority levels than CEl) to to be locked out of some accesses (indicated by a #) until each reaches a phase relationship in which no further conflicts arise, wherein each CE's access pattern is offset by an even number of cycles from that of the CE with the next highest priority. This phase relationship will be reached by all the CEs no matter what their initial phase relationship.
The cache accessing sequence shown in Table 2 actually shows only half of the accesses that occur during a typical initiation of vector-concurrent ~3~258g~
processing on an eight-CE system (with all eight CEs participating in the concurrency complex). The sequence of Table 2 depicts the accessing occurring every other 85 nanosecond cache cycle for four of the eight CEs, the same accessing pattern occurs during the other cycles for the other four CEs. ~ssuming for simplicity that all eight CEs initially contend for the same cache quadrant, those that lose on the first cycle will contend on the very next cache cycle (not shown in the Table 2 sequence). One CE will win during that cycle, and the remaining ones will contend during the third cache cycle (the second cycle shown in Table 2). One by one the eight CEs will find a niche on one or the other of the alternating cache cycles, with the result that l~ four will access the cache on odd cache cycles, and another four on even cycles (with the resulting assignment to odd or even cycles being random).
The superior performance of the preferred interleaving is most pronounced when seven to eight CEs are part of the concurrency complex.
This superior perf~or~ance assumes, of course, that lockstep is achieved. This is not always the case. Frequenk cache misses and cache access conflicts (as the result of software that does not access memory at the expected strides) will prevent lockstep from occurring. When this happéns, the lowest priority processors (CE6, CE7 in the absence of any priority rotation) have the greatest difficulty accessing the cache quadrants, and thus take longer to complete tasks. The resulting difference in processing time can result in substan~ial inefficiency when the processors have been assigned comparable tasks and each is required to wait until all the others have completed their tasks before proceeding with a new task. In the absence of ~3~
priority rotation, the highest priority processors (CE0, CEl) would finish their tasks much earlier than the lowest priority ones (CE6, CE7) and would unproductively remain idle until the lowest priority processors are finished. Examples of such situations are: (1) vector concurrent operations (i.e., operations in which the elements of a vector are divided evenly between the eight CEs for concurrent processing~ where lockstep does not occur; (2) scalar concurrent operations where the 10 number of iterations equals the number of CEs in the concurrency complex; (3) situations where all eight CEs are assigned different (but e~ual size) jobs and the operating software waits for all to be completed before proceeding.
An illustration of how cache misses can result in greater processing delays for lower priority processors is given in Table 3. The lockstep phase relationship is disturbed when there is a cache miss because each such miss has a ripple down effect, such that delays are introduced in the access patterns of all CEs having a lower priority level until the patterns are forced back into locks-tep. An example of a s~ngle cache miss and the resulting pattern of delays is i~ ustrated (for the simplified use of processors) in Tab~lë 3 (a "*"
indicates a cache miss):
Table 3 CE
W * X X W Y Z
2 Z Y W # X X W
3 y Z Z Y # W X
4 X W Y Z Z # Y
The illustrated miss causes a single clock cycle delay for each of the CEs having a lower priority level. The lower priority CEs experience cumulative delays from - ~ 3~2~ii8~
misses occurring in higher priority CEs. Thus, for example, the pair of CEs having the lowest priority level (CE6, CE7) accumulates delays whenever a cache miss was experienced by any other CE, whereas the pair of CEs having the highest priority level (CE0, CEl) experiences only those delays associated with their own mlsses.
The rotation of priority levels achieved by the invention is intended to prevent the burden of the delays caused by cache misses and cache access ccnflicts from being disproportionately borne by the eight CEs.
An uneven distribution of the burden will cause an undesirable lopsidedness in the time required for the eight CEs to finish jobs of equivalent length. This condition, in which the CEs having lower priority levels accumulate proportionately more delays, is ter~ed the "tail effect". Where cache misses or cache access conflicts are occurring frequently the tail effec-t becomes significant (e.g., the same task might take 165 clock cycles on CEO but 231 cycles on CE7).
Rotation of the priority levels causes delays to be distributed more evenly among the CEs and -thus there is a significant reduction in the tail effect (e.g. in the example just given, the difference in processing time is reduced from sixty-six to thirty-seven clock cycles). While the CE having the lowest priority level will still accumulate the most delays, any given CE retains the lowest priority level for only fourteen cycles, after which it acquires the highest priority level.
Competing considerations govern selection of the length of the period between priority rotations.
The period should be long enough so as not to impact substantially on achievement of lockstep in applications ~.3~25~
where lockstep is possible (if a priority rotation occurs in the transition period before lockstep has ~een fully achieved, there will likely be delay in reaching lockstep), but short enough to reduce the tail effect in 5 applications where lockstep cannot be achieved (too long a period will provide little or no averaging of delay between processors for tasks comparable in length to the period length). The lockstep phase relationship is not disturbed by the rotation of the priority levels after lockstep has been initially achieved, because priority levels only have an effect when more than one CE seeks access to the same cache quadrant.
For the disclosed embodiment, in which there are eight CEs and four cache quadrants (clocked at twice 15 the frequency of the CEs, and thus capable of handling two accesses in one processor cycle), a fourteen cache cycle period between priority rotation is optimal.
The priority rotation is done in pairs (i.e., the highest priority pair of processors become the lowest priority pair, and so on) because adjacent processors (e.g., CE0 and CEl) do not significantly compete with one another for cache access, and~thus effectively have the same priority. This happens because the caches, which operate at twice the speed of 25 the processors, are able to service two processors "simultaneously", i.e., permit two processors to access the cache in one processor cycle. Thus, for example, if both CE0 and OE l were simultaneously to access the same cache quadrant, the fact that CEl has lower priority 30 than CE0 would not ordinarily prevent CEl from achieving access.
Other embodiments are within the following claims.
Claims (14)
1. A digital processing system, comprising: a plurality of memory elements, a plurality of parallel processors, said processors each having means for initiating an access of data from said memory elements simultaneously with accesses of other processors, said memory elements being interleaved (i.e., each is assigned memory addresses on the basis of a low order portion of the memory address) so that the access patterns generated by said processors when accessing data from said memory elements at a predetermined stride permits said processors to reach a phase relationship known as "lockstep") with other said processors in which each said processor is able to access a different said memory element simultaneously without creating access conflicts, means for deciding, based on fixed priority levels, which said processor is granted access to a given memory element when there are simultaneous access requests to the given memory element, and priority-altering means for altering at a predetermined periodic interval said fixed priority levels to vary the highest priority among said processors, said predetermined periodic interval being selected so as not to lengthen substantially the time required for said processors to fall into lockstep and said predetermined periodic interval being selected so as to reduce differences in the times required by each of said processors to complete comparable tasks not amenable to lockstep execution.
2. The digital processing system of claim 1 wherein said priority-altering means comprises means for varying priority by rotation of processors (or sets of processors) between priority levels.
3. The digital processing system of claim 2 wherein said priority-altering means comprises means for causing the processor (or set of processors) with the lowest priority to shift immediately to the highest priority, and for the other processors (or sets of processors) to shift to the next lowest priority level, each time said priority levels are altered.
4. The digital processing system of claim 1 wherein said priority-altering means comprises means for varying priority in sets of processors where the number of processors within a set equals the number of processors that can successfully access a said memory element in one processor cycle.
5. The digital processing system of claim 1 or 3 wherein said memory elements operate at N times the speed of said processors (i.e., N memory cycles equal one processor cycle), and wherein said processors are divided into sets each consisting of N
processors, within which the priority level is the same.
processors, within which the priority level is the same.
6. The digital processing system of claim 5 wherein N is equal to two and said priority-altering means includes means for altering said priority after completion of 14 memory cycles.
7. The digital processing system of claim 6 wherein there are eight said processors and four said memory elements.
8. The digital processing system of claim 1 or 3 wherein said priority-altering means includes means for programmably adjusting the period between priority alterations.
9. The digital processing system of claim 1 or 3, wherein said memory elements are caches.
10. The digital processing system of claim 1 or 3 wherein said memory elements are so interleaved that the access pattern generated by said processors for a given power-of-two stride greater than one meets the condition that the pattern tolerates being offset with respect to an identical pattern by an OFFSET or any multiple of said OFFSET, said OFFSET being equal to the length of the access pattern divided by the number of memory elements (wherein tolerating means that no access conflicts arise).
11. The digital processing system of claim 10 wherein said memory elements are so interleaved that the access patterns generated by said processors for said given stride meet the further condition that the patterns include at least one conflict at every offset other than said OFFSET and multiples of said OFFSET, and said conflicts force said processors to assume a phase relationship with each other wherein the offset between access patterns equals said OFFSET or a multiple thereof.
12. The digital processing system of claim 1 or 3 wherein said memory elements are so interleaved that the access pattern generated by said processors for a given power-of-two stride greater than one meets the condition that the pattern tolerates being offset with respect to an identical pattern by an OFFSET or any multiple of said OFFSET, said OFFSET being equal to the length of the access pattern divided by the number of memory elements (wherein tolerating means that no access conflicts arise), wherein said memory elements are so interleaved that the access pattern generated by said processors for said given stride meets the further condition that the pattern includes at least one conflict at every offset other than said OFFSET and multiples of said OFFSET, whereby said conflicts force said processors to assume a phase relationship with each other wherein the offset between access patterns equals said OFFSET or a multiple thereof, and wherein there are four said memory elements W,X,Y,Z and said memory elements are interleaved so that a stride of one of the access pattern is WXXWYZZY and at a stride of two the pattern is WXYZ.
13. The digital processing system of claim 11 wherein there are four memory elements W,X,Y,Z and said memory elements are interleaved as follows:
Address Bits on Which Memory Elements are Interleaved Memory Element 1 0 0 y
Address Bits on Which Memory Elements are Interleaved Memory Element 1 0 0 y
14. The digital processing system of claim 1 further comprising means for resetting the priority levels to an initial ranking after a preselected period of time has elapsed without an access request to any memory element.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US7975687A | 1987-07-30 | 1987-07-30 | |
| US079,756 | 1987-07-30 |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CA1302584C true CA1302584C (en) | 1992-06-02 |
Family
ID=22152608
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA000573400A Expired - Lifetime CA1302584C (en) | 1987-07-30 | 1988-07-29 | Parallel processing computer in which memory access priorities are varied |
Country Status (5)
| Country | Link |
|---|---|
| US (1) | US5133059A (en) |
| EP (1) | EP0301921A3 (en) |
| JP (1) | JPS6484361A (en) |
| AU (1) | AU2022188A (en) |
| CA (1) | CA1302584C (en) |
Families Citing this family (41)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5371874A (en) * | 1989-01-27 | 1994-12-06 | Digital Equipment Corporation | Write-read/write-pass memory subsystem cycle |
| US5265257A (en) * | 1990-06-22 | 1993-11-23 | Digital Equipment Corporation | Fast arbiter having easy scaling for large numbers of requesters, large numbers of resource types with multiple instances of each type, and selectable queuing disciplines |
| US5724548A (en) * | 1990-09-18 | 1998-03-03 | Fujitsu Limited | System including processor and cache memory and method of controlling the cache memory |
| JP2575557B2 (en) * | 1990-11-13 | 1997-01-29 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Super computer system |
| US5282272A (en) * | 1990-12-21 | 1994-01-25 | Intel Corporation | Interrupt distribution scheme for a computer bus |
| US5341485A (en) * | 1991-05-07 | 1994-08-23 | International Business Machines Corporation | Multiple virtual address translation per computer cycle |
| US5388237A (en) * | 1991-12-30 | 1995-02-07 | Sun Microsystems, Inc. | Method of and apparatus for interleaving multiple-channel DMA operations |
| US5301283A (en) * | 1992-04-16 | 1994-04-05 | Digital Equipment Corporation | Dynamic arbitration for system bus control in multiprocessor data processing system |
| US5379393A (en) * | 1992-05-14 | 1995-01-03 | The Board Of Governors For Higher Education, State Of Rhode Island And Providence Plantations | Cache memory system for vector processing |
| EP0575651A1 (en) * | 1992-06-24 | 1993-12-29 | International Business Machines Corporation | Multiprocessor system |
| US5845329A (en) * | 1993-01-29 | 1998-12-01 | Sanyo Electric Co., Ltd. | Parallel computer |
| US5890222A (en) * | 1995-01-04 | 1999-03-30 | International Business Machines Corporation | Method and system for addressing registers in a data processing unit in an indirect addressing mode |
| US5832533A (en) * | 1995-01-04 | 1998-11-03 | International Business Machines Corporation | Method and system for addressing registers in a data processing unit in an indexed addressing mode |
| US5887183A (en) * | 1995-01-04 | 1999-03-23 | International Business Machines Corporation | Method and system in a data processing system for loading and storing vectors in a plurality of modes |
| US5680338A (en) * | 1995-01-04 | 1997-10-21 | International Business Machines Corporation | Method and system for vector processing utilizing selected vector elements |
| US5564062A (en) * | 1995-03-31 | 1996-10-08 | International Business Machines Corporation | Resource arbitration system with resource checking and lockout avoidance |
| US5983327A (en) | 1995-12-01 | 1999-11-09 | Nortel Networks Corporation | Data path architecture and arbitration scheme for providing access to a shared system resource |
| US5974506A (en) * | 1996-06-28 | 1999-10-26 | Digital Equipment Corporation | Enabling mirror, nonmirror and partial mirror cache modes in a dual cache system |
| US6167486A (en) | 1996-11-18 | 2000-12-26 | Nec Electronics, Inc. | Parallel access virtual channel memory system with cacheable channels |
| US5909561A (en) * | 1997-04-14 | 1999-06-01 | International Business Machines Corporation | Apparatus and method for separately layering cache and architectural specific functions in different operational controllers to facilitate design extension |
| US6061755A (en) * | 1997-04-14 | 2000-05-09 | International Business Machines Corporation | Method of layering cache and architectural specific functions to promote operation symmetry |
| US6173351B1 (en) * | 1998-06-15 | 2001-01-09 | Sun Microsystems, Inc. | Multi-processor system bridge |
| GB2385174B (en) * | 1999-01-19 | 2003-11-26 | Advanced Risc Mach Ltd | Memory control within data processing systems |
| US6708254B2 (en) | 1999-11-10 | 2004-03-16 | Nec Electronics America, Inc. | Parallel access virtual channel memory system |
| GB2378277B (en) * | 2001-07-31 | 2003-06-25 | Sun Microsystems Inc | Multiple address translations |
| US7133972B2 (en) * | 2002-06-07 | 2006-11-07 | Micron Technology, Inc. | Memory hub with internal cache and/or memory access prediction |
| US7117316B2 (en) | 2002-08-05 | 2006-10-03 | Micron Technology, Inc. | Memory hub and access method having internal row caching |
| US6820181B2 (en) | 2002-08-29 | 2004-11-16 | Micron Technology, Inc. | Method and system for controlling memory accesses to memory modules having a memory hub architecture |
| US7120727B2 (en) | 2003-06-19 | 2006-10-10 | Micron Technology, Inc. | Reconfigurable memory module and method |
| US7107415B2 (en) | 2003-06-20 | 2006-09-12 | Micron Technology, Inc. | Posted write buffers and methods of posting write requests in memory modules |
| US7260685B2 (en) | 2003-06-20 | 2007-08-21 | Micron Technology, Inc. | Memory hub and access method having internal prefetch buffers |
| US7120743B2 (en) * | 2003-10-20 | 2006-10-10 | Micron Technology, Inc. | Arbitration system and method for memory responses in a hub-based memory system |
| US7330992B2 (en) | 2003-12-29 | 2008-02-12 | Micron Technology, Inc. | System and method for read synchronization of memory modules |
| US7188219B2 (en) * | 2004-01-30 | 2007-03-06 | Micron Technology, Inc. | Buffer control system and method for a memory system having outstanding read and write request buffers |
| US7213082B2 (en) * | 2004-03-29 | 2007-05-01 | Micron Technology, Inc. | Memory hub and method for providing memory sequencing hints |
| US7162567B2 (en) | 2004-05-14 | 2007-01-09 | Micron Technology, Inc. | Memory hub and method for memory sequencing |
| US7519788B2 (en) | 2004-06-04 | 2009-04-14 | Micron Technology, Inc. | System and method for an asynchronous data buffer having buffer write and read pointers |
| US20060168407A1 (en) * | 2005-01-26 | 2006-07-27 | Micron Technology, Inc. | Memory hub system and method having large virtual page size |
| US20080091924A1 (en) * | 2006-10-13 | 2008-04-17 | Jouppi Norman P | Vector processor and system for vector processing |
| US9304714B2 (en) * | 2012-04-20 | 2016-04-05 | Violin Memory Inc | LUN management with distributed RAID controllers |
| US9389807B2 (en) | 2014-01-23 | 2016-07-12 | International Business Machines Corporation | Conflict management for application directed data placement in storage environments |
Family Cites Families (18)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US4048623A (en) * | 1974-09-25 | 1977-09-13 | Data General Corporation | Data processing system |
| US3931613A (en) * | 1974-09-25 | 1976-01-06 | Data General Corporation | Data processing system |
| US4009470A (en) * | 1975-02-18 | 1977-02-22 | Sperry Rand Corporation | Pre-emptive, rotational priority system |
| US4232294A (en) * | 1979-04-30 | 1980-11-04 | Control Data Corporation | Method and apparatus for rotating priorities between stations sharing a communication channel |
| US4293910A (en) * | 1979-07-02 | 1981-10-06 | International Business Machines Corporation | Reconfigurable key-in-storage means for protecting interleaved main storage |
| US4449183A (en) * | 1979-07-09 | 1984-05-15 | Digital Equipment Corporation | Arbitration scheme for a multiported shared functional device for use in multiprocessing systems |
| JPS5619575A (en) * | 1979-07-25 | 1981-02-24 | Fujitsu Ltd | Data processing system having hierarchy memory |
| WO1983001135A1 (en) * | 1981-09-18 | 1983-03-31 | Rovsing As Christian | Multiprocessor computer system |
| US4586133A (en) * | 1983-04-05 | 1986-04-29 | Burroughs Corporation | Multilevel controller for a cache memory interface in a multiprocessing system |
| US4636942A (en) * | 1983-04-25 | 1987-01-13 | Cray Research, Inc. | Computer vector multiprocessing control |
| US4661900A (en) * | 1983-04-25 | 1987-04-28 | Cray Research, Inc. | Flexible chaining in vector processor with selective use of vector registers as operand and result registers |
| US4638431A (en) * | 1984-09-17 | 1987-01-20 | Nec Corporation | Data processing system for vector processing having a cache invalidation control unit |
| EP0214718A3 (en) * | 1985-07-22 | 1990-04-04 | Alliant Computer Systems Corporation | Digital computer |
| US4794521A (en) * | 1985-07-22 | 1988-12-27 | Alliant Computer Systems Corporation | Digital computer with cache capable of concurrently handling multiple accesses from parallel processors |
| US4783736A (en) * | 1985-07-22 | 1988-11-08 | Alliant Computer Systems Corporation | Digital computer with multisection cache |
| US4663756A (en) * | 1985-08-29 | 1987-05-05 | Sperry Corporation | Multiple-use priority network |
| US4722046A (en) * | 1986-08-27 | 1988-01-26 | Amdahl Corporation | Cache storage priority |
| US4953081A (en) * | 1988-12-21 | 1990-08-28 | International Business Machines Corporation | Least recently used arbiter with programmable high priority mode and performance monitor |
-
1988
- 1988-07-29 JP JP63190491A patent/JPS6484361A/en active Pending
- 1988-07-29 AU AU20221/88A patent/AU2022188A/en not_active Abandoned
- 1988-07-29 CA CA000573400A patent/CA1302584C/en not_active Expired - Lifetime
- 1988-08-01 EP EP88307097A patent/EP0301921A3/en not_active Withdrawn
-
1991
- 1991-01-10 US US07/639,413 patent/US5133059A/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| US5133059A (en) | 1992-07-21 |
| JPS6484361A (en) | 1989-03-29 |
| EP0301921A2 (en) | 1989-02-01 |
| EP0301921A3 (en) | 1990-07-18 |
| AU2022188A (en) | 1989-02-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA1302584C (en) | Parallel processing computer in which memory access priorities are varied | |
| US5437042A (en) | Arrangement of DMA, interrupt and timer functions to implement symmetrical processing in a multiprocessor computer system | |
| US5392436A (en) | Two level system bus arbitration having lower priority multiprocessor arbitration and higher priority in a single processor and a plurality of bus masters arbitration | |
| US4783736A (en) | Digital computer with multisection cache | |
| US5369748A (en) | Bus arbitration in a dual-bus architecture where one bus has relatively high latency | |
| US5168568A (en) | Delaying arbitration of bus access in digital computers | |
| US5016167A (en) | Resource contention deadlock detection and prevention | |
| US5596729A (en) | First arbiter coupled to a first bus receiving requests from devices coupled to a second bus and controlled by a second arbiter on said second bus | |
| EP0325419B1 (en) | Method and apparatus for caching interlock variables in an integrated cache memory | |
| JP2742728B2 (en) | Reservation ignoring regular priority mediation of microprocessors in multi-process computer system | |
| US5282272A (en) | Interrupt distribution scheme for a computer bus | |
| KR910002929B1 (en) | Pipeline cache memory apparatus | |
| US5692216A (en) | Direct memory access controller having programmable timing | |
| EP0214718A2 (en) | Digital computer | |
| EP0325420A2 (en) | Organization of an integrated cache unit for flexible usage in cache system design | |
| JP2002182976A (en) | Dynamic serial conversion for memory access in multi- processor system | |
| EP0325421A2 (en) | Organization of an integrated cache unit for flexible usage in supporting multiprocessor operations | |
| US5796968A (en) | Bus arbiter used in a computer system | |
| US5625824A (en) | Circuit for selectively preventing a microprocessor from posting write cycles | |
| US7383336B2 (en) | Distributed shared resource management | |
| US5590338A (en) | Combined multiprocessor interrupt controller and interprocessor communication mechanism | |
| JP2504818B2 (en) | Common memory control method in multiprocessor device | |
| EP0923031A1 (en) | Method for reading data from a shared memory in a multiprocessor computer system | |
| CA1270064A (en) | Digital computer with interleaved memory elements | |
| Pri-Tal | MicroStandards |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| MKLA | Lapsed |