CA2022249A1 - Memory device - Google Patents

Abstract

ABSTRACT OF THE DISCLOSURE
A memory for storing data in a computer system.
Integrity of data transferred to or from a memory array is monitored by transferring two sets of EDC or ECC data corresponding to a longword of data between the memory array and two separate memory controllers. The probability of an undetected error is very low because the two sets of EDC or ECC data are compared to ensure that they match. The number of lines and pins used is minimized by multiplexing the EDC or ECC data with address signals and cycle type signals. The address and cycle type signals are placed on the time division multiplexed bidirectional lines at the beginning of a memory transfer cycle, and the EDC or ECC data is placed on these time division multiplexed lines at times when a longword of data is being transferred on a set of bidirectional data lines.

Description

2~,2~
~ç~o~ OF ~113~ I~NTrON
The lslventlon relato to th~ tran~fsr of in~orm~tion to and fro~n a memory array in a computcr 8y~te~.
rn any coml?uter system, the tran~er of d~ta to and from a memory array i8 a critic~l func~ion. It i3 es~ential that he co~puter 9y tam 5~ rante~ the ialtegrity of the data stored in and access~d from its m~mory array.
In ~ome application~ f or computer systemR, any error can have a di~astrou~ of fect. In th~s~ pplic~tions, such a~
computer 8ya1:em8 that control the op~ration of nucl~ar reactors, there i8 a need for fault tolar~nt ~y~
Typic~lly, the m~ory arrays ln these sy~tems communica~ce with two sep~rate but id~ntic~l 8~t~ of digital logic. Thi~
u~e of multiple compononts gu~ranteo~ An ex~remely lo~ ~
probability of us~detec~ed ~yst~ error~. :
In mo~t computer ~y8tællUI~ the integrity o datA
tran~farred to or from a me~ory array i8 monitored usir~g ~n erroE det0cting and~or correcting code ~EDC ox ECC), ~hsnevelr re~d or wr~te dAl:el 18 tr~n~a~rr~d, the EDS or ECC
data corresponding to that read or write d~ta al~o i~
tran~f0:~r~. I~l these co~nputer ~yste~ DC or ECC data that i~ ~ae~iv~d ~%oqll the ~omputer ~y~tem or ~cce~ ~d from ~torage by th~ D~ory arr~y i13 oompa~s~ with anoth~r ~et of ~C or ~CC dat~ that i~ ~enerated locally by the me~aory ~rr~y u~ing the resd or wri~e da~a that i3 belng tr n~ferred. If the gen~rated ~DC or ECC d~a ~atch~ ~he received or ~tor~d EDC
L~W orrlce-'`;NEG'\I~i, HEI`IDEFL50N
5F.~.R.~90W C~RRE~r ' i6 DUNNER
'~'i177~r~T.r~ w. -- 1 --O~J. O C ~000 1~0~ 0-50 ",~

'';

.. ' : ,,.:', , : ' ~ ' ': ': ' . ': ', , ~ :
. ~ ~ ' - ,' ,:
~,:..... ' : :'~ '' ' I 2~22~9 or ~CC data, the prob~bility of an undetected error being p~e~snt in the read or WTita dAta i0 very low.
In faul~ toler~nt 9y~t~, r~liability of d~ can be further increased by tran~ferring the ~ams d~ta u~ing ~he two ~eparat~ ~et3 of component~. Preferably, the logic i8 each ~et i~ i301at~d from fault~ that ~re lik~ly to occur in the other se~. In thiQ w~y, undetected ~rrors can be prevented by co~paring the data tran~ferred by ona ~et of logic to th~
data tran~ferr~d using th~ oth0r s~t of logic.
Although th~se techniques ar~ important i~ improving data integrity and preventing undatectsd error~ fro~
occurring, th~re al80 ar3 rea80n8 to mlnimize th~ nu~ber of :-¦ lines ~nd pin~ u~d in th~ computer sy~te~. In particul~r, ¦
the msch~nical reli~bility of th~ y~e~ i~ raduced, and ~he amount of hArdw~re ~nd the ay5t2~ C08t $~ increa~d, when separa~ lin~ and pins ~re added to th~ ory array to accomplish th~ functlons described above.
Therefore, there i~ a n~Rd for ~ computer 3y8tem in which th~ number o~ lin~ and pin~ u~d in communicating with a memory arr~y i8 minimlzed. ~ow~er, thera ~180 i~ a noed for a co~put-r ~y~t~ ln which th~ r~ll bLlty of dat being tran~~ r~ to and fro~ the me~ry array i~ high and the probab11ity of unde~ect~d error~ i~ low a~ a ra~ul~ o~ tha U80 of er~or d~tecting/corr~cting ~ode~ and/or the u80 of two soparate 8at~ 0~ logic that co~Funic~e with the ~e~ory arr~y.
~w o~lc~ I ~1 ~1 :~NNEC~N. HENDERSON
I.~R.~OW. ~RRE~r l; DU~NER
177~ 7R~T, N W. -- 2 nl~lOT01~. 0. C 20000 IZ02~9~ 0 . ;l , ' , ' , , . , , I ~222~
It i~ an ob~ect of the in~ention to overco~e at least ~ome of the problemS associatsd with the prior art.
It i~ therefore desir~ble ~o provide a memory that minimlzes the probability of und~tecte errors. It i~ also beneficial to provide a memory without increa~ing the number f lines and pin requir~d in order to communicata with the mamory.
Additional ob~ect~ and d~antagos o~ the in~ention will be ~et forth in part in the d~scription which follows, and i~
part will be obYious from the d~cription, or ~ay b~ laarned by pra~tico of the invontion. The ob~ects ~nd ad~ntages of th~ inv~ntion m~y be realiz~d and obtain~d by ~eans of ~ho inst~um~ntalitie~ ~nd co~b~n~iong p~rticul~rly point~d out in the ~ppanded clai~.
To ~chiev~ ~ha ob~oct~ and in accordanc~ with the purpose~ of th~ invention, a~ Qm~odi~d and broadly d~scribed herein, a mQ~ory is provided for storing data in a computer io7ystem. Th~ computsr syst~m includs~ ~ ~e~ory ~on~roller m0an~ for wTiting or r~dlng d~t~ durlng ~ mamo~y tran4fer cy~le~ and a m~mory interfac~ bus for coupling tha mamory to th~ m~mor~7 controller ~e~n~ The m~o~y interface bu~
includ~ a pluEality of bidirsc~ional d~ta line~, a plurality o~ timo di~ion multiplexed bidirec~ional lina~ d a cyclq timln~ llne for provid~ng cycle ~iming ~ignal~ to the ~mory.
The me~ory in~lude~ ~emory array ~eans ha~lng a plu~lity of addre~able s~orage locn~ions, for 3toring d~ta and ~CC
LAwor~c~- 5i~nal~,7 ~equenc~r m~an~7, coupl~d to the cycle timing line, '`~NBCW H~NDER50N
F.'~RABOW G~RR~Tr 6 DU~eR
177~ ~ 77~ 7, IJ.w -- 3 IIIGT01~, Cl C 2000--1 2 0 ~ 1 0 .
:
: ,, ,;~ '.. ". ' :''' i 2~222~
~or generating a plurality of sequ~nce t$mlng signalR in accord~nce with a clock ~ignal and the cycle tlming 8ignal8;
data buffer mean~, coupled to the plurality o bidir~otional data line~, to the memory array me~n~, and to thQ ~equencer means, for transferring d~ta between the bidirec ion~l d~t~
lines and the memory array mean~, when en~bled by th~
3equenc2 t$mlng ~ignal~ during th0 m~mory transfer cycle; and control buffer means, coupled to the plur~lity of tim~
divi~ion multiplexed bidiroctional lines, to th~ memory array, and to the ~equencer ~e~n3. Th2 control buf~er mean~
i~ provided for transferr$ng ECC ~ignala, between the time di~ision multiplexed bidirection~l l$ne~ ~nd the ~a~ory ~rr~y me~nsl at ti~es when data i~ transferr~d betw3en the bidirectlonal data line~ and ~h~ me~o~y ~rr~y m~an~, ~h0n : en~bled by sequence tlmlng 8ignal8 ~fter th~ b~ginning of the me~ory tranafer cycle. In one ambodiment o~ th~ invsn ion, ' -: thQ control buffer me~n~ also tran~fer~ cycle type 3ig~al~, , provid~d on a fir~t portion o the tl~ divis1on ~ultiplexed , `
bidirectional linea by the ~æy controll~r m~ans, from the time division ~ul~ipl~x~d bid~rection~l lines to the; control bufer ~e~n~, to ~nabls r~dlng or wr~tlng of da~a in ac~o~danc~ with the typ~ of me~ory tran~f0r cycle 3p~C~ fied by ~h~ cy~l~ typs signalY, when ~nabl~d by ~equence t~lng 3ign~1~ at th* b~ginning o~ the m~ory tr~nsf~r cycl~. In a futh~r embod~ent of ~he inv~ntion, ~he control buffar me~ne i~ providxd for ~r~n~ferring ~ddre~ siqn~l0, provided on a second portion of the tim~ dl~i~ion ~ultiplexed bidirectlonal ::~INEC,~N, HENDER50N
F.~R~90W C~RRE~r 1~ DuNNER~
li,' 177~ TaC~T, ~1 W. -- 4 :: ~i~Sr ~I~IGT0~ . 0 ~ jtOG0--~20~9~ 50 1.~'` .
'~' .' , ~ j. ''.,' ,`
: ' ' ' : : ' ~ `
,'" , ' , ' " , '.~ '~ ' " ' -' , .
. , ~ .

2~222~ 1 line~ by the mQmory controller meann, from the time divi~ion multiplexad bidirection~l line~ to the mamory a~ray mean~, ~o acce~ the addre~qable 3torage locationa in the memory array me ns ~pscified by the addr~ss ~ignals, when enabled by l sequence timing eignal~ at the beginning of the memory i tran~fer cycle.
- The accomp~nying dr wings, whlch ar~ incorporated in and constitute a part of this spscifie tion, illu~trat~ preferred embodiments of the invention and, togethsr wit~ the descrip-tion of the inYentlon ~ explaln the prlnciplas of the inven- j tion.

Fig. 1 is a block diagra~ o~ a pr~fer~ e~bod~ment of fault tolerant computor ~y~t~ which practic~s the pr~s~nt inYention;
F~g. 2 is an illu~tr~tion of the physic~l hardware containing the f ult tol~r~t computer sy~tem in Fig. l;
Fig. 3 i~ a block diagr~m of the C~U modul~ shown in the fault tolerant computor ~y~ta~ shown i~ Fig. 1;
Fig. 4 i~ ~ block dlagram of an int~rcon~ct~d CPU
module an~ IOO ~odule for th~ co~put~r syst~ ~hown in Plg.
:.~, l;
Flg. 5 1~ ~ block diag~a~ of ~ m~m~y m~dule for ~h~
fault tolerant c#~put~r sy~tam ~how~ in Pig. l;
Fig. 6 i~ a de~ailod diagram of ~h~ ~lemants o~ th~
control los~lc in the m~ory modul~a ~hown in Fig. 5;
~ o~r~ct-;NEC~N HENDER50N
F.'~RAEOW G,URETr :~6 D~ NER
'1~77~ T~ W. -- 5 H:~TO'I. O. C ~000~
~o~

,1 2~22~ 1 Fig. 7 is a block dLagram of portions of the primary memory controller of the CPU module ~hown in Fig. 3:
Fig. 8 i~ a block diagram of the DI~A engine in the primary memory controller of the CPU module of Fig. 3;
Fig. 9 i~ a diagram of error proces~ing circuitry in the primary m~mory controller of the CPU module of Fig. 3;
Fi~. 10 i~ a drawing of 80mo of the registar~ of ths cros~-link in the CPU module shown in Fig. 3;
Fig. 11 i~ a block diagram of the element~ which route con~rol 3ignal8 in th~ cro~e-link~ of the CPU module shown in Fig. 3;
Fig. 12 is a block diagram of the ele3l1ent~ which rou~s d~t~ and ~ddre~ ~3ign~18 in the pr$~ry cro~a-link of the C~U ¦ .
module shown in Fi51- 3;
Fig. 13 i~ a ~tate diagra~ ~how~ng th~ state~ for the croa~ link of the CPU ~odule ~hown iIl Fig. 3;
Fig. 14 i~ a blo~k di~gr~m of the ti~lng 3y31:em for the fault tolerant co~upu~er ~y~em o~ Fig. 1;
Fig. 15 is a t~m~ng di~gram for th~ cloclc i~al3 gsner- , .
atad by th~ ti~ing ~y~tem in ~ig. 14;
Flg. 16 ~ a dot~iled diagr~m o~ a pha~a detE3ctor for tho tlD~g ~y~t~a l~7hown in Fig . 1 4;
~ lg~ 17 i~ ~ block diagram of an IJO module for the com-pute~ ay~ of FigO 1;
~ ig. 18 i~ a block diagr~ of tha fir~w~ le3ll~7n~ in ~h~ IJO module ~hown in Fig. 17;
LAW o~ C~--FI~NECA91, H~DEA50N
FAAA90W. CARRE~r l; DU~ ER
,,7, ~ ~TI~C~f. ~. ~ -- 6 G T O ~ . O C ~ 0 0 0 0 (2G~1~9~ ~-950 ;' , ,. , , . , .. , :. .

~ ,' '` ' . '~ " ' ' `' :`
.
.' '' ' . ~`''.' `.. ' :

1 2~222~
Fig. 19 is a deta.iled diagram of the element~ of th~
cros~-link pAthway f or th~ computer syst~ of Fig . l;
Fig~. 20A-20E are data flow diagramz for tho co~puter systQm in Fig. l;
Fig. 21 i~ a block diagram oiE zone 20 Qhowing the rout-ing o re~st sign~ls;
Fig. 22 i~ a block diagra~ of thQ components involv~d in r~8~!t in the C:?U modula ~hown in Pig. 3; and F$g. 23 is a d~graDI of elock r ~e'c circuitry.
I. ~
Refer~nce will now ~e D~ade in detell to a pr~sently prB-ferred e~odim0nt of th~ inv~ntion, n ex~pl~ of which i8 illu~tr~ted in th~ 2~companying drawing~
A. SY~T~ D~SCRIPTION ¦:
Fig. 1 i~ ~ block diagra~ of a fsLul~ tolerant comp~ter ~y~t~ 10 in acçordan~Q with tha pr~nt invention. Fault tolerant ~omputar 8yl~'Cem 10 include~ duplic~te ~3y8tQm8, call~d zone~ . In the norm~l mode; the two zones 11 and 11 ' operat~ ~imult~neou~ly. The duplic~tion e2l~ur~s th~t th~sr~
is no ~ingle point of failure and th~t ~ ~ingle error or fault in on~ o~ th~ zonas 11 or 11' will not d~i3able ~omputer sy~t~ 10. Furth~rmor~, all ~uch f~ult~i ~an bOE~ corr~c~ed by di~ablln~ or i~nor~ns the devic~3 or 0~ 0n~ whtch cauBsd the .
fault. ~one~ 11 and 11 ' are ~hown in F~g. 1 a~ re~pectiYely includlng duplic~te proc3ss~ ng i~y~te~ 20 and 20 ' . Th~ dual-ity, how~v~r, 510~8 b~lsyond the proce3~ing sy~te~.
w o~
~` iNNEG~N. HENDER50N
F.~RAEOW GARRE~r a 3UNNER
~¦ 17~ !;TR~T. ~ ~. 7 .v~ llOT011. 0 C ~000--120~)Zli~00~0 '' .

.:

" ! ;
., .' ' I ' '. ~ . ~

2~22~

¦ Fig. 2 contains an illustration of the phy8ic l hardwars of fault ~olerant computer 8y~t~ 10 and gr~phically il-lu~trate~ the duplic~tio~ of the 8y8tem8. E~cb zon0 ll ~nd 11' i~ housed in a diffexant cabinet 12 and 12', re~pectively. Cabinet 12 include~ battery 13, power regula-I tor 14, cooling fan3 16, and AC input 17. Cabinet 12' j include~ -4eparate element corr~sponding to el~ment 13, 14, 16 ~nd 17 of cabinet 12.
A~ explained in gr2 ter d~t il below, proc20sing sy~tam~
20 and 20~ include se~eral mo~ule~ in~erco~nec~ad by backplanen. I~ a module co~tain~ a f~ult or error, th~
module may be remov~d and replacad wlthout di~abling c~put- , .
ing 8y~t~m 10. Thi~ i~ bec~use pro~e0~ing y8tem~ 20 and 20~ ~
are phy~lcally ~ep~r~t~, h~v~ s~parat~ backplane~ into which the module~ are plugged, and c~n opsr~ indep~ndently of each other. Thus m~dule~ can b~ r~mo~d ~ro~ ~nd plug~d into th~ backplan~ of ene proc~s~ing ~yst6~ wh$1e ~hs other proce88ing 3y~ on~inue~ ~o operA~e.
I In th~ pr~erred embodiment, the duplicate pre~e~ing ~yste~ 20 and 20~ are ident~cal and ~ont~n identical modules~ ~hu~, only pro~sssing sy~ 20 will be de~crlbed co2plot~ly ~ith th~ understnnd~ng that proce~tr.g ~y~te~ 20 opora~ gulv~lently.
Proce~sing sy~t~m 20 includ33 C~ module 30 which is shown in ~r~ter d~tail in Pig~. 3 .nd 4. QU ~odule 30 is . interconnect~d with C~U module 30' in proc~sing ~y~tem 20 LAW O~lCt~ I by a cros~-link pathwAy 25 wh~ch i~ described in greater INNEC~N. HENDER50N
F.'~R~30W ~;ARRETr 6 DUNNER ;
17~ TI~ T, ~. W.
iTO~. O. C 20002 ~0~ 0 ,..
. '.;
' ' ' ~ ' . ,; ."' , ' ~' ~ ,'; `
, 1 2~2~ 1 I
detall below. Cro~-link pathway 25 provides data tran~mis- I
sion p~th~ betwsen proce~sing ~ystem~ 20 and 20 ' and ~ rrie~
timing ~ignals to en8ure thAt proco~sing ~ys~em~ 20 and 20 operate ~ynchronously.
l Proce~ing ~y ~em 20 al~o includes I/O module~ 100, 110, I and 120. I/O modula~ 100, 110, 120, 100', 110~ ~nd 120' are indspend~nt de~ic~. I/O module 100 i~ ~hown i~ great~r detail in Fig~. 1, 4, and 1~. ~lthough multipls I/O module~
are shown, duplication o such modul~ i8 not a requirement of th3 ~ystQm. Wlthout ~uch dupl~catio~, how3Y~r~ ~O~Q , degr~e of fault tolerance will ba loat.
~ach of the I/O module~ 100, 110 ~nd 120 i connec~ed to CPU ~odule 30 by dual rail module int0rGonnect8 130 and 13~. ¦
Module interconn0ct~ 130 and 132 d~rre ~ the I~O
in~erconnect ~nd ars routed acro~s ~h~ b~ckplane for procs~s$ng ~yste~ 20. For purpo~ o~ thi~ Appllcation~ the data pathw~y including CPU 40, me~ory controller 70, cross-link 90 and ~odul~ interconn~ct 130 ~8 con~idQr~d a~ one r~il, and ~he dat~ pathw~y including C~U S0, ~emory control-ler ~5, cro-~ link 95, and module interGonnec~ 132 i~7 considar~d a~ ano~her rall. Durlng prop~r op~ra~ion, the d~ta on both r~ the sam~.
~. F~ O~E~A~T SY~T~ P~ILOSOP~Y
Fault tol0r~nt comput3r 8y8t~m 10 does not h~v~ a singl~
point of failure bec~u~ e~ch el~m~t i~ dupl~c~d Proc~s~ing sy~tems 20 and 20' are e~ch a fail ~top proce~ing .. systQm which mean~ th~t thos~ 8y8tem8 can det~ct fault~ or I~NEC~N ~E~DERSON
F.~R~30W GAR1ETr ~ DUNNER
17T~ ~ ST~T. N ~. _ 9 '~A~ NCTON. C. C. 2000--1-0~ 0 1 2~2~3 error~ in ~ha qubsy tem3 and prevent uncontrolled propag~tion of such fault~ and ~rrors to othsr ~3Ub8y8telDI~, but they have a ~ingle point of f ailurQ becau e the elsment~ Ln ~ch ¦ proce88ing ~y8te~ are not duplicated.
The two fail 9top procQ8sing 8y8tem8 20 and 20 ' are . interconnected by certain elQm~nts operzting in a def lned ~ manner to form a fail 3afe ~y~tem. In the fail ~afe sy~te~
embodied as fault tolerant computer 9y8tei~ 0, the erl~cira computer 4y8telll can continue proce~ . ing even if o~e of ~che fail stop procossing sy~tem~ 20 and 20~ i3 f~ulting.
The two ail 3top proce~in~ ~y~tem~ 20 and 20 ~ ar6s considered to opsrat~ in lock~tep syllchroni~m because CP~
40, 50, 40~ ~nd 50~ opar~ta in auch ~ynch~; Th~r~s a~:3 throe si~niflcan~ ox~eption~. ~he fir~t i~ ~t $nit~aliza~ion ¦
when a boot~tr~pping ~cachnique brlng~ both procs~sor~ into ~ynchrsnis~n. Th~ qscond exc~ption is wh~s~ th~ proceY0ing ~y~em~ 20 and 20' oper~t~ indape~dQn~ly (~ynchronou~ly) on i two differen'c workload~l The third exception o~cur~ when certain ~rrors arise in proc~ssin~ syst~ls 20 ~nd 20~. In thi~ la~ axc~p~iorl, ~h~ CPU and ~e~ory elQments in ona of the proc~ing systQ~ i~ diaabled, th~r~by ending 3yJlehrOnOU~ o~ tion~
Who~ the 8y~11t~am ill rwul$ng in lo~k~t6~p I/0, only one I/0 device i~ being Ac~e813ad at any on~ ~i~e. ~11 fou~ C~U~ 40, 5 0 , 4 0 ' and 5 0 ', however , would receive the 8am~ da~a f rem th~ I/0 devic~ at ~ubst~ntially th0 s~me ti~n~. In th~
~AWOt~lC~ I following discu~on, it wi 11 be u~d~rstood that lockstep CIYNECAN. HENDER50N
F~RAEOW C~RRETr 6 1: UNNER
177~ TRt~7, ~ W. -- 1 0 w~nln~TOI~. O, C 2000-I~ O ~ O

~' , " ,. '' ' ~", '' .
. . . .

~, : ..
'~
- , , .

I 2~222~ 1 ~yn,chroni~ation of proce88ing system8 mean that only one I/O
mod,ule i~ being ~ccessed.
The 3yn,chroni~m of duplicate proces~in,g 9y8tem~ 20 and, i 20 ' i9 implemented by treating each system a~ a deterministic machine which, ~,tarting in the sam,e known ~tate 2,nd upon, rec~ipt of the s~me inputs, will alw~,y~ enter the same machine 3tatQs and produc~ the same re~ult~ in the ab~ence of error. Proce,Ysing 3y8tem~ 20 and 20' are configured idonti-cally, receive the ~am,e inpu~ , and ~h~refor~ p~i8~ through the 3ame ~,tate~,0 Thus, a~ long a3 both proce~sor~ oper~te ~yncbrenously, they should produce the ~am~ re~ult~ and enter thQ 8ame ~,tate. If the proeassing 8y8t~m8 ar~ not in th0 i 3a~e.~,tate or produce diferent ~esults, it i8 as~u~ed, that I ~
one of the proce~ing 3y8t~m8 20 and 20' ha~ ~,ult~d,. Th,~ I -sourc3 of th~ f~ult mu~t th~n be i~ol~t~d ln ord~r to take corrective action, suGh as disabling th~ faulting modula.
~rror det~ction gen~r~lly involve~ ov0rhea,d in the form of addition~l proce~sing tim,~ or logic. To m$nimize ~uch oY~rhe~d, a sy~tem ~,hould ~heck ~or ~rrors ~8 infraquently a~
poff~ible ~on~lstent with fault toler~nt op~ration. A~ ~he very le~t, error checking mu~t occur b~for~ d~ta i~ output-~d fro~ CPU ~odul~ 30 and 30'. Other~ise, internhl proce-81ng error~ ~y C~U~H improp~r op~ration in ext~rnal yst~, lik~ a nuclear re~ctor, which i~ the condition that fault tolerant syat~m~ are de~gned to p~avent~
Th~re ar~ re~son~ for additional error ch~cking. For wOrrlc~ ex~m~le, ~o i~olate fault~ or ~rrors it i~ desirable to check ECW. HENDERSON
F.~R~OW G,~RRE~r ~ DUNNER I
177~ r~l~t~ N W. -- 11 --IINI~0~ . 0. ~ 000 1202~9~ .~a~o ., 2~222~ ~

the data received by CPU modules 30 and 30' prior to storaye or use. Otherwise, when erroneous stored data is later accessed and additional errors result, it becomes difficult or impossible . .
to find the original source of errors, especially when the erroneous data has been stored for some time. The passage of time as well as subsequent processing of the erroneous data may destroy any trail back to the source of the error.
; "Error latency," which refers to the amount of time an error is stored prior to detection, may cause later problems .;, ;ii 10 as well. For example, a seldom-used routine may uncover a latent error when the computer system is already operating with diminished capacity due to a previous error. When the computer system has diminished capacity, the latent error may cause the system to crash. ;~
Furthermore, it is desirable in the dual rail systems ~;
of processing systems 20 and 20' to check for errors prior to -transferring data to single rail systems, such as a shared re-source like memory. This is because there are no longer two independent sources of data after such transfers, and if any error in the single rail system is later detected, then error tracing becomes difficult if not impossible.
The preferred method of error handling is set forth in Canadian application Serial No~ filed on this same ; date in the name of Digital Equipment Corporation and entitled "Methods of Handling Errors in Software."
' :

, "

2~ 2~

C. MODULE DESCRIPTION
1. CP~ Module The elements of CPU module 30 which appear in Figure 1 are shown in greater detail in Figures 3 and 4. Figure 3 is a block diagram of the CPU module, and Figure 4 shows block diagrams of ~' , ~' ~ - 12a -~ ' ! ' , ' ' ' , . . ' : ,;. . ~ ' ' !

. ~ ~ ~1 2 ~
CPU module 30 and I/O modula 100 as well as their intercon-nections. Only CPU modulo 30 will be described ~ince th~
op~r~tiun of and the elem9nt8 included in CPU modules 30 and , 30' are g~nerally the ~am~.
1, CP~ module 30 contain~ du~l C~8 4~ and 50. CPUs 40 and 50 can b~ standard c~ntr~l proc~s~ing unit~ known to p~r~ons . of ordinary ~kill. In the preferr0d em~odimen~, C~U~ 40 and ! 50 are ~AX mlcroproces~ors ~anufactur~d by Dlg~tal Equip~ent I Corporation, the a ~ign~ of thi~ pplication.
A~ociated with CPU3 40 and 50 are cache memorie3 42 and 52, re~p~ctively, which are tandard c&che RA~8 o~ uf~iciant ~amcry size for the CPU~. In th~ pr~f~rred emb~diment, the cache RA~ i~ 4R x 64 bit~. It is not nece~sary for th~
pre~ent inven~Lon to havo a cache RA~, how~ver.
2. ~9~e~Y_~e~Yl~
Prefer~bly, CPU'~ 40 and SO c~n shar~ up to four m~mory module~ 60. Fig. 5 i8 a block dia~ra~ of on~ m~mory module 60 3how~ connec~d to C2U ~odule 30.
DurLng m~Aory tran~3r ~C~ r ~3tatu~l ragister tran~f~r cycl~, and EæPRQ~ tran~fer cy~ , eaeh ~emory ~odule 60 t~an~f~s~ data ~o ~nd ~ro~ prim~ry ~e~ory controllar 70 via a bidls~ctionsl d~tA bu~ 85. ~ach ~e~ory ~odulQ 60 also s~a~lvea ~dd~es3, control, timlng, and ~CC s~gnals f~o~
memory controller~ 7~ &nd 75 via bu~Qs 80 and 82, re~pecti~ely. The addre~ 31gnal~ on buse~ 80 ~And 82 includ~
board, b8nk~ and row and column addres~ signal~ that identify ~AW O~.IC~
FINNEC~N. HeNDERSON
F.~AA~OW. GARRETr ~i Dll~lNER
1??~ Il S~rl~--~t. N W. -- 1 3 ~A~ tO~. ~. C ~00 12 0~ 0 I
.

' " . :; ~. ~ , , :'' ' :~ ' , . . ' ; ": ' ' ". ~ ', :' ! 2~2~2 tho memory bo~rd, bank, and row and column ~ddres~ involved in th~ data transfQr. I
A~ shown in Fig. S, each me3~0ry module 60 inelud~s ~
I mem~ry array 600. Each memory arr~y 600 is a ~tandard RAM in I which the DR~B ar~ organized into eight bank~ o memory. In ~he preferred embodiment, fast p~go mode t~pe r)R~8 are used.
Memory module 60 al30 include~ control loglo 610, ~ata transceivers/regi~ters 620~ memory drivers 630, and an E~PRO~
640. Data transceiv~rs/receiY~r 620 provida a d~ta buffer ;;nd data interface for tran~ferrln~ d~ta b~t~een memr)ry array 600 and the bidirecSion~l d~t~ llne~ of data bu~ 8S. Memory driv~r~ 630 dlstr$bute row and co1umn addr~ss ~ign~l~ and cont~ol signa1s froD~ control logic 610 to e~ch banlc in m~ry arr~y 600 to en~bl~ traD~If~r o~ a long~ord of d~t~ and it3 corr~a~ponding ~CC ~ignal~ to or ~rom ~cha me~ory ba~ electl3d by th~ atemory bo~rd and b~nk addre~ ~igna1 E~PRQPg 640, which c~n b~ any type o~ NvRA~ (nonvo1ati1 RA~), stores ~ory error data for o~f-lLne rep~ir and conf$gura~cion data, ~uch ~ modu1e iz~ Q the memory module i8 removed aft~r Zl fault, stor~d data i8 extracted fro~ B~RO~I 6~0 to detein~ th~ c~u~ of the fault. E~PROM
640 i~ ~ddr~sed vi~ row addr~ ef~ fro~n driver~ 630 and by ~B~PR(:~ control slgna1~ fro~Q contro1 1Ogic 610. ~PF<O~ 640 ~r~n~f~r~ ~igh~ blt~ o data to a~id fro~ ~ thirty-tw~ bi~c internal m~ory da~a ~us 6~5.
~ Control lo~ic 610 route~ addre~!lEI sign~ to the ele~m~3nt.~ ~
~wo~ct~of me~ory modu1e 60 and yenerateu interna1 t~ing and control . INNEC~N HENDER50t F.~R~EOW. (;ARRETr . 6 ~U~;~;ER

TOr O c ~ooo~
12011~-~5-~0 ':~

'' '' . ,:: ~::

2 ~ ~ 1 ~ignals. ~ shown in s~re~ter detail ln F~g. 6, con~:rol logLc 610 includes a primary/mlrror designator circuit 612.
Pr~n~ry/mirrOr de~ nator circuit 612 recelve~ two qet,8 I of memory board addres~, banlc addre3s, row and column addres~, cycle typa, and cycle timing ~ignal~ fram ~Q~ory controllers 70 and 75 on bu~es ~0 and R2, and al~o trAn~fer~
two 30t8 of ECC ~ign~l~ to or ~ro~ the memory controllers on bu~Qs 80 and 82. Tran~celvers~egi~ter~ in do ignator 612 provida a buffer and interfac~ for tran~erring ~he~e signal~
to and from s~lemory busa~ 80 and 82. A primary/mlrror mllltiplexer bit stored in statu~ regi3t~rs 6 1 8 ind$~tes which one c~f i~ory controllei~ 70 ~ 75 $~ d~ nat~ ~ i the primary ~eDIo~y controller asld wh~h ~a~ed ~ the mirror me~ory con1:~ollor, ~nd a p~-imary/~isror D~ul iple~xs~r signal is provid~d fro~ ~tatus regi3ter~ 618 to d~si~&tor 612.
Primary/mirror de~ignator 612 p~ovide~ two ~t~ of sigrlal3 for di~ribution in co~trol logic 61G. Om3 ~et of 3i~al~ include~ de~ignat~d pri~ry ~moxy board addre~, b~nk addr~s~ ~ row and column addre~-, cy~l0 t~ , cyGle timing, ~nd lSGC ~ignal~. The othe~r set o~ al~ include~
da~ t~ o~ ~ory bo~rd addrao~, b~ ddre~, row and colu~ add~a~, cycle ty~po, cyclo t~ng, and ~CC ~Lgsl~la.
The p~i~ary/mirror multipl~xe~ E7'ign~ u8ed ~y de~$gn~tor 612 to ~elect ~he~her the ~ lF7 o~ bu3es 80 and 82 ~ill be~
r~p~ctlvely rout~d to tha line~ for c~rryiLng dasign~ted ~W 0~71C~7~
I~,NEC~N, HENDER50N
F.~R~OW CARReTr 6 DUNNeR
1775 5 5~R~t~
-0~, O C 20001 ~20~2~ 0 I ~ 2 ~ ~
primary eigs~al~ and to the linsy ~er chrrying d~a~ ted mirror 8ignal8, or vice-vers~
A nuD~r of time di~ ion multipl~x~d bldirec~ional lines ~re includ~d in bu ~ 80 and ~2. At c~rtain time9 after the beginning of m~n~ory tran~fQr cycle~, ~tatus regi~ter trAnsfer cycles, and ~PP~OII tr~naf~r cycl~, ECC
~ignals corr~3pondlng to data on data bu~ a5 are placed on these time di~ision multipl~xad bidirectionaL lin~. If the tran~far cycle i3 a writs cycle, r~emory ~ le 60 rec~ e~
d~t~ ~nd ~CC ~ f ro~ the m~ory contreller~ . If thQ
tran~fer cycl~ i8 a read cycle, ~kory modul~ 60 tran~it~ I
data alld ~CC Ylgn~la to the me~ory ~ontroller~ other tim~ durlnç~ tran~fer cycl~ ddre~, control, and t~m~
8ign~18 ~re r~ceived by m~Dnory module 60 on the ti~e di~i~ion multiplex~i bidirectLonal lines~ Prefer~bly, at the beginning of ~mory trar~ ar c:ycl38~ 8t21tU8 r~gistsr tran~fer cycl0s, and XX~0~ tran~fer ~ycl~ ~ory controllers 70 and 75 tran~Dlit ~:~ry board ~ddraesr bank ~ddrasa, and cycle type ~ignals on th~e t~mesh~r~ line~ ~o eAch mamory module 60.
Pr~g~a~ly~ ~ow addra~ sigllal~ and column addre~
Yign~ls ~re ~ultipl~x~d on the ~ ro~ ~nd column addre~
lin6~- d~in~ tr~n~fer ~ycle~. Pir~ row addre~s i~
prsvl~ed to ~Lory slodul~a 60 l~y the me~ory ~ontroll~rs, follo~d by ~ column ~ddre~s abou~ y nano~eco~ad~ later.
A ~quen~ 616 recel~0~ ~8 ir~put~ a sy~te~ clock ~Lgnal wo~ce- and a re~st signal fxo~ CPU ;nodule 30, and rec0i~ che :N;NE~N~ HENDERSON
FARABOW. GARRE~r ~ ~UNNER
,7" ~ ~r~teT ~ W. ~ 16 O~ON. a. c ~oooo 1 ~ 0 ~ 3 0 ~, , ' , .

`` i 2~2~
d~3slgnat~d primary cy~ timing, dç~8ign~t~d prim~ry cyclo type, dç~sigslatad mirr~r cycl9 timing, and deaignat~l mirror cycle type signal8 from th~ tran~ceivex~/:rogi~ters in desi~ator 612.
Sequ~ncer 616 is a ring counter with a~ociated steering logic that ganer~te~ and di~tribu~es a nulaber of control and ~ ~e~uence timing sign~ls for the m~mory modul~ that are naeded I in ordar to execute the variou~ typos of cycleR. Ths controland ~uence timing aign2~1s are gener~t~d from thQ 9y8~cem ¦ olock signal~, ~he deRigna~ed primary cy~l~ t~ming E~igna I and the d~ign~ted prim~ry cycl~ typ~ ~ignal~.
¦ S~quencer 616 al~o gener~ste~ a duplicat0 ~t of sequQnce~ng 8igllal8 from ~h~ ~yste~ clsck 8igll~l8, tha d~3signa~
mlrror cycle timing 8ign~1~, and the d~lgn~t~d mirror ~ycle ~ype signalc. Thes~ duplicate 0~ence ti~ g ignal~ ~r~
U8~!d for error checking. Fs~r clat2~ tr n~ferR of mul~i-long . ~ word~ of da a to ancl frola memory module 6û in a f~t pag~
mode, each 80t: of column addr6~8sQs 8tarting wi h ~ha firYt 6~t i8 follow~d by the n~ column addr~ 120 nano~econd.~
later, and ~ch lon57 word of da~a l~ moved acro~ bu~ 85 120 n~no~oconds ~f~r the pr~lous long word of d~ta.
¦ S~ue~e:er 616 al~o gen~3ratQ~ tx/rx r~gister con~rol 1 ~ aA13~ The 1:x/rx r~sgi~ter control ~ignal~ ar~ provlcled te .: I control ~:he operation of d~t2l tr~n~c~ r~/re~i~ter~ S20 and th~ tran0c3ivar~register~ in de~lgn~or 612. l`he direction of d~ta f low L~ det~n~in~d ~y ~h~3 s~Qering log~c ln s~ ncer wOrrlc~ 616, whlch respond~ to the do~ignatQd pri~ry cy~l~ type INNECAN HeNDE~50 .: F.'~R~BOW GAQRE~ T
~i 3UNNER.
177~ ~ STRt~ W -- 1 7 WI~NINO~ON O C 2000--~ 2 0 ~ 2 ~ 10 .' .
. . ' .

2 ~ ~ 1 ~ignal~ by gen~rating tx/rx control and ~uence timing sign~ls eo indicat~ whQth~r and when data and ECC sLgn~ls 3hould b~ written into or read from the tran~caLvers/
registQr~ in mem~ry module 60. ThuR, during m~mory write cycl~s, tatus regi~ter writQ cycles, and E~P~OM writ~a cycl~s, dat and ECC 8ign~ w~ll b~ latched into th6~ :
transCeiVers/rQgi~t~r~ from buse~ 80, 82, and 85, while dur-ing memory read cycle~, ~t~tu8 regist~r re~cl cycle~, and EEPRO~ read cycle~ 9 data and ~CC 3ign~118 W~ 11 bQ latched into the tran~ceivers~ragi~terY from ~emory arr~y 600, ~t~tu~
register~ 61~, or ~SP~aM 640 for output to CPU ~odule 30.
Sequf3ncer 616 al~o generAte~ ~PROPI cnntrol ~ign~ to control the operation of ERPROM 640.
~ he ti~ng ~elat~onships that oglst in sll~ory m~duls 60 are ~pec~fied wi~h refer~ e to the ri~e tl~ o~ thQ ~ystem cloclc ~ignal, which haa a ~riod of thirty nano~econd~. All ~tatu~ resli~ter read and ~rlte cycl~3s, ~nd all me~ory re~d and wsite cycle~ o~ a ~ingle loslgword, are p~rformed in ten ~y8tem cloclc per$ods , i . e ., 300 naIaos~cond~ . M~a~ry read and writ~ tx~n3fe2 cycle~ m~y C03~ t o~ multi-lo~word transfær~. ~or each addition~l lon~rd th~ is ~xansferr~d, the m~ory tran~fer cycle i~ extend~d for four add$tional sy~t~ ~lock p0~1Od~. ~emo~y refre~h cycle~ asld ~RPRO~ write`
cy~ requir~ al; le~t ~ Ye ~yst~n clock periods to ex~cute, and ~BP~O~ x~æd cycl~ req~ire at leas~ tw~anty systan clock p~riods, L~W orrlc~7 NE~AN. HENDER50N;
FARAEOW, cARRE~r ~ DL'NN'E~ ¦
17~q ~ IITRIC~T. R W. ¦ '-- 1 8 W~51~ GT0!1. 0 C ~000--~20~1 ~9~ ~21~0 ~ .

':', , ' , ' ' ~'; ' 2 ~
Th~ dasignat9d primary cyclo timing ~ignal cau~es s~quenc~r 616 to st~rt gen~r~ting th~ sequence ti~ing and control ~ign~l~ that enable the m~mory module ~elected by tha memory bo~rd addre~ ~ignals to implement a raquested cycle.
The tran~itio~ of the d~si~nated primary c~cle timlng ~ignal to an acti~a ~tat~ m~rks the 3tart o the cycls. Tha ~tuxn 9~ tha de~ignated prim~ry cycle tlming 8ign~i to an inacti~e ~tate marks the end of the cycle.
The ~equenc~ timing ~ign~l ~enerated by ~aqu~cer 615 are a~sociatsd with th~ dlfferent tate~ entered by the 3~quencer a~ a cycl~ r~que~ted by CPU module 30 i8 e~eouted.
: In order to specify ~he tlmlng relatlon~hip ~mong the~e dif f ~r~nt ~tat~s (and the timing r~lat~on~hip among 8~que~c~
timlng ~lgnals corresponding to e~ch o~ the~ st~t~), the discrete s~a~a~ that may be ~n~er~d by ~equenc~r 616 ~re ; ~ id~ntifi~d a~ ~t~t~ S~Q IDL~ and SEQ l to ~Q 19. Eac~
tate la~ts for a sinqle ~y8t~m clock pariod (thirty ¦ nano~econds). En~ry by ~qu~nc~r 616 in~o aach diff~rent .~ ~ tH i~3 trigger~d by th~ l~ading edg~ of th~ y~t~ clock i~n~l. Th~ le~dlng ~dgQ~ 0~ ~he ~y~t~ clock ~ignal that :- cau~e ~Qquenc~r 616 to en~r 3~ate~ S~Q IDL~ and SEQ 1 to SEQ
19 ar~D re~rr~d to ar3 transitlon~ T rDI~ and Tl to Tl9 o ~el~t- th~a to thss ~ nc~r 3tde,1Z~, i.e., ~ is tho t~yE7'tem ' clock aign~ din~ ~dgs that CaUklS38 ~uerlcer 616 to e7nter ~ta~e SEQ ~1.
A~ time~ wh~3n CPU module 3Q i~ no~ dir~ct;}ng m~ory LAW or~t-snodule~ 60 to 9XelCUte a cy~ , the de~ t~t pri~ry cycle :: INNeC,~N HeNDeR50N
FARA~OW cARRE~r :~ ~ D~ R
, ` 177~ Y ~T11C~T. ~ ~. -- 1 9 .'/A31~1110T01~. 0 C ~000~-02~ 0 , . , . ~ ' ~ ' ' :
,. ~ .
'' ~' , .. . .

~: , 2~2~ 1 t~ing ~i~nal i~ not al3~rted, and the saquoncer re~ainz in stata S~Q IDLl:. Tha 9~B~uencer L~ ~tarted (enter~ tate S~Q
1) in respon~e to a~ertion hy memos~ controller 70 of the cycl~ tlming ~ignal on bu~ 80, provided control logic 610 and ~equencer 616 are loc~ted irl the m~mory module ~el~cted ~y , I memory board add~e~s 3ignal~ al80 tran~mitt~d from m~ory I ~ cor~troller 70 on bu~ 80. The rising ~dge of th~ fir~t ~y~tem clock ~ l following assertion of the de~1 anated primary cycle active ~ l corra~pond~ ~o tr~n~itlon Tl.
1~8 indica~d prs~iou~ly, in the~ ca~e of tr~ f ers of a 31~gle longword to or from me~ory array 600, the cycl~ i8 performed in ten 8y8t0m cloc~ period~. Th0 . equence~r proc~d~ ~rolD S~Q IDL~, to ~t~t~ SBQ 1 t~fi S~Q 9, ~nd~ I
return~ to S~Q IDL~. ¦
~emory rs~d and ~ri'ce cycl~a ml~y be ex~ended, howeYar~ ;
to tr~n~f~r additlonal long~ord~. Ne~ory ~rr~y 600 prefer~bl~f u~e~ "fast pZlgR ~od~" DRAM~. During multi-longword re~d~ ~nd wrlt~, transfers of d~t~ to and ~rom the m~mory arr~y a~sr tr~n3~er of thç~ fir~t lon5~word ar~
¦ accompl~sh~l ~y x~ps3at~slly upd~tir~ h~ column addre~ and . , regener~ting ~ QS ~ colulon addr~s~ strob 3 ) ~lgnal .
; I ~urlng multi-longwosd transf~r cycle~, the3~ upsl~t~ of tho c:olu~n addre~s can b~ l.mplemented b~cau~ s~quencer 616 repeatadly loopl~ f roDI qt~ee~ S~ZQ 4 ~hrou~h 51~1;) 7 un~ 11 of th~ longwords a~s~ tr~nsferre~. ~or eal:~mpl~, if thr~
longword~ are b~3ing read from or writ~en into memory ~r:r~y wO~r~cl~J ¦ 600, tha sQqu~ncer enter~ ~7tate~ S~Q ID~, 5EQ 1, Sl~Q 2, S~Q
-~NNEC~N. HENDERSON
F~ D~A3OW~ C~RRETr UNNER
17r~ Tr/~T,t.l ~I ;~!O
V/~9~ 0~0!1. D. C 1000~
~o~ o .~' .
.
,' ' , , : - ~

, ~ '~ fS3 2 ~
3, S~Q 4, S~Q 5, SEQ 6I SEQ 7, s~a 4, SE~ 51 S~Q 6, S~Q 7, SEQ 4, S~Q 5, SEQ 6, SEQ 7, SEQ B, SEQ 9, and S~Q IDL~.
During a m~Qory tran~fer cycle, the de~ignated primary . cyclo timing ~ignal i8 monitor~:l by ~ uencer 616 duxing ~¦ transition ~6 to det~in~ wheth0r to axtend the memory read 11 or write cycle in order to tr~n~fer at le~t ~ne ~dditiollal longword. At times wh~n th~ d~ign tad primary eycle timing ~ign~ a~ert6~d during transition T15, the ~equencer in tato S~Q 7 will re0pond to the~ n~xt system clocls . ignal by entering stata S~Q 4 instead n~ enterirlg ~t~e ~E~2 8~
In th~ ca~e of a multl-lon~word t:ran~er, th~ de3lgJlated prlmary cye:l~3 timing 8ign~ll 1E3 a3aerted a~ l~a~t fifteen nanos~cond~ befor~ the fir~t T6 translt~ orl and r~ ins -836~rt~ unt~l the final lons~ord l~ transf~rr~. In ord~r to ~nd a m6mory tran~s~er cycle af~Qr ~he flnal lons~word ha~
be~n tran~rr~d, the dusiqTlatQd prilDary cycle tlmin~ Rignal i~ dea~0rted at lea~t fif~een nanose~cond~ before tha la~t T6 ~ran~itl on and rç~ain~ d~3a~er~d for at lea~ ten nano~econds after th~ la~t T6 tran$ition.
During ~ory ~ran~fer cycl~3~, thQ do~igna~d primary row ~ddr~sc ~ ancl the d~ t~d pri~ry col~n addra~s ~ ual~ ar~ pr~n~d a~ differ*n~ gima~ by d~Lgn~tor 61~ in con~rol logic 6lO ~o ~n~ory drlv~rs 630 on a 8e~ of ltimeT
divi~i~n mul~ipl~x~ lin~. Th~ output~ of driver~ 630 ~re appli~d ~o ~h~ addx~K inpu~ of ~h~ DR~Mg in me~nory array 60û, and al~o ~r~ r~u:rned to ~on~rol loglc 610 for ~wo~ct~ compari~on wil:h ~he d~ignated ~rror row ~nd column ~ddre~s . I~INEC~:`I. H~NDERSON
.F.~R~OW GARR~Tr DUNN~R
In~ ~ sr1~ w w~ ~ 2l wo-o~. O c ~000~
110~9~ o ~. .

l ~ 2 ~
l .
si~nal~ to check for Qrror3. During ~t~tu~ regi3ter tran~er cycle~ and BEPROM tr~n~fer cyclea, column ~ddress ~ 18 are not needed to select a particular ~torage location.
During a med~ory tran~fer cycle, row addre ~ ~ign~ls are the ~ir~t s$gnal~ pxe~ent0d o~ the tlm~g~hared row and column addr3~s linoa of busea 80 and 82. During state S~Q IDII~, row addra3s ~ignalR ar~ tran~mittsd by tho m~mory oontrollors on ~he row and col~n addr~s~ lin~, and the ro~ addresR i ~tAbl~ from at loast fifteen nano~econd~ ~efore th0 Tl tran~ition until ten n~no~econds af ~r the Tl transi'c~on.
Next, column add~e~s ~ligrL~!118 ara trana~itt&d by the msmory con~roller~ on th~ ro~ and colu~n addroa~ line~, and ~he~
;- column addre~s i~ ~tabl~ ~ro~a a'c lea~ ten nanosocond~ bs~f~re the T3 tran3ition until fifta~n n~no~cond~ aft~r the T4 tran~ition. In ~he e~o of ~nulti-longword ran3f~r~ during memory tran~fer cycla0, Rub~ nt columa . ddr0~ ~ignals are then tr~nsmi~ct~d on tha row ~nd colum~ addrsss line3, ~nd ~, the~e ~IUb8eqll~11t column addrQ0s~s ar0 -~t~bl~ f rom ~n nano~econd~ b~for~ the T6 tr~n~i~lon un~il f i f ~e~an nanosQcond4 a~ter the r7 ~cran~i ion.
G~nerator/Gh~3cker 617 rece~ir~ ~h~ ets of equç~rlce t~g ~1gnal~ rat~d hy ~e~uanc~r 616. In addltion, the d~s~t~d yr~sry cy~lo typQ and bsnk ae~dre~0 ai~n~ls an~
th~ d~ Dirror cycl0 typ~ and ban~c addrf~ are 'cr~n~mitted to gen~rator/c:heck~r 617 ~y d0~1gnator 612. ~n the generator/check~r, a ma~r o pri~ control 3ignal~, -~W O~IC~9~ i . e ., RA5 ( row addre~s ~robe ~, CAS ( column ~ddre~s Rtrobe ), . I~NECA~. HE~DERSON
F.~R~90~ G~RRE~r ~ DU~;~E~
~77~ Tn~T. ~ ~. 2 2 ~snll~oTot . o C 000--1~0~ 9~0 ", ~ ~ ~ ' ' ' ' `
'' , ' ` ` ' ` ' ' -': , and W~ (writQ ena~le), are gen~rat~d ~o~ distribution to drivsr~ 630, u~ing th~ primary 3equence tlmlng 8ign~18 and the d~signatQd prim~ry cycla type and ~nk addre~ 8iyn~
A duplicate ~et of these control signal~ ia g~nerated by 1 generator/checker 617 from the duplica~Q (mirror) ~equenc~
! ¦ timing ~ignal3 and th~ designated mirror cycle type Rnd b~nk : ¦ address slg~al3. ~h~e ~irr~r RAfi, CAS, and writ~ enab?e I signal~ are u~ed for error checking~
: ¦ When the primary cycle t~pe ~1gnal~ indic~te ~ mamory transfer cycle i~ b~ing perfox~ed, tha prl~ry b~nk addr~ 8 I sign~ls id~ntify one ~e1QCted bank of DR~M~ in m~mory array 600. ~ory driver~ 630 includ~ ~ep~rat~ ~AS dri~er~ ~or each bank of D~A~ ln ~3~0ry array 600. In ~nor~tor~ch0cX~r.
617, the primary R~S ~gn~ gen~rated during th~ m~mory t~n~er cy~ nd damultiplexed onto on0 of the lin connecti~g th~ gene~tor/~h~cker to th~ RA8 dri~r~. A~
~ result, only the RAS dri~er corr~pe~ding to the .elee~d : DR~N b~nk r~c~e3 ~n ~e~tsd RAS signal during th~ m~ory tran~r cyclQ. Durlng ra~r~h cycla~, the prl~ry R~S
~ign~ not de~ultiple~d and an a~sert~d ~S sl~nf21 i~
recelv~d by ea~h XAS drlver. ~uring s~a~u~ register tran~fer cyclo~ ~d ~PRO~ tr~n~fer cycles, th~ bank addre~ ~ign~l~
ar~ unn~ce3~ary.
~ emory driv~rs 630 ~l~o incl~d~ CAS driver~. In generator~ch~ck~r 6l7, the prima~y C~S ~ignal i~ gen~rata~
during ~o~y tr~n~e~ cycle~ and refre~h cycl~. The ~AW O~C~--~,Fl~;~EOAN. HE~DER~O~
F.~RAaO~ RRETr E13 D~ER
177~ ~ S~ . -- ~! 3 '. ~A~ 0~0~. 0 C 2000GI
: ' IZ02~ ~9~ !10 '``..', ~' , .

primary CAS qignal i9 not d~multiplex~d and an a~8erted CAS
signal ~ reoei~ed by e~ch C~S driv~r.
Durin~ me~ry write cycles, the prim~xy W~ ~ign~l ~ 8 generat~d by generator/checker 617. The as~erted W~ si~nal i provided by driver~ 630 eo eaeh DRA~ ba~k in memo~y array 600. However, a write can only ~e executed by the sQl~ad DRAM bank, which also raceive~ a~ert0d RAS and CAS signal~.
In tho praferr~d embodLm~nt of the inventlon, during ~emory tr~nsfer cycle~ the pr~m~ry RAS ~ignal i~ a~sertad during th~ T2 transition, 18 3t ble from at least ten nanos~conds before the T3 tr~n~itio~, and i3 de~erted during tha last T7 tr~n~ition. ~he pr~ary CAS 8ign~1 $~ i a~rt~d fifteen n~noseco~d~ afte~ e~h ~4 txa~sitionr and i~ l dea~Qrt~d during e~ch T7 tran~ition. During ~emo~y writ~ !
cycl~8 the pri~ry W~ 8ign~ a~ext~d during the T3 transition, ia stable from at leas~ ten n~no~econds before the ~irst T4 tra~ition, and i~ dea~rted during the last T7 txa~3i~0n.
Wh~n th~ prim~ry cycl~ typ~ 8ign~18 indlc~te a m~mo~y r~æe~h cycl~ i~ b~ing perform~d, ~en~r~tor/ch0c~er 617 cau~e~ ~ffmory axray 600 to p~r~orm ~ory re~resh ope~atlon~ ~
in rQ~po~ to th~ primary sequ~nc~ timing 8i~n~1~ provid*d by ~qu~nce~ 616. During the~ re~r~h operations, the RAS
a~d C~8 ~lgnal~ a~o g~n~t3~ ~nd di~t~ibu~d ~y the gen0r~0r/c~2cker ln re~er~ ord~r. Thi~ m~de o re~e~h r~quir~ no extern~l addr0~in~ for bank, rvw, or column.
~ Ol""C~
INNEC~N. HE~DER50N
F ~R~aOW CARRETr ~ D~.~NER
STl~tt. ~ ~. ' -- 2~ --Na~O~ . ~ c ~00--~0~ 6~0 ' ~: ' ;:
', 1, 2 ~ ~ 1 .` i :)uring tran~f/~r cycle~, ECC 8~ 8 are tran8fQrred or the tLme dlviYion multiplexed bidi~ection~l line~ of buse~ 80 I and 82 at times when data L9 bein~l trar~8ferred on ~u~ 135.
Howevsr, these same line~ are used to tran~fer control (e.g., ~, cycle type ) and addres~ ( e . g ., mqmory bo~rd addreaa as~d ~ank ¦ address ~ ~ignal~ at other tim~3G d~r$ng the transf er cycle.
I ~ The ~ran~ceiYer~/regi~ters in prim~ry/mirror deslgnator ¦ 612 includ~ receiver~ ~nd tranMmitt~rs that are respon~ive ~o I sequence timing signal~ and tx~rx rE~gi9ter control 3ignal :.; provided b~ sequencer 616. The 80quenc3 timing 13igJI~ 8 and tx/r~c regi~t~r control ~ligS~l8 en~ble multipl0xi~g of lSCC
~ignals and ~ddr~ nd control ~ignal~ orl th~ time di~i~lon multiplea~ idir~cti.on~l lin~0 of bu~e~ ~2.
Pr~er~bly, control ~nd addres~ signal~, ~uch a~ cycle typ~, m~mory bo~rd ~ddxe~ nd bank addre~ ~!5ign~l3, ar~
trans~itt3d by m~mory controller~ 70 and 75 an~ pre~ente~d on the time~har~ line~ of bu~a~ 80 and 82 at the beginn1ng of either ~ingl~ or multi-longword tr~n~i~0r cyol~. The~e I ~ atarl: th~ir tr~m~ltion (w~ile the ~equencGr ia in the I S13Q ID~l~ state) concurre~nt w~th ~ctivation of the cycla . ~ ti~ng ~lgna~, and r~main 3tabl~3 th:rough T2. Th3re~0re" in tho t~ca~l~c~iY~r~/r~gi~ter~ of d~ tor 61~, the rece~er~
~o ~bl~d and tho tr~n~mit er~, ar0 ~0t into their tri~tate .. mode ~t le~t until the ~nd of iq'tate S3~Q 2.
The cycl~ t~pa ~ign~ls identl~y which o~ the followlng ,~ . li~ted fu~ctions will be per~ormecl by me~Qory array 6~ ~uring ~wO,,,c~ ~ch* CyclQs m~aory read, mGmory wrLts, ~ tu~ regl~r ref~d, ,;. I~INECAN, HENDER50N
, .FARA~OW G~RRETr D~ EA
~,,",, ST~t~-.Y ~ I 25 G T 0 ~ . 0 C 2 0 0 0--~2021~ 0 j.',' "'~
''';
';' ':' '`,; . ' ~' ?

. . , I ~ ~ r 1 2 2 ~ ~
~tAtu~ regis~ex wrlte, ~PRO~ r~ad, ~BPRO~ wrlt~, and r~re~h. The designated prima~y cycle type ~lgnal~ reoQiv~d by de~i~nator 612 ar~ pro~ided to 3equ~ncer 616 and u~sd in l generating tx/r~ control ign~ls and ~equ~nce timing ~lgnals.
For exampl0, in data tran ceiver~/reglqters 620 and in the tran~c~iver~/regl~ter~ of dssignator 612, the r~c~ivsrs are enabled ~nd ~ha tran~mitters ~xe ~et into thsir ~ristate moda by sequencer 616 througheut a write cycl~. How~ver, Ln d~t~
tr~n3c~iv~rs/register3 62C and in tha ~ransc~ivers/regi ter3 of de~i~nator 612 during a read CyClQ, th~ rQceiv~r~ ar~ ~et into th~ir tri~tate mode and th0 transm~tt~rs are ~nabl~d by s~quenc~r 616 after the cycle typ~, m~mory board addxos~, and bank ~ddr~s 8i~n~18 have bean recalved a~ the beginning o~
'th~ cycle. !
In the preforrQd e~bodiment, da~a tran~ferred to or from mQmory array 600 l~ chQcked in Qach me~ory ~dul~ 60 u~ing an ~rror ~e~ecting Code (~DC), whlch i8 pr~fer~bly ~ha ~ame code ; r~quired by m~ory controller~ 70 and 75~ ThQ pr0ferr2d code i9 a 3ingle bLt correcting, doubl~ bit d~t~ctLng, error :~ correctin~ ~ode (~CC).
During a ~6mo~y write cycla, m~ory co~trollor 70 tr~o~it~ ~t l~a~t one longword o~ d~ta on dats bu~ 85 and ;~ ~ lt~a~o~ly tr~nsmlts a corre~pondin~ 3Qt o~ ECC ~lgnal3 .~ on ~u- 80. ~n~hile~ ~Q~ory con~roll0r ~5 tran~wit a s~eond ~ot o ~CC ~ign~la, which also correspond ~o tha l~ngword on da~a bu~ 85, on bu~ 82.
~W O~r~Ct-I~INECAN HENDE~S~3tJ
.:F.~R~30W G~RRE~r DUNNER
17T~ T-I S~'Z, N W. -- 2 6 NINOTON. O C zoooe 20~

~ " :
, l ~6,~2~
embodied herein, durin~ a m~ory write cycl~ the data and the ~CC slgnal~ for each longword are pre8e~ted to the ., receivera of data transceiverstregistar3 620 and to t~e receiYers of the transceivers/regi~ter~ ~f designator 612.
The data and tho ~CC qignals, which are ~table at le~8t ten n~nosQconds b~re tho T4 tran~itian and remain ~tabl~ until fifteen nano~conds after the T6 tran~ition, ~re latched into thea~ tran~ceivers/register~. During thi~ time period, memory contrQll~rs 7Q and 75 do not provld~ address and con~rol ~ignal~ on ~h~ tl~e~har~d lina~ of buses 3G and 82.
The designat~d primary ~CC ~ignal~ recaived by dQsignator 612 and the lon~word of data rec~1ved b~
tran ceiver~/regi~tar~ 620 during th~ m~ory write cycl~ ~r~
providsd to the data input~ of the DRa~8 in each of the eight banks of ~e~o~y array 600 a~d to ECC gen~rator 623. The gen~rated ECC i~ co~pared ~o ~ho de~ign~ted primary ECC by co~p~r~tor 625. ~he de~ignated prl~ry BCC ~i~nal~ al~o are provid~d ~o ~CC co~parator~ 625, ~age~h~r wi~h th~ de~$gnated mirror ~CC ~ig~als.
A~ e~b~dled her~in, d~r~g a m~ory r~ d ~ycle, at leas~
~ on~ long~o~d o~ da~ ~nd a correspo~ding set of RCC ~lgnaL~
a~0 ~oa~ frs~ m~ory ~rray 600 ~n~ r~p~ctlv~ly ~teered to : da~a tr~sc~iv~B/r~gi~ers 62a and to th~ ~ran~oiv~r~/
r~gi~t~r~7 0~ d~ign~tor 612. During tr~n~ition T7 of the m~mory read cyclo, ~he d~a ~nd ~h~ ~CC ~ignals for each I longword ~r~ ~v~ ble frD~ m~o~ arr~y 609 and aro la ched L~wOr,lc~. I in~o ~h~D~ transc~i~er~/r3g~ter~. The da~ al30 FINNECAN. HENDERSO-FARA8OW G~RRETr ~ DUNNER
177~ 1~ 5TRetT, R W. _ 2 7 w~s~ TO~. O c :OCO~
, 12011~0~0~90 , . : , .
. . ' ' " ' ~

~ 2 ~ ~
pre~ented to the ECC ~nerator 623 and it0 output i8 comparad to the ~C read from memory, After latching, the data and the ~CC 3ig~als are pres~nted to d~t~ bu9 85 and to buRe~ 80 and 82 by t~e . transmitter of data tran~ceiv2r~/re~i ter~ 620 and by ~he tran~mitt~r3 of the transceiv~r~/regi~t~r~ of d~slgnator 612.
~hs Rame ~CC s$g~Als are tr~n~ittQd fro~ th~ transcei~ers/
regi~ters in designator 612 to ~mory controller 70 and to . memory controller 75. The data And th~ ~CC slgn~l~
tran~mitted on tata bus 85 and on bu3e~ 80 and 82 ar~ ~t~bls from f$fte~n nanoseconds a~ter the ~7 tran~itlon until five ~, nano~eeonds befor~ the following T6 tran~ition ( in th~ caae of ~ ~lti-longword tran3fer3 or until fiY~ nano~econd~
before the following T ID~B tran~ition (in the cas~ of a ingle longword tr~n~er o~ ~he 1 ~t long~or~ of ~ m~lti-longword tr~n fer). During this time perlod, m~mory controllers 70 and 7S do not provid0 addre~ and control $gnal~ on the ~ime~h~r~d llnes of bu~es 80 and 82. ~he l tran~mitters o d~ta trPn~ceivers/regl te~c8 620 and the .. Itr~n~itter$ of th~ trasl~c~i~ver~/s~gl~tars o~ de~ tor 612 1 a~e 3et ~nto th~i t~t~ts ~nodo du~ng th~ ~ollow$slg ~ ID~
I san81t~0n-¦C~r~tor 614 i~ proYided ~o co~p~g~ ~h~ addre~, con~rol, ~nd tl~ing ~ign~ 8 origina~ing from cont~oller 70 ;~ Iwith the ~orr~ponding ~ddr~, con~rolr and 1;~ming ~ ls ¦ origin~ting fro~ coatroller 75. The da~ignatad primary cycle ~wo~ce~ . ti~ing 8ign~1~, cycle ~yp~ sig~ m~ory bo~rd ~ddras~
FlNNeGAN. HENDER50N
F.tRA901~ RRE~r li D9~N?`lER
17~ RttT, W W. l -- 28 W~ OTOW. O. C ~OOO-- ¦
(ZO~ O
':' '.
~,' '' - , .

'' ~ '' , " ~

~ 2 ~ ~ 1 signal~, and ~ank addre~9 ~ , tog~th~r with the da0ignat~ mirror cycla timing 3ignal~, cycle type ~ignal~, memory bo~rd addrQ~s signals, bank addre33 8i~1a11~, row . addre~s signals, and column addr~R ~i~nals, are provided i from de~ignator S12 to comparator 614. ThQ designatecl primary row addre~ als and colu~n addres~ ~ignal8 ar~
pro-rided from the output~ of drit~r~ 630 ~o comparator 614.
Both ~st~ of ~ignal~ are then compelr~d.
If there i~ ~ misco~np~r~ betwee~n any o~ th~ ~ddro ~ontrol, and timing 8ignal8 originating froDI the mQmory controllor~, comp~rator 614 g~nerates n appropri~tQ error sign~l. A8 ~hown in Ei~ur~ 6, ~ard addre~s exror, b~nk addr~ error, row addre~a error, colu~n addr~s errOÆ, cycle type addre~ ~rror and cycla t~ng ~rror ~ al~ ~y be , . output by the compar~to~.
Gener~tor~chsck~r 617 comp~r~o th~ pr~ry control as~d timing sign~la g~nar~ted ~y segu~nce~r 616 and generator/
chacker 617 u~ g the de3ignat~1 priffl~ry bas~lc address, cycle I type, ~nd cyc:le timins;~ ~ignal~ with th~ mirror control and ¦ t~ s ~ g~marat~ ua~nS~ the d~ at~l mlrror bas2k .; addre~o, cycl~ typ~, and cycl~ ~i~ng ~i~al~.. The two ~et .~ C~ ~110~11C0 l~ ng ~ 8 I!I~Ce prQ~YldT~ by ~ç~uencer 616 to 9~0æ~1tOXt~C~18CkOr 617r The p~ R~, CAS, and W}~ ~i~als .
a~ pro~id~d f~co~ ~he output~ of ~riYars 630 to genara~or/
chec~eer 617. A~ ind~c~t~d previoul31y, th~ ~rror RAS, CAS, and W~ ~ignal~. ar~ gen~rated inte~ lly by ~he ~en~ or/
.AwOrvlc~ checkE1r. Gener~tor/checker 617 co~paæsæ. the~ pxi~y ~, EC,~, HE~DeRSON ¦
F.~R~30W. G~RRE1r !
6 DU~;NER
IT7~ 1~ STl~t~ W. -- 2 9 ~5111.1~1T01~ Cl C 200011 120~9~0-50 .~ I
_ ,`;', ~ ' ;" , , ' ' ' ''" .~' :' ' 1 2~.2~ 1 CAS, WE, and s~quenc~ tim~ng 8ign~18 to the mirror RAS, CAS, W~, and ~equen~e ~ ng ~ignal~3.
I~ th~re is a mi~compare between any of the control ~nd timing ~i~n~ls originating from s~qu~nc~r 616 or generator/
check~r 617, the generator/checX~r generates an appropriat~
error ~ignal. A~ 3hown in Fi~ure 6, s~qu~neer error, RAS
~ error, CAS error, and W~ orror ~i~nal8 ~ay be output by generatortch~cker 617.
'IErrox ~i~nal~ are provided from compar.tor 614 and from generator/chacker 617 to addre~s~control Qrror logic 621. In respon~e to receip~ of an error ~ignal frsm ~omp~rator 614 or : from gencrator/ch~cker 617, addre~s~oontrol ~rror logic 621 tran~mlt~ an addre~s/control ~rror ~igna ~ module 30 to indicate the d~tQct$on of a fault due to a mi~co~pare ~st~en any address, control, or tlmlng signal~. Th~ addres~control error 8ign~ se~t to ~rror logic in memory controller~ 70 and 75 fcr error handling. ~he trans~ission of the addr~R/
con~rol error signal to CPU m~dule 30 cause~ ~ CPU/ME~ fault, which i~ discuss~d in gre~ter detail in oth~r ~actlon~.
: ~h~ ~rsor si~nals fro~ cc~p~ra~or 614 and fro~
:, genorator/cheek~r 617 al~s are pro~id~d to ~tatus reqi~ters ; 618. In tha ~t~tus register~, the e ror ~i~nal~ and all of tho ~3~0~t control, t~ming, d~ta, and ~CC ~ignal~ r~levant to th~ fault ~r~ t~mpor~rlly stored ~o anabl~ error diagno~i~
and recovery.
.In accordanc~ with ons aspe~t of th~ inventlon, only a 8inglc thirty-two bit dat~ bus 85 ~8 proYr~d~d ~tween CPU
-IN~E~N, HE-~DeR50N
F.~RAEOW. C;,~RReTr 6 DUN~;ER
In~ K ~r--err. ~ . ~ 3 0 w~ TO~ O ~ ~ooo~
(~0~1~0~ ac50 . . I

': , 2~22,~
module 30 and memory module 60. Th~refora, memory module 60 c~nnot co~ara two ~eta of data from memory controllor~ 70 and 75. Howev~r, data integ~ity i~ verified by ~emory ~o ul~
60 withou u~inq a duplicate 30t c)f thirty-two data line~ by chacklng the two separate ~et~ of ECC signals that are tran~ tted by memory co~troll~rs 70 and 75 I:o memory module 6~.
A~ ~hown in Fig. 6, control logic 610 include~ ECC
generator 623 and ~3CC comparators 625. The~ desigII~ted .~ primary and mirror ECC ~ are pxo~d~cl by de~ or 612 .: ~o the l~CC compar~tors. During a me3~ory ~rite cycle, ~he designat~d primary ECt: 8~ 8 as~ co~paresd to the de~ ted mirror l~CC si~al~ a r~P~ult, me~ory module 60 verii~s wheth~r m~mory con~rolle~ 70 ~nd 7S are in agre~e~t and ¦
whethox th-3 ds~lgn~ter~ pri~Dary ~CC ~lgnal~ beir~çl 3t~xecl la !~ the DE~S~ of m~3mory arr~y 600 during the memory write cycl ar~ corr~ct. Furth~more, tha d~ta pr~nt~d ~o ~ho dza~a input~ o th~ DR~l~ during the m~mory write cycle i8 pro~7id0d ~o ~CC genor~tor 623. 13CC ~en~r~tor 523 producs~s a set of g~n~rat~d ~GC ~lgnals tha~ corr~apond ~o the da~ nd provld~ ~ch~ g~0r~t~d ~CC 8~ 1 to ~CC comparatur~ ~2S.
Th~a d~ t~ pri~ry ~CC . ign21~ ~r~ compared to th~
gOE7no~t~i ~CC ~ nal~ to vlari~y wh~ther the dat~ t~n~mi~ted ' on dat~ bu~ ~5 by m~}mory controll~r 70 i~7 th~ sa~e ~ the da~ b~ing ~or~d in th~ DR~7 o~ m~n~ rr~y 60û.

~uring a D~ory r~ad cy~::le, th~ da a r~d f~o~a thff ~AWO?r~Ce~ ~1HCt~d bank of DR~Pqs i~ pre~ent~d to the ~CC gE~ne~ator.
-INNE~N HE~DERSON
F,~R~EOW GARRETr ~ DUNNER
,77, ~ ,Tr.~ . w -- 3 1 ~YIY~Or~.O c ~000--110~1 ~9~ 0 ,~ ~ '. `
`~
,' ' ' . ' ' ~ ' ! 2~22~ 1 I . 1.
The gener~ted BCC 8ign~18 then ~re pxo~id~ to th~ ~CC
comparator~, which ~l~o rocelve 3tor~d ECC ~ re~d f rom ~h~ 8~1act~d ban3c of DRAM3. The generated and stor~d ECC
~igl'1~13 ara compared by ~CC comparator3 6 25 .
If there i~ a mi~compare be't:ween ~ny of pairs of ECC
¦ ~ignal~ monitored by ~CC comparator~ 62~, the ECC comp~ratorq I generate an appropriate error 9ignal. As ~hown in Fi~ure 6, ! prima~/mirror scc error, pr~ary/generatsd E~CC error, and I m~amory/~anerAted ~cc error 8~ gXlalB may be~ output by th~ ~CC

comparators.
These ~CC error 8~dl~ froDIl ~CC ~o~p~ra~ors 625 ar~a provided to ~tatuP- regiater~ 61a. In the 8tatu8 r*gi~er~, ¦
each of ~ha ~CC error 8igna~18 a~d all of th~ addre~
control, timing, data, and ~CC si~ r~l~vant to an ~CG
fault ar~ te~por~rlly 0torod t~ ~anabl~ ~rror dlagno~i~ and recovery.
An ~CC errer sign~ a~se~ted by BCC compar~ors 625 on an ECC ~3rror lln~ ~ad tr~n~itt~cl to CPU module 30 to indic~tt3 ths d~tç~ctio~l of ~n E~CC ~ ult due to a ~i~co~pPre.
The mi~com~?aro c~n occur d~ring 0ith~3r of the two ~CC chsclcY
peron~d dur~n~ ~ m~oxy writ~ cycle, or dur~slg th~ ~ln~l~
~CC ¢h~!ack p~xfor~ed during ~ m~o~y re~d oycle.
A~ ~ho~dn in Pigure 6, board ~ele~:~ logi.c 627 r~calve~
olos ~ ~o~ ~ ~e~ory bac:kpl~ . Th6~ 810t ~lgn~l~
speci~y ~ 810~; locatlon or e~ch m~mory ~odule 60.
Bo~rd 31E~lli\C~: lo5~iG 627 then comp~res the ~lo~ ignals with ~worr~c~ the design~t~d prlmary be~rd add~:a~ i~al~ ~r~n~mitted from 3NNECAN, HENDER;ON
FARA~OW. ~ARRETr 6 DU~ER
1~75 11 5~RetT. N W. ~ 3 2 ~NINO~ON. O C iooo~
120~ 61~0 ' ' ~

2 ~ ~ 1 ona of the memory corltroller~ vi~ deai~n~tor circuit 612. A
board saloc:ted ~ignal i8 gensrat~d by board 891e~ct logic 627 if th~ slot si~al~ are the ~ame as th~ d~ at~d primary boaxd addre~ ~ignals r thereby sn~bling th~ othsr circuitry co~trol logic 610.
3 . ~emor~ Contro~l~, Nemory controller~ 70 and 75 control the acce3s of CPU~
40 and S0, re~pectively, to memory module 60, auxiliary memory ~lem~nt~ and, in th~ pr~f ~rrsd embodiment, perf orm certain error handllng oper~tion~, The au~lllary m~ory elsmen~s coupled to memory cont~oller 70 includ0 ~ystem ROPI
43, E~PR0~ 44, and ~crAtch pad ~ 45 R0~ 43 hold~ ~e:~in ~Sand~rd code, ~uch ~!18 di~gnostlc~, con~olQ driver~, ~nd E~r~
of the boot~trap code. ~13PRO~l 44 i8 UE513d to hold inf onnation ~uch a0 error inf or~atlon ~etected during the operation of CPU 40~ which ~y ~a00d to bo modified, bu~ whîch 3hs: uld not be lost when pow~r 1~ r~soved . Scra~ch pad R~ ~5 L~ u~3d for cQrtatn oper~tiona p~rfonn~d by CPIJ 40 an~ ~o conv~rt rail~ lnfomlstion (e.g. ~ informatio~ 8p~c~ fic:
to condition~ o~ cm3 rail whlch is ~ra~lable to only ola~ C~V
40 or 50) ~o ~on~ fon~tlon (e.g.~ information which can be acc~ by both CPU~ 40 and 50)~
~ quival~n~ ~lem~nt~ 53, 54 ~nd ~5 are coupl~d to mQmory' con~roller 75. Syqt~m R~ 53, E~PROn 54, ~n~ scr~ch pad 55 ar~ the ~ Al~l 13yl31:~DII R0~ 43, 2~PR0~ 44, and ~cr~ch p~d RA~ 45, r7~ tiYely, and p~rforle th~ 8~t~ function~.
~w 0~
INNEC~N HENDeR50N
:, F.~R,~30W GARRETr h Du~ JeR
. ~ 17~ T, N W. -- 3 3 w~ a~o~ a c looo~
120AI j~9~ 1~51~0 : ' ' ,~
:~
' ` ' ~ 2 ~
The detail~ of the preferred e~bodiment of primary ma~ory controlles 70 c~m be seen in Fig~, 7-9. Mirror me~ory controller 75 has the ~ame e~ment~ as ~hown ln ~ig8. 7~9, bu~ diffars 31ightly in op~ration, Therefore, o~ly pr~ry memory controllar 7 0 ' ~ oper~tion will b63 d~scribed, ea~cept wh~re the operation of m~mory co~troller 7S di fer~. ~emory controllers 70~ and 75~ in proc~sing 8y5t~m 20~ have the same alements and act the 8ame ~ m~mory contrcll~r~ 70 and 7S, re3pectivoly.
Th~ elaments shown in Fig. 7 control the flow sf dat~, addr~sas and ~ignal~ through prLmary ~emory controller 70.
Control logi~ 700 ~ontrol~ the s~at~ o~ th6~ v~riou~ el~m~nts i~ Fi~. 7 a~cording to the ~ignh~ c~iv0d by llt~mGry controller 70 and th~ st~t~ eng~n~ of th~t m~mory co~troller which i~ ~tored ln control logis 700. ~qul~lpl~r 702 selec~s addre~se~ fro~ one of thr~ ourcea. The ~d~rf~
cAn eithQr eom~ fro~ CPU 30 via recQiver 705, from th~ D~A :
engine ~00 dQ0crib~d b~low in re~erence to Flg. 8, or from a rQfre~h ra~ync ~ddrl3s~ line which i~ u~ ad to gen~ra~e an artificl~l r~fr~h during cert~in bullc memory tran~r~ from onta zone to ~no~h~r during resy2lchroniz~tion op0ra~10n~.
q!ho output of ~ultipl~ax~r 702 i dll input ~o ~hultipleace~
710, ~ da~ fro~ CPU 3U recQiv~d via receiver 705 and dat~ from D~A engln~ 800. The output o~ mul~iplo~r 710 provide~ d~ta to me~ory module 60 ~18 ~A~Ory interconn~c~ 85 and driver 715. Driv~r 715 L~ di~abled i'or ~irro~ mamory ~ or~lc~
FINNEC~N HeNDeR50N
FARAaOW, CARRe~r ~ D~NN~R
177~ ~ 9Tl-ttT. ~. ~. 34 ~A~ TO~. O C ~000 9 ~ ~ 0 ~ 0 '' ~ " ' "

control m~dules 75 and 7S' b9cau8e only one ~et o m~mory dAta i~ sent to memory :module~ S0 and 60', re~p0cti~ely, The data sent to memor~ interconnect R5 iaclude~ eithar data to be ~tored in memory module 60 from CPV 30 or DMA
I ~ngin~ 800. Data from CPU 30 and addresse~ from mul~lplex~r 70~ are al30 ~ent to D~ engine 8û0 via this path and also l - vi~ receLv~r 745 asld ECC corrector 750.
I Th~ addresse~ from multiplexer 702 also provide an input to demultiplexer 720 which dlvides the addrs~es into a row/column addre~ portion, a board/b~r~X address port~on, and a single board bit~ Th~ cwenty~t~o bit~ o~ the row/column addre~ ar~ Dlultiplexed onto eleven l~e8- In the pre~0rr~d i embcdi~ent, the twenty-two ro~/~olumn ~dd~lt~ ar~ ~e~t to m~ory module 60 via drivQr~ 721. ~rh~ ~ingl~ board bit is prefera~ly ~Qnt to ~mory module 60 vla driver 722, and th~
other board/bar~lc addre~s bits ar~ ~ultiplexacl with ECC
nals.
~ ultiplexer 725 c~mbine- a nor~al refrYI~h command for memory controller 70 along wi~h cycle typ~ in:Eorma~ion frolo CPU 30 (i.e., read, writ~a, etc. ) and DMA cy~le type i~forma-tion. ~h~ nar~l r~fresh co~nd and tha refre~h r~sqync ~d~ :
dr~ both cau~o 3~0ry ~odul6~ 60 to initiat~a a m~mory refr~ h op~ra~i.on.
~ output of multiplexer 725 i~ an ~nput to ~ultiplexer730 alons~ with th~ board/banlc addre~ froD~ d~ultiplsxer 720.
Anoth~r input into multiplexer 730 i~ th~ output of 2CC
A~o ~ generator/checke~ 735 ~ultiplç~xer 730 ~ eots ona of the :~NECAN, HENDER50N
FARA30W G~RRET~
a D~ eR
1". ~ ~r~T .. w -- 35 --.Y~IIII~G70r~ 3. C 2000--120~ 0 '~

.
' :.
: "~ ' '~ " ' ' ~nputs and pl~ce~ lt on the time-division multlple~ced ~CG/
addre~ ne~ to s~emo~ module 60. ISultiplexe~ 730 allowa tho~e ti~e-divii$orl rnultl.plexed line~ t~ carry board/ban~c ad-dres~ and additional control in~o~tion a~ w~ll as ECC
information, al~hough at different time~.
ECC informatiorl i~ receiv~ from memory modules 60 via r~c~iver 734 and ~ 3 pro~lded as an input to 13GC g~n~rator/
checke~ 735 tv comp~re~ the ECC genar~ited by memory module 60 with th~t ge~nera~d by memory controllsr 70.
Another input into ECC geners~or~h~cker 735 is the oul:put of multiplexer 740. Depanding upon wheth~r ~h~ ~nory trans~iction i~ ~ wri~e tran~action or a re~id tran~actiorl, ¦
laultiplea~er 740 recQi~res a~ inpu'c~ the m~mory data sen~ to me~iory module 60 fro~ multipleacer 710 or th~ m~ory data r~c~ived from m~mory module ~0 vi~ rec~iver 7~5. ~ultiplex0r 740 ~ t8 on~ of the~e 8e~ of D~e~ory da~a ~o bc the input to ~CC gen~ra~or/check~r 735O Gen~ra~c)r/checXer 735 ~han ganerates thçl appropriato ~CC cod~ which, in 2ddltion to b~aing sQnt to mul tipl~xer 730, i~ al~o ~3nt to ~CC correcitor 75û. rn th~ pregerred em~ nt~ ~CC ~orrec~or 750 correc~:s any 8in~1~ blt ~rror~ in ths rse~ory data received from m~mory modul~ 60.
Th9 corr~ t~d m~aory d~ta from ~CC chacker 753 i~ then ~nt to thæ D~ ~ngin~ Dhown ln Fig. 8 a~ w~ll a~ to ~ultipl~xer 752. ~rhe oth~r inpu~ into multiplZaxsr 752 i~
error infor~lon fro~ thc erro:~: handAling logic delcribed ~A~ O~ S~
:NNeC~N. HæNDERSON
F.'~RA~OW C~RRE~r E~ DUNNER
177~ Ct't. ~ ~. -- 3 6 ~snll~o~Ot~ c ~ooo~
0 ~ ) ~ 9 ~ 0 '.~

" ` : ~ ,. , '-, ':

~ ~ l below in conn~ction With ~L~J- 9, The output of m~ltiplexer 752 i8 ~lllt ~0 CPU 30 via drlYer 753.
Co~parator 755 comp~re~ the d~ta 98~1t from multiplexer 710 to memory module 60 with a copy of 'chat data aft~r it pas~es through drivQr 715 and rec~iver 745. Thi2 checking datermin~ whether driver 715 and receiY~r 745 are opexating correctly. The output of comp~rator 755 i~ a C~P error signal which indicates tha pre~ence or absenc~ of such a comparison error. The CMP error feed~ the error logic in Fig. 9.
Two other elemen~s in Fig. 7 pro~ide a different kind of error d~stection. Elemont 760 i8 a parl~y g~ner~tor. ECC
d~ , generated ~i~her by ~he ~ory ~on~roll~r 70 on da~ to b~ ~ored in meslory medule 60 or generated b~ m~ory module 60 on data re~d from memc~ry module 60 i8 s~nt to a parity generator 760. The par~ty ~ignal from g~nerator 760 i~ ~ent, via driver 762, 'co co~par~tor 765. Co~p~rator 765 comp~re~
the ECl' pArity ~ignal ~ro~ gsnerator 760 with an equivalent ECC p~rity ~Ls~nal ge~nerated by ~ontroller 75'.
Parlty g~n~rator 770 perform~ th~ typ~ of a check on tho ro~/col~L ~nd ~ lG bit board addre~ ignals rec~ d fro~ d~ultipl~xor 720. The addrla~s p~rity . ign~l ros~l psxlty g~ r~or 770 i~ ~ranan~t~d by ~ driYer 772 to a compara~or 775 which al~o rec6~ive~s ar~ ~ddre~s parity signal fr~m cos~troller 75. ~rhe outpllts of eo~pax~tor 765 and 775 are p~ri~y erxor ~ign 18 which ~e~l th~ ~rro:r logi~ in Fig.
~w O~ c~- 9 .
I~NEC,~N HENDERSOY
F~R/~EOW ~ARRE~T
~i D ~ E R
1~9 ~ sT~c~T ~. ~ 3 7 oTol~. o C ~OOO--~o~ o , I

., ' ' :

:
.

I ~ 2 ~ ~
F~g, ~ ~how~ the fundamentalB of a DMA engin0 800. In tha p~e~rr~d ~bcdiment, DMA ~ngi~e 800 re3ide8 ~ memory con.roller 70, but th~e i8 no requira~ent for ~uch pla~ement. A~ ~hown in Fig. 8, DMA enginQ 800 include~ a data router 810~ a ~MA control 820, and D~A r~giAter~ ~30.
Driv~r ~5 and recQiver 816 provide ~n interfac~ b0twe~n memory controller 70 and cro3~-linX 99~
DMA control 820 receivQa in~ernal control ~ignal~ from control logic 700 ~nd, in r~spons~, 80nda con~rol 3ign~1~ to-place data router 810 into the appropria~e configur~tion.
Control ~20 l~o causes d~ta routar 810 to Ye~ it~
conflgur~tion to rout~ d~t~ nd control ~ign~l~ frsm cro~
link 90 to the me~ory control 70 cir~uitry shown in Pi~. 7.
D~ta router 810 sa~d~ it3 8ta~u8 si~nals to DXA control 820 whlch r~lay~ ~uch ~lgnal~, along with o~her ~NA inform2tion, to error logi~ in F~g. 9.
Regis~er~ 830 include~ a D~A byt~ counter regi~tar 832 and a D~A addre~ register 836. ~hes~ regl~rs ar~ set to initi~ lu~3 by CPU 40 via routar 810. Th~, during DMA
cycle~, control 820 caua~ , ~ia router 810, the counter ragist~r a32 to incre~nt and ~ddr~s r2gi ter 836 to de~æ~t. Cont~ol 820 al~o cau~ the content~ of addra~
r~gl~ s 836 ~o ~ s~n~ to ~ory ~odul~ 60 through router 810 ~d the ~ircuitry in Plg. 7 duri~ D~A operation~.
A~ e~plzl~d abo~e, in tha pref~xr~d ~bodi~ent o~ thi~

inven~io~, th~ momory co~roll~r~ 70, 75, 70' and 75/ al~o ~wo~e~ 2er~or~ certain undamsnt~l error operatlon~. ~n sxam~le of FlNNeC-~N. HENDeR50tJ
F.~R.~50W, G~RRETr a D~ eR
177~ ~ STR~T, ~w - 38 .VA~ OT0~ . 0 C ~000--120~ 29~ 0~0 . ~ ' ;

2 ~ f~ 2 2 ~ ~ ~
the pref erred QIQ~odim~llt of the h~rdware to perf orm such er~
ror operations are ~hown in Fig. 9.
A~ qhown in Fig. 9, certain me~3ry controller internal ~ignal~, ~uch as timeout, ECC error and bu~ isco~par~, are inputs into diagno~tic error logic 870, as ~re cartain external ~ignal~ ~uch a~ rall error, firew~ll miscompare, and address/control orrox. In tho pr~3f~rred em~odlment, diagnoatLc error logic 870 recei~r~s error sigT~al~ from the oth~r co~nponents of sy~t~ 10 vi~ ~:ros~ nka 9û and 9S.
Diagno~tic error log~ c 870 ~Or~113 errox pul~e~ Lrom She error igrlal~ and f rom a control pul~ sLgnal g~nsr~t~d f rom the b~sic ti~ng of memory con~;roller 70. Th~ error pul8e~
genoral:~ by dl~gno~tic ~rror loyic 870 cont~in carta~Ln 6~ror in~orm~tion which i~ ~toxod lnto ~pprop~$ate loG~tions ln A
d~agnostic errE~r re~i~to~c 880 ~n acco2~dance ~Lth cer~aln timing ~Lgn~l~. 5y8telll f~ult e~or add~3s~ register 865 5tOrle8 th~ addr~ in memo~y module 60 which CPUs 40 and 50 wers communic~ting with when an e~rc~r oc:curr~d.
Th~3 ~rror pul~Q~ ~rom diagno~tic ~r:l:ox logic 870 are al~o ~3nt ~o ~ror c~ go~ization logic 850 wh~ ch z~180 receLvs~ infox~ation fro~n CPU 30 lndic~t~3lg thel cycle type (~.g., ro~d, weiL~s, es:c:.). Fro~ ~ha~ infor~a.Lorl and the er~or ~ 8elll3 ~ o~cror ca~egoriz~io~ lo~ic 1350 determine~ the pr~s~3nc~la o~ PU/I0 61rror~, DMA 6rror~, or CPIIJI~ fault~.
~ CPU/I0 error ~ ~ an error on ~n op~ration ~h~t i~
direetly attrlbutable ~o a CPU/I0 cy~l~ c~ bu~ ~6 and m~y be h~rdware reco~v~arablQ, a~ expl~ined below in regard to re~9t5.
ECA!J. HE~IDERSON
F.~R~80W C~RRETr ~i D ~ E R
5~ . -- 3 9 ~n~ To~. o ~ ~ooo~
1202~ 0 : _ ~ ~3 ~

DMA errors are ar~or8 1:hat occur duriny ~ D~ cycle and, in the pra~err~d embodiment, ara h~ndled principally by ~oftw~re. CPU/ME~ fault~ ar~ error~ that for which the . correct operation of CPU or the contents Qf memory calmo~ be ¦ guaranteed .
j The ou~cput~ from error catagorization logic ~50 ar~ sent ¦ to encod~r 955 which forms a ~pQcific orror cod~. Thi~ error cod~ hen ~ent to cro~s-links 90 and 95 via ~ND ga~e 856 when the error disable signal i~ not present.
After r~c~iving th~ error cod~s, Gro~-links 90, 95, 90 and 95 ' sQnd a r~try requs~t siqnal back to the memory controll~rs 0 A~ ~hown in F~ g . 9, an encod~sr 895 in mamory controll~r ~0 receiv~0 the rQtry r~que~t~~along ~rlth cycl~ infon~lation and th~3 orror 3iSIa~ COlleCtiY~ y qhown ~ cy~le qualifier~ 3 . ~n~od~r 895 ~hen g~n~arat~ n appropria'c~ errc~r cod~ f or ~torage ln a 8y81:em f ault srror registar 898.
Sy~t~ fault ~rror r~gist~ar 898 do~ not ~tore the Ram9 informeltio~ ~L8 di~gno~tic ~rror r~gi~t~r 880. Unlika the ~y~ce~ f aull: error reg~ster 8 9 8, the~ dl~gnos~ic Qrr~?r reç~i~ter 880 only eontaln~ r~il uni~ue inform~tion, ~uch as an ~rror on on~ input fro~ A CrOelel~ k rail, and 20ne lmique da~a, ~u~h a~l an uncorrectable ~CC ~rror iII memory module 6û.`
.~ Sy8t~ ault error regi~ter ~98 al~o co~ta$ns ~ver~l bits which ar~ ~ for error har~dling. 'rhes,a i~Lclud~ a NXM
bit indicat~ng that a desi3:e~d mems~ry locat~ or~ i8 ml~ins~, a L~lwO~iC~ NX~O bit indic~ting th2t a desir~sd I/O location 1~ ing, a FINNEC,~N HENDER50N ¦
F.~R~gO9V CARRET7.
a DUN~ER
9TIII~T, r~ W. ~ 4 0 W~ 070, . c,. C ,000-~o~ u~o l l , -- . .

~olid fault b$t and a tran8ient bit. The tr~nslent and ~olid bit~ ~ogethor ls~dic~tg the f ault la~lrel . Th~a tran~ien~ bi~
al~o c~u~e~ ~y~tem fault error addrQas registex 865 to f re~e .
~ lemory controll~r st~tua register 8~5, although techrli~ally not part of the arror logic, 1~ ~ho~n in Fig. 9 also. Regi~ter 875 3tor~8 certain statu~ inform~tion uch a~
a DNA ratio cod~ in D~ r~tio portion 877, a~s error di~abl~
codQ in error disable portion 878, a~d a mirror bus driver e~n~ble code in mirror ~u~ driver en~ble portion 876. The ~
ratio code ~pecifie~ tha fraction of ~e~ory bandwldth which c~n be allotted to D~IA. Th~ ~rror di~abla codo proYides a 2ignl~ for di3ablln~ AnD g~t~ 856 z~nd thu~ tho srro~ co~
T~e m~r:cor bus driYar an~bl0 code p~o~v~de~ ~ ~lgnal for enabling th~ ~irror bu~ drivers ~or ce~t~in d~a tran~actions .

4- 5~Q~Li!i~
Z:ata for mffmory resync, D~a an I/O operatiorl~ pa3i~
through cro~ nks 90 and 95. G~ner~lly, cros~ links 9û and 95 pro~fld~ co~ic~tlon~ betweon CPU ~odsle 30, CPU module 30', I/O ~dul~ lO0, llO, 120, ~r~d l/O modul~ lO0', 110', 120' (~ q. 1).
C~o~ 90 ~nd 95 contain both par~llel reglster~
910 a.~d s~r~l r~i~tq~r~ 920 A~ 3ho~ in Flg. lO. ~3o~h type~
of r~g~l8t~ are u~d for in~rproc~a~or c:omm~anic~atlon ~n the pr~ferr~ ~æbod~D3Qn~ o~ thi~ invarltion . Dur~ ng no~
~WO~C~ op~ratiorl, proce~ing y8tem~ 20 andl 20' area synchror~ d and 8NNEG~N, HENDER~ON
FARA~OW~ GARRelr ~i DuNNeR
177~ n sTnteT, t.~. w. -- 41 w~9rlll~1TOt~, 0. C ~0000 ~20~ 21~ 0 , data i9 eacch~nged in p~r~llel 'D~tW~3n proceBsing 8y8te~1~J 20 ~n~ 20 ' using parallel r~gi8ter~ 910 in cro~ 90/95 and 90 ' /95 ', l:e3peCtiVQly. ~en pxoce~in~ ~y~tem~ 20 and ~0 ' ar~ not 3ynchronized, mo~t notably during boot~trapping, data i~ exchang~d b~tween cro88-links by way of ~erLaL register~
l 920 .
¦ Th~3 addre ~ of th~ parallQl reglster~ ar~ in I /0 ~pace a~ opposed to ~ ory spaca. ~IQ~O~ ~pelCe re~ers to lot::3tion~;
in memory module 60. I/O space r~sr~ to loc:atlons such as I ~O and internal ~y~t~m re~g$5tex8 9 wh~ch a~e no'c in m~mory module 6 0 .
Within I/O ~pac0, addr0~e~ ca~ either be in ~y8~
addre~s ~ ce or zon~ addr~ pace . Tha ~erm 1~ 9y8~C~I , I
addres~ ~pace" ref~r~ to addre~3e~ th~t ar0 a~cas~lbl~
~hrous~hout he e~tire sy~t6m lO, and ~hus by both proce~sing sy~tem8 i!O and 20 ' . Th~ t~ zone addres~ ~p~cel~ r~fer~ to adtxe~a~ ~h~ch a~e ~cce~ lQ only by the zone conta~nialg ~he p~ticul~ çrs~ nk.
Thel p~rallel registars ~hown in Fig. lO irlclude a com-:~ munic~tl,on~ r3gi~ex 906 ~nd an I/0 re~et r~g~ ~t~r 90~ . Com-mun~ catior~s resgidt~x 906 coTlt~ln~ da~ to be e~ch~nge~d b~tw~ zons~. SuGh d~t~ i8 u~ually zsn~ ue, ~ch ~ a m~xy ~o~t 02er~ ~it ~ ~lms~t beyond the r~ n of pxo~blli~cy th~at ~ao~ modull3~ 60 ~d 60~ would Lndep~nd~ntly eJcperi~nG~ th~ e~ror at ~he s~m~ tim~).
BlaC~U138 the d~ to be 8~0~ed into r~gl~er 906 i8 .~wOr~ g un~qu0, tho add~0s~ o~ com~unlca~cion~ r~gi~tE3r ~06 for ;INNECAN HENDeR50N
FARAaOW GARRETT
6 DL'~;:`IER
177~ ~ gr~rCr. ~ ~. -- 4 2 nlnGrO~0 O C 2000--1~0~ 0~1~0 "''~' ' ', ~", ` ' . ;

2 ~ d~
l purposes o~ w:riting muat be ir xor~e addre~ space.

I Ol:he~wise, proces~3ing 8y8t~ma 20 and 20 ', becaus~ they are in loclcstep RynchronizatiOn and executing the ~ame ~rlaa of instruction at ~ tantially the ~ time, could not store z~ne unique dat~ int~ only the communication~ ragl~te~a 906 in ~orle 11; thay would ha~ to stor~ th~t 5am0 data in~o the i COmm~miCAtiorl~ regl~t~r~ 906' (no~ ~hown) in zone 11'.
¦ The~ addreas o~ communication~ regl~t~r 906 for r~ading, howover, i~ in ~yatem addr~s ~pace. Thus, during hronous operation, both zon2~ can ~imultaneously re~d th~
communications regi~t~r fro3n on~ zono ~nd the~n si~ul~neously re~d th0 communications ra~i~tçlr Pro~ th~ other zone. I
I/0 re~et regi~ter goa re~id~ n 3y8tem addresa 8pl~1CI~.
Tho I~0 re~et r~s~is'cer includlss on~ bit per I~0 modul~ to indicat~ whether ~he correspondin~ ~odule i~ in a r~et ~tate. When an IiO modul~ i~ in a res~t ~ate, it i~
ef f ectlvely disal:~lQd .
Parallel regi~t~r~ 910 al80 include othlar re~ ar~ ut an und~r~nd~ng of tho~ other r~gls~ar~ iB not nec~aa~ to an underst:ancling of ~h~ pre~ent inve2~tion.
All of tho ~orial cro~-link regi~tQr~ 9 20 are in the zorl~ specific ~p~c~ sinca they ar~ u~ed either f c~r asyRchronou~ co~unlcation or con~Ain only zone ~peci~ic ¦ infos~ation. ~ purpo~ of th~ serial cro~ link reglsters and thQ ~ri~l cro~-link i8 to allow proc~s~ors 20 and 2D~
to co~aunicate eY~rl though they ~ no~ running in looX~ep i~ynchronizntion (i.e., pha~e lock~d clock~ and j9llmÇ' mamCI~7 .~iNNEC~N, HENDERSON
F.'~R~EOW G~RRETr ~i Dl~NNeR
1~7-- ~ ZiTRt~T, 1~ W. 4 3 W~ll?lOTOtl. D. C ~00011 120~) ~9~'0--0 .;'`
. ' '' ' , ' ' . , ' ' l ~22~
t~t~). In the p~ef~rred em~odlmant, thsre are several ~arial regi~tar~, bu~ they ne~d not be de~crib~d to undeEstand this invention.
. Control and ~tatus register 912 i~ a s~rial regi~ter , which contains ~tatu~ and control flags. One of the flag~ i8 an OSR bl~ 913 whlch i3 u~d ~or boot~trapplng and indica~es wheth~r the proce~sing ~y~tQ~ in the corre~pondln~ zon~ h~s already begun its bootstrapplng proc~ or whether the opera~ing ~y~m for that zone 1~ currently runnin~ either bec~u~o its boot~trapping proce~ ha~ complet~d, or be~au8e it undarwont a ra~ynchronlzation.
Con~rol and ~ta~u3 reg$~ter 912 al80 contain the mod~
bit~ 91~ for identifylng the ~urre~t m~de of cro~link 90 . and thu~ o~ proc~ing ~y~tam 20. Pref~rably moda blt~
include re8yXlC mode blt 915 ~nd crosa~ k mods bits 916.
~e~ync mode bit~ 915 Ldontify cros~-link 90 a~ bein~ elther i~ re~y~c s1av~ or ~e~ync ~tor mod~O Th~ cros~-link mod~
bit~ 916 identi~y cro~ nk 90 ~ being ~ither in cross-link off, duple~/ cro~ k master, or cro~-1ink slave mode.
: On~ of tho u~e8 f or the ~erial r~g~t~rs i3 a 3tatu#
re~d op~ratlo~ which allows the cro~-link ln on~ zone to : raad th~ ~t~tua of the other zon~'~ cro~-l$nX. Set~ng a ~ st~tu~ r~d r~qu~3t 1ag ~18 in so~i~l co~trol ~t ~t~u~ ~
r~gi~t~r 912 ~and~ A requ~t ~o~ ~t~ ormation to croe~
link 90'. Upon raceipt of thi~ mea ~09 cro~a-link 90' ~end~
the cont0nt~ of it~ ~eri~l cont~ol ~nd tatus r~gist~r 912' ; ~AWO~C~ ba~k tQ ~ro~ 11nk 30.
. ~' FINI`JEG~N. HeNDER50N
F.~R~ RReTr ~ DUNNER
1~7~ ~ ST17St7, N ~. -- 4 4 ` ~ W~S~llNC~rO)~ O C 200011 20~93~1J-0 `';
. ~ ,, .

`': ' , : , ' .
' ' ':`
- '; ':

~ 2 ~ ?, ~ ~ I
~ ig. 11 show~ some of the element9 ~or routing con~rol and 3tatu8 ~iqnals (referred to a~ ontrol codes" 3 in primary cro~ link 90 and mirror cro3s-link 95.
Corr~sponding cross-link el~ment~ axi3t in the preferred ~ embodiment within cro~s-links 90 / and 95 ' . ~rhe~a code~ are i ~ent betwaan the memory controller~ 70 an~ 75 and the I/O
~ moduleæ couplQd to moduLe int~xconslact~ 130, 132, l~0' ant l 132' .
I Fig. 12 ~how~ th~a ale~n~nts in the pre~e~rred e~odiment of prim~ry cro~-link 90 whlch are u~ed ~ox routlng data and addre~ ~Lgn~ls. Corre~l?onding cro~s link elem~3nts exi~t in cross-link~ 9S, 90' and 9S'.
rn Plg. 11, tha ale}lleJlt for both tb~ry cro~s 1ink 90 ~nd mirro~ cro~R-link 95 in proce~ing ~y~te~n 20 ar~ ¦
shown, although the hardware i~ idsnt~c~l~ becau~ of an import~nt int~rconnection betwe~n the eleD~snt~. ~he clrcuit ~l~m~nt~ in mirror cros~-linlc 95 which ~ ralent to elements in primAry cro~-link 90 ~rs sho~n by th~ 8aml~
mamber, eacc~pt in the mir:cor controlle~ ~che lettsr ~m~ is pl~Ged ~t~ ~h~
W~th r~r~nc~ to Fig0. 11 and 12, ~h~ ~lameJIts include latche~, mult~ pl~x~r~, driver~ and recei-rar~ . S~me o~ t:he la~ch~ uch z~ ch~ 933 and 9333~, act ~ del~y el~m~nt~
tc~ ensur~ th~ ~rop~r timl ng through th~ c~o~ and thereby ~in^~ain synchronlz~tion. A~ shown in ~ig. 11, con~rol code~ froD~ ~Aory eontrollar 70 ar~ ~en~c via bu~ 88 ~AW or~c~3 tolatch 931 asld them ~co latch 932 . Th0 re~son ~or such CINNEC~N, HENDERSON
F.~RA30W G~RRETr ST~T, ~ W. ~IL 5 WA~ .IG~O~. O C ~000--~æo~ e--o .'' ' ~ .

~22~
latchin~ is to prosride appropr~ate del~ys to en~ure that data from m~:ry controller 70 passez through cross~ k 90 3i~51111tanl30U8ly with data from m~mory controller 70~.
I~ coda~ from m~mo~ controll0r 70 are to be ~ent to processing ~y~tem 20 ~ via cro~s llnk 90 ~, then driver 937 is enabl~d. The control cod~s from memory con'croller 70 al80 pass through latch 933 ~nd into multiplexQr CSM~A 935. If control codo~ are r~c~ivesi into primary cro~s~ c 9 0 f rom croa~-link 90 ~, then their path i~ through rece~ver 936 into latch 938 and also into :multipl~er 935.
Control codHs to multlplexer 935 detarmine the sQurca of d~t~, that i~ either from m~ory controller 70 or from m~nory controller 70 ', and plao~a tho0~ codl3~ on 1ne ou~put of multiple~c~r 93~. That output Ls stor~d in latch 939, aS~ain A~or proper dQlay pUrpO8~, and driveE 940 i~ enabled if the code~ are ~o be s~nt to modul~ interconnec~ 130.
ThQ path for data and addres~ nal~, a~ shown in Fig.
12 i~ ~om~what ~imilar to th0 path of control signals ~ho~m ; ~ in Fig. 11. Th~ differences r~lect th~ fact th4t during any on~ ~r~ns~c~:ion, data and addre~es are flowing in only one dTr~c:tiorl ~hrough ~roa~-lLnlc~ 90 and 95, but con~rol ign~ 3 .~ can. ~ flow~lng in ~h dire~tion~ durlnç~ tha~ rans~ctlorl.
For t;hat B~IU31 rea~ofl the data llne~ in bu~es 88 and 89 ~r~
bid~r~ct~o~, but th~ con~rol eocle~ ar~ nok.
Data ~nd addr~e4 fro~ th~ ~Q~Aory ~ontroll~r 70, via bu~ 88, ent~r latch 961, then la~eh 962, and thQn latch 964.
~A~O~ C~S A8 in ~ig. 11, thelatch~s in F15~. 12 provide proper timing .NNECAN HE~DER~C-J
FARA~OW CARRETT
a C)u~;NER
`,. 177~ ~ 9tl~ W. -- 4i IIINOTON. i~ C ~C)C10--1~0~0-~0 ,~
' '~ ' , . ~ `
:'; ~ ' , ' ' ;' ' ' .j .'. .. ~ , '. . .

. ~, ' ~.. :

~ ~ 2 2 ~
to maintaLn synchroni2 tion. Data fro~ mem~ry controller 70 i~ bu~far~d by receivar 986, stor~d in latch 988, an~ then routed to ~he input of multiplexex MW~ 966. ~he output of mult~plexer 966 i~ stored in latch 96~ ~nd, if dxiver 969 i8 enabled, i~ ser~t to module intorconnect 13û.
The path for control codes to b~ ~en~ to m~3mory con~ol-ler 70 i~ ~hown in Fig. 11. Codes from modul~ interconn~ct 13 O are f ir~t ~tored in latch 9 41 and then pr~ented to multiplexer CS~XC 9~2 . I~ult~ plexer ~42 al~o receivea corltrol coda~ fro~ pzlr~llel cxo~ T~ r3g~te~ 91O and select~ eLth~r the parallsl regi~tor coda~ or th~ code~ frem latch 941 for ~rans~ssion to l~tch 343. If thos~a coIl~rol codas are l:o bs tr~nsmlttsd to cro~ lLnk 9 0 ', then driver 946 i~ en~bl~d. Control code~ frola cro~-llnk 90' (~nd thu~
from memory controllQr 70r ) arQ buf~ered by receiver 947, stor~d iIa latch 948, and prezent0d a~ an input to multiplexer CSMUXD 945. CSl~UXD 945 ~l~o r~ceive~ a~ ~n input t~ output of latch 944 whicll ~tore~ the con~nts o~ la~ch 943.
~ qulti~l~x~r 945 ~el~cts eith~r th~ code~ from mod~l~
in~orconDect 130 or $ra~ cro~-llnk 90 ' ~nd pres~lt~ tho~e ~iqnal~ as ~n inptlt to ~ultiplç~cer CS~E 949. l~ultiplea~er 949 al~o r~C~t~ s a~ input~ ~ cod~ fro~ th~ deco~e logic 970 (~or bulk ~ tranel~ers that occur during re~yncllro~lya~lon)~ codes ~ro~ ~he s~3rial cro~-lLnX
resli~ter~ 920, or al pr~de~enn~n~d srror code ER~.
Multlplexer SIT~49 than 8el!eC'~ one~ of tho~ inputE~7, undor the appropria~e control, for storage i~ latch 950 ~ I ~ho~o N~`IECA~I. HENDEliSON
F.~RA~OW, C~RRE~r 9 DUN~ER
l77~rl~tT,t~v~ ~ 7 .v~ OTOtl. O C ~OOO~
120~r~0~0 . .

.

: ' ' ' . :
'; ~ , ' ' ' " ' ~, , ' ,, ~ ~ ' ' ' ' ~ ' , : ' ~.

~3~2~ 1 i cod0~ are to ~e ?ient tcl m~nory controller 70, therl d~lver 951 i~ act~vated.
The puxpo e of th~ error cod~ E~, which i8 an input into mul~iple~6er 949, i9 t;o erl~urel th~t an error in one o~
the rails will no~ caus~ th~ CPUs in th~ 3~me zone as the rail to proce~s dif~erent Lnform~tlon~ I~ thi~ oc:curred, I C~U module 30 would detect a fault which would cause dra~tic, j and perhaps unnecQ~sary action. To aYoid thi4, cross-llnk 90 contain~ an EXCLVSIYE OR gate 960 whicA compare~ the outpu~Y
of multipl~xer~ 945 and 945m. If th~y differ, therl g~te 960 caulYQ~ multiple~er 949 to ~ele~t the 13RR eode. EX~LU5I~ OR
gatOE 960m similarly cau~e~ multiplQx~r 949~ al o to 8~15s:t ~n ERR code. Thi~ cods indicat~s to m~nory controllers 70 and 75 that there ha~ b~en a~ errox, but avo~d~ cau~ing a CPU
modu~0 error. The ~ingle rall interface to ~smory module 60 accompl$shes ~h~ 8~e reRult ~or da~a and addr~se~.
The da a and addrQss flow ~hown in Pig. 12 i~ ~imilar to ¦ thQ flo~ of control si~nal~ in ~lg. 11. Da~ and addres~e~

I fro~ modul~ intsrconn~ct 130 ar~ stor~d in lat~h 972 ~nd ~hen pro~ided ~ ~n in~ut to m~l~lple~ B 974. Da~a fro~ the parall~l reg$8~r8 910 provide a~oth~r ~nput to ~ultiplexQr 974. Th~ o~tpu~ o m~lt~ple~er 974 ~ n input to mul tipl~x~r ~U~C 976 wh~ch ~l~o rec~i~e~ dat~ and addressQ~
Y~ored in laSch 961 th~t w~r~ originally sent ~rom me~ory controller 70. ~ult~plexer g76 th~n ~91~t~ one of the inpu~s for ~.tor$~ latGh 978c I th~ d~t~A ~rAd addre~7e~

~A~ O~IC~ either from ~h~ module in~erconnec~ 130 or ro~ the .~mory lECA~J . HE~IDERSO~I
F.~ aOW, G~RRETr 6 DU~:~:ER
177~ r~ . N ~. -- 48 .VA~ING70N. ~1. C ~0001-' ~ 2 0 ~ 0 ~ ' controller 70, aro to be 8ent to cro8~-lin)c 90', then driver 984 i~ en~b~ed.
! Data from cro~-link 90' i~ buff2r~3d b~ receiv~r 986 and sto~od in la~ch 988, which al80 provide~ an inpu~ to multi~
plexer ~UXD 982. ThQ other input of multiplexer MUXD 982 i~
~he output of l~tch 980 which contains data and addr~ssQs from latch 978. Multiplexer 982 th4n 8eloc~t8 one of it~
input~ which i~ then ~tored into l~tch 990. ~f th~ d~ta or addr~sse3 ar~ to be ~ent to memory controll~r 70, then drivex 992 is activated. D~ta from ~rial regist~r~ 920 are ~en~ to m~mory controllor 70 via driv~r 994.
The data routin~ in cro~-llnk 90, and ~or~ part~cularly the 3conreol elelaen~ in both Fig~. 11 and 12, 1~ con~crollad by ~ ral ~ignal~ gen~3ra~ed by dacod~ logic 970, d~cod~ I
logic 971, decod~3 logic 996, ~nd deeod~ logic 998. ~rhis lo~ic provide~ the signal~ which control multiplexers 935, 942, 945, 949, 966, 974, 976, and 9~2 to ~elec~ the approprLat~ input 80ur~. In addit~on, the decods logi~ al~o con~rol~ dr~rer~ 940, 946, 951, 969, 984/ 992, and 994.
~ o~t of the control ~ignal8 are gen~rated by de~ode logic 998, ~ut ~o~e are ge~asra~Ged ~ decode loglc 970, 971, 97~, 971~, 2md 996. De~ode log$~ 998, 970 and 97ûm are ~osu~ t~d at po~i~ion~ tha~ will ~n~ure ~ha~ th~ logic wlll r~co~Y~ th~ data and cod~ ne~c~s~ary for coQtre~l wheth~Pr ~he data and God~ ~r~ recaived from lt~ o~n zo~e or from other ¦ zone .
o~-ce-FINNEG~N. HeNDER50N ¦
F.RRAEOW G~RRETT
~ DliNNER
177~ ~ sr~lrc~ ~ w ~ 49 ~
'.i W~91~ J701'~. 0. C 7000-I~O~ZI~ O

. ~, ' :' , .

'I I
~22~
Th~ purpose of d~code logic 971, 9~1~ and 996 is to en~ura th~t the dri~rer~s 937, 937$1 and 984 are 8et into the I proper ~t~te. ~h$~ llearly decode~ makes ~ure that da~a . addrRssa~ and code~ will b~ forwarded to the proper cros~-link~ in all ca~es. Without ~uch e~rly decod~ logic, the ¦ cros3-links could ~ll bo in a state w~th heir driver~
¦¦ disabled. If one at ~he m~ory contro~lers were al80 ¦ ¦ di~abled, then its cro~a-links would never r~c~ive addr2sses, data and control cod~, ef~ectively di~abling ~ll the I/0 module~ connected to th~t cro~s lin)c.
~ rior to de~cribing the driv~r control ~igr~als g~nerat~d by decode logic 970, 971, 970~, 97lm, a~d 998, it i~
- I n~c~ssary to und~rst~nd th~ di~ferent Da~t the~e ZS:~110~
and th~r~afor~ the cro~-link~ 90 and 95, c~n be ln. Flg. 13 contain~ ~ diagram o~ ~he d~ff6~r~nt ~ta~e~ A-~, and ~ t ble ! explaining th~ stato~ which corr~spond ~co ~ach laod~.
¦ At ~tar~up and in o~h~r in~tances, bo~h zon~s ara in ~tate ~ which i~ known a3 ~ho O~F mod~ for b3~h zones. In th~t mode, thQ computor 8y2~t~1EI in both ~one~ ~re op~r~lting indep0ndan~?y. ~ftar OllQ of th~ zoneEI ~ operatillg 8y8t03112 r~gueats the ability to communic~e Wi~}l the IJo of the o~her ~ono~ and th~ r~ue~t i~ honored~ thQn ~hei z0nlE~8 entar the m~torJ~lave mod~, ~hown a~ stat~ B and C. In auch moàs~, th~ 20rle whlch i~ ~he m4~ter, ha~ an operA~ing CP~ and haa control of ~h~ I~0 ~odule~ of lts zone and o~ the other zon~.
Upon ini~lation of r~ynchronization, ~h~ comput0r L,~wOr,,c., ~yste~ leave~ the ~ tor/~lav~ moda~, eith~r ~tateP ~ or C, INNEC~N HENDER50N
F.'~RABOW. ~ARRETT
~ DUNNER
177~ rjT. ~ W. ' 5 0 o-or~ c 200C~--~20~D3~0 ' ' - ?
-" ,~

2and antQr~ a re~ync slave/re~yY;c ~ster mode, which ia shown a~ state~ E and F. Ir those mode~, the zone th&t wa~ the ma~tar zosle i~ charqe of bringir~ the CPV of the otl~.er ~ zone on line. If the res~cha:on$æ~tior~ f~ , the zc~n0~
I revert to the ~ame m~terfs1ave mode that they wexe in prlor to the resynchronization . tk~pt.
l If th~.resynchronizal;ion i~ uooe~fu1, how~vor, then I the zone~ entsr ~tate D, wh~ch 1~ the full duplQx mode. In thi~ mod~, both zone~ are op~r~t~ng toqe~her in 10ck~ep ~ chroniæ~ion. Op~r2tion continue~ in this mode until there i~ a CPU/MJ3M fault, in wh~ ch c~ the sy8t~3m enter~ one of the ~ ter/~l~Y~ modes. TA~ i8 ~h~ zon~ who~ ~
proee~eor e;!~pe~ienced the CPU/~ ~ult.
Whe~a op~r~ting in s~ate D, t~l~ ful dupl~x mode, cer~Ain ~rrors, mo~t no~ably clock pha~e errors, neces~ita~e ~plitting the sy3t~m into two ind~nd~n~ procQg~ing systems.
Thi~ c~uses sy~tem 10 to go back into s~ate A.
Decod~ logic 970, 970m, 971, 971m, and 998 ~coll~ctively rf~f ~rx~d to a8 th6~ croa0~ k co~trol logic), wh~ ch ar2 ~hown in ~q~. 11 and 12~ have ac~e~ to ~he re~ c ~od~ bit~ 915 and the cro~-link mod~ bit~ 9160 wh1ch are shown in Flg. 10, in o~d~r to d~t~r~Qine how to 3~t the cro~-link drivera and mul~ipl~ers lnto th~ proper ~t~t~ n addition, tha oross-l~nk deGode logic ~l~o reCeiVQs and analyze~ a portlon of an addre~s sent fro~ SIU3D10~7 sontrollers 70 ~nd 75 during d~a transac~1ons to 0attr~ct addreil3inS~j7 ~nformation th~t fur~her ~w 0~
INt`lEC~I. HE~DER5a~
F~R~\BQW~ CARRETT
~ DUNNER
IT7~ 1~ 5Tr257. r ~ 5 1 r aT01l. D, C ~000 ~02) ill~ 2~0 1: .

, " ' ~ .
,.

2 2 2 ~
indicate8 to tho cro~8-link de~cod~ lo~ic how to ~et the ~tate of tha crc:38-link multiple~er8 and driver~.
T~e inf ormation need~d to sel: the state~ of thQ
: multiplexer~ i~ fairly straightforward onco 'ch~ differQnt I modes and ~ran~actions are wlderstood. Th6~ only ¦ det rmination to ba mad~ i th~ 130UXCQ of the da~a. ~hUY
when croas-links 90 ~nd ~5 are~ in the 81~VQ mod~, multipleJcer~ 935, 93S~, and 966 w~ e1QCt data addrQYRe~
and code~ from zoIle ll'. Tho~e IGult~pl~xl3rs will also ~elect d~ta, addrosses ~nd ced~ fro~ tA~ other zona if croa~-links I gO ancl 95 are in full duple~ mode, the add~Qss of an I ~0 ¦ ins~crue~lon i8 for a ds~ric~ connec:ted to an I/0 modul6~ Ln ¦ zon~ 11, and thQ cro~a-link with the affect~d multiple~xer i9 ¦ in a cross-over mode. In a cros~ov~r mode~, the data to be ., Qa~ on the modulQ intQrconn~ct l~ to be rec0ivQd from th~
other zone f or ch~cklnq . In ~A~ pre~rred embosliment, module I interoonnac:t 130 would r~c:eive da~a, 2ddre~q~ and codea from ¦ the primary rail Ln ~one 11 and m~dul~ intQrCOnn~act would race~$v~ dat~, addr~æ~ ~nd code~ fro~ ~h61 mlrror rail in .. ~one 11 ' . Al~6~rn~tlv~1yt ~odlul~ ero~nect 132 could .~ r~ce~ive da~c~, addre~e~ and code~ from the primary r~il in zors~ 11' which woulcl allow ~h~3 pri~ry r~Ll of on~ zcsne to b~
co~ræd wlth th~ mirror rall of tha o~he~r zon~.
~ul~$ple~c~r~l 945, 94SDI, and 982 wlll bsa 8eal: to acc~p'c '.i, data, addr~o~ and CCld~38 from whiche~r~r zon~ i~ th~ source of the da~a. Thil3 1B 'crua both when all ~h~3 cro~ ara in wc,r,~c~. full dupl3~ moda ~nd the da~ic, addre~ ~nd cod~s ar~ r0ceived -INNECAN. HENDER50N
~; F.~RABOW G~RRETr a l )U)`;~ER
,' 17~ ~ DrW~T, ~ W 5 2 ~
OTOI~. O C . ~ooon ~Z0~ 02~0 ~ ' ' ' ' ~
,: ' ~ ' ` '' 2 ~
from I/O mo~ule8 and when the cYoss-l~nk is in a re~yne ~la~
mode and th~ data, addr~ and code~ are racai-rQd fro~ the m0mory con~roller3 of tha other zone.
If the ~ddra~sing inforlr~tion from m~ory controllQr~ 70 and 7 5 indic~to~ that the ~ource of re~ponsa data and code~
i the cros~-li~'s owrl parallel regi~ter~ 910, then multiplexor~ 942, 942m, and 974 are ~et to ~elect data and codes from those ragi~ter~. Similarly~ if the addre~sing informa~ion from memory ~ontroll~r~ 70 and 75 indica~Q~ ~chat ~h~ ~ourc~ of re~pon~s dat i~ th3 cro6~-linlc~ own ~er~al regi~ter 920, thQn multiploxer~ g49 and ~9~ are ~a~ to solect data and code~ froDI ~ho ~ r~gis~Qr~
Multiplo~cer~ 949 and 949m ar0 al~s ~et to ~el~ct data from d~code logic 97a ~nd 970m, r~-pec~iv~ly, if t~le -infonnation i9 ~I control code during m~ory r9~ync operation~, and to ~slect tho ~RR code if the ~C~VSIVE: OR
gat~ 960 and 960m id~ntlfy . mi3compar~ b~tween l:he dat~
tr~n~tted vi~ cro~-linkR 90 and 95. In thi~ latter C~Be, th~ colltrol of th~ multipl~xer~ 9~9 and 949m i~ generat~l from th~ 2~CI.U5IV~ OR ga~ 960 and 960~ rath~r ~han from the crc~-link co~atæsl logi c. ~ultiplex~a 949 and 949m also ~ C COdellJ from s~rial cros -l~nk ~e~i~t~r~ 91a whell thoso r~l~r~ are r~e~t~d or the ou~put of mul~iple~ora 945 and 9~5DI whol~ tho~e cod~ are ~ quoat~. ~ iplexer-~ 945 ~md 945m ~ellsct eith~r ~he oUtpUtB fro~ ~ul~iplexsr~ 942 and 942~ re~p~ot$v~1y, or ~O cod~ fr~D~ cro~s~link~ 90' and 9 5 ~, re~ ctively .
F)NNeC~N. HeNDERSC)N
F~R~DOW. C~RRETr F~i DUN~ER
,", ~ 9rl~t~T ~ ~ - 5 3 ~A~ 140~0id 0 C /000111 ~02)~ 10 ~.

..

2~2~ `
lNultiplexer 976 ~3elect9 either dat~ and addres~e~ from ¦ module interconnect 13 ~ in the ca~e of a tran8action with an I~0 module, or data and addre3ses from m~mory controller 9û
when the data and addre8~es are to be sent to cro~s-link 90 either for I/0 or durlng memory resynchronlzation.
Driver~ 937 and 937~n are actLvAt~d wh~n cros~-link3 90 ¦ and 95 are in du~l~x, ma~ter or re~ync ma~tar mod6s. Driver~

940 and 940m ar~ activated for I/0 transactions in zon~ 11.
I Driver~ 9~6 and 946m ar~ actlvat~i wh~n cro~s-llnk~ 90 ~nd 95 ¦ ar~ in th~ duplex or ~lave m~des . Dr~ ver~ 951 and 951DD arQ
alway~ ~ctivated.
lDrLYer 969 is actlvated during I/0 Wl'ite8 to zon0 11.
- ¦Driver S84 ia actlv~t~d when cros~-link 90 i~ ~ending df~t~
¦ and addroa~ea to I/0 in ænn~ 11', or when cro~s-link 90 i~ in th~ r~ync mgl~t~r mode. Racalv~r 986 r~c01ve~ da'ca from cxoss-link 30 ' . Driver~ 992 and 99~, are activ~ted wh~n data i~ being 8ent to memory co2ltroller 70; drlve~ 994 i~
ac..ivat~d whea the content3 o~ the s~xl~l c:~o~-link regi~ker 910 ~rs road and driYar 992 i3 ~ctiv~ted d~Ang ~11 other readæ .

5. ~L~Q~
When both proc~ssing ~y~ , 20 snd 20 ~ ~r~ each ;~ Ip~rorming the ~ function~, in th~ 4ull duple~ mod~, it i~7 ~p~3rativ~ that CPU module~ 30 and 30~ p0rfo~ opesation~ at th~ r~t~. Otherwi~,e, ~8ive ~mouI~t~ o~ proc~a~sing tL~

will be con~ seaynchronizin0 ~roc~in~ 8y8t~m8 20 and AW Or~lC~!i 20 ' ~ox IJ0 and inte~proce~or ~axror cha~kin~7~ In the ~;eG'~.He~EiSO~ ¦
,:~ F.~R~a~w CARP.ETr a DiJNNER
,,7, ~ ~T~T, .~ W. 54 ~rll~3TO~ . a c ~oooo ~0~ 0~bO
.

' , , : ' ' ~ ',,, :
', ' " :' '' '~:
';
' ' , ' . .`: , , ' ' , , ', : .

~ 2 preferred embodiment o~ proceasing 3y8tem~ 20 and 20', their b~ic clock signal~ are ~ynchroniz~d and pha3~1Ocked to each other. The fault tolerant Com?llting system lD includs~ a I timing sy~em to control the ~:requency of the clock ~ignal~ ;
I to proce3sing ~ystem~ 20 and 20 ' and to minimize th~ ph~se I di~er~nce bstw~n the clock sign~l~ for each proca~sing 1 3y~te~
¦ Fig. 14 3hows a block diagra~ of tha timing ystem of I thi~ invantion ~bedded in proc0~sing 8y8t3ml~ 20 and 20'.
¦ The timing sy~te~ compri~es o;~cillatGr sy~tQm 200 in CPU
module 30 of proce~sing sy3~ 20, and o~cill~tox sy~te~ 200' in CPU module 30' o~ prv~eglsinS~ ~y~t~a 20~. ~h- ale~nt~ of oscillator 200 ' are equi,vals3s~t to thos~ - o~cill~to~ 200 and both o~ tor ~Iy8te~A8 ' opo~at~on i~ ths ~ame . ~hu8, only the elsm~nt~ ~nd operatloa of o~ill&tor 3y~ti~m 200 will be descrlbed, ~xc~p~ if the oE~r~tion~ of o cillator ~sy8tem8 ; I 200 and 200' d~ffer.
Fig. 14 show~, lauch of o~clllator ~y~te~n 200, ~pecific lly th~ digital logic, lie~ ln~ide of cro~-llnk 95, but that plac~3nt i~ not r~uir~d for th~ pre~en~ in~en'cion.
Osrillator ~y~t~ 200 is~eludes a voltag~-controlled cry~tal o~ tor (~CIE0 ) 205 which g~n~ratç~ a ba~ic: o~cillator ~ l prefer~bly at 66.66 Mhz~ The fre$uQrlcy of VCX0 205 can bQ ~d~u~t~ by th0 voltag~ leval at th~ lnput~
Cloçk die7~cributisn chip 210 di~ricl~ do~ ~he ~l~GiC
¦ o~cillator ~ nd pre~r~bly produces four pri~ry clocXs L~O,r,c............ all hav~ng the sa~ fr~jue~ncy. ~or primary CPU 40 th~ clock~
~I~EC,W, HENDR50N
r iRAaOW, C,/CRRETr Dl;~j:`lER
. ,,7. ,( srRC.. ,. w. -- 55 ~U~ ITON. O C ~000 120~ 0 ,.~ .', . .
';
.
~' . .
.

l 2~22~ 1 are PCLR L and PCL~ H, which ar~ logic~l inver3~s of each other. For ~rror C~U 50, clock d~eribution ch~p 210 prod~ce~ clock ~ al~ ~qCI~ I. and MCI~ H, which are also logical inver~e~ of oach other. ThQ timing and pha~e ll relationship of ~hese clo~k signal~ are ~hown in Fig. 15.
¦ Preferably, frequ~ncy of clock Rignals PCLR L, PCLR H, MCL~
L, and ~5C~ ~ i8 about 33 . 33 Mhz . Clock chip ~10 al~o produco~ a pha~e-lock~d loop slgn~l CLRC H at 16 . 66 M~z, al90 shown in Fig. 15. This pha e lock~d loop Mignal i s~nt to clock ls~gic 220 which buffer~ th~t sign 1.
Clock logic buff~r 220 ~end~ the C~C H ~ignal to 08cil la~cor 200' for u~e in ~ynchrQniz~tio~. Clock logic buf~e~ j 220 ' in o~clllator 200 ' ~end~ ~ts o~n ~u~fer~d phasæ-locked loop signal CL~C:' H to pha~ll3 detector 230 in oscillatsr 200. ¦ -Ph;~e dsee~tor 230 also rece~v~3~ th~ bu~ered phaf~e locked loop ~ignal C~RC ~1 from cloclc logic 220 through del~y element 225. D~alay el~ent 225 appro~c~at~ th~ delay due ~o ~he cable run fro~ cl~ck logic buffer 220'.
Ph~s~ d~atectsr 230 co~par~ it~ input pha3e lock~d loop ~ i gnals and generate~ ~wo outputs . One i~ a ph~e dlff~ren~ ignal 235 which is ~erlt through loop a~plifiQr 240 to the volta~e input of VC:I:O 205. Pha~e dlfferances sigml~ 235 ~ll cau~ ~mpl~ fl~r 240 to g~3neral:e~ a 8ig~1al to al~c~r ~cho ~r&~ency ~f ~ o 205 to cogll~nsa~e for pha~e dif ~rance~ .
Thç~ other output o~ pha~6l det~ctor 230 iB a phaae Qrror LAW O~IC~ signal 236 whi~h lndicate~ po~ibl0 ~ynchroni3m fault8 ~N~ . H~DER50N
F.~RA30W G~RR3Tr 6 DU~R
STl~t~T, ~ vr -- 5 6 ~TO~ 000--,, 0 ~ . 0 ~ -.

. :.

~22~
! Fig. 16 is a d~t~ d diagram of phaae detec~or 230.
¦ Pha~s detecto~ 230 include~ a pha~ comparator 232 and a oltl~ge comp~rator 234 . Pha~e comparator 232 receives the cloGlc ~ l from delay elament 225 ~CLP;C H) and the pha~e lock loop clock signal from o~cillator 20û' (CLXC' H) and geslerates pha~e di~erences signal 235 a~ a ~rel'c~ge Level I rspre~enting ~she pha~e di~fere~ce of tho~ ~igna1s.
I I p~oces ing ~y~tem 20 ware the " s1a~e~' ~or pW:pOf~el8 of .. clock ~ynchronization, ~witch 245 would be in the "SI~VE~
position ( i .e ., clo~ed) and the voltag~ 1ev~1 235, after be-ing amplif ied by loop a~plif $er 240, wou1d control the f re-q~uen~y of VCX0 205 . If both ~witche~ 245 and 245 ' arR in the "2na~ter" posilt~on, proca~sin~ ~3y~ alll8 20 ~nd 20' w~uld no~ be pha~e-1ec3ced ~r~d weu1d be runnlng a~hronou~ly ( indep~n-dont1y ) .
The vss1tags 1~vel of pha~ differenoe~ s15~na1 235 i3 al~o an input to voltage co~parator 234 a~ ar~ two re~erence vo1tagss, Vr~f1 and Yref2, repreRe~nt1ng acceptab1e range3 of . ¦ phass lead and lag. If th~ ph~ differ0ncs iB within to1er-.~ ance, the P~ ~RROR signa1 ~ill not be dct1v~ted. If the phAs~s dLf~0r~nee iel out of toler4rlce, then the P~S~ EMOR
siga~,a1 236 will be act~v~t~d and ~e~lt to oross-~ 1nlc 9S via c10cX d~cad~r 220.
': 1 Fig. 17 ~hows a prafsrred ~bodimellt of an IJO al~odule l 100. The prin lpl~ oi~ opç~ratlon I/O module 100 ~re .~wor~lc~ I applicabl~ to the othar I/O modu1e~ a~
~NECAN HENDERSON
FARA~OW GARRErr 6 D~NNER
177 ~ T . N W. -- 5 7 ING~ON. O C 2000--~2021~D~ O

.' ''.
., .

, ` : ~"' ; :' ;

. " ' ' ' :
' ', I 7'~2~
FLg. 18 ~hows the elementc~ in the preerred embodimant of flrew~ll 1000. Firewall 1000 includes a 16 bit blts interface 1810 to modula int~r~onnect 130 and a 32 bit bus interface 1820 for connection to bus 1020 ~hown in ~ig. 17.
Interface~ 1810 and 182~ are connected by an internal flrewall bu3 1815 which also interconnect~ with the other elemsnts of ~ir~wall 1000. Pref2rably bus 1815 i~ a p~rallel bu~ eithar 16 or 3 2 bit~ wide .
I/O module 100 i~ connected 40 CPU x~odula 30 by means of dual r~il modul~ in~erconnects 130 and 132. Each of ~he module interconrlects i~ receiv~d by fir~lls 1000 ~nd 1010, r~spectively. One of th~s ~irew~ , ~ieh i~ u~ually, but not a}way~ ~lrewall 1000, write~ eh~ data from m3dul~
int~rconne~t 130 onto ba~ 1020. ~h~ other firew~ll, in this casa ~irew~ll 1010, check~ th~l: d~ ag~ns~ it~ own sepy recsi~ed from modul0 intercosm~t:t 132 uRiQ~ fir~w~ll comparison circult 1840 ~hs~wn in Flg. 18. That checkl~ i8 eff~c~ive dua ko the lock~tep synchron~ation of CPU modules 30 and 30' which CAU8el!1 data ~ittan ~o IJo ~odule 100 from CPU ms3dul~ 30 and 3û' to be ~v~ ble at ~irewalla 1000 and 1010 ~ubsta3ltially s~ultaneou~ly.
~ iraw~ll co~p~ri~on circuit 1840 only chec36~ dAta r~c~lvad ~ro:~ CP~J moAule~ 30 and 30'. Dat~ t to CP~3 module~ 30 ~nd 30 ' fro~ an I/O davic~ h~vo a co~on orlgin and thu~ do not roquire che ::king . In~ad, d~ta r~ceivQd ~rom An I/O devlcs to bs~ eI~ to CP~ modul~a~ 30 ~d 30~ is che~ked by an er~or det~3ct~0n cod~ (~DC), ~uch a~ a cyclical CI~ IEGU~I. HE~IDER50N
F.~A~30W GARRETr a DU~ ER
,"" ~ sT".. ,. w. ~
WAullllCTON. O. C ~0001~
~o~ r,-~o . . ~
- ; .:

'' 2i~22l,~, redundancy check (CRC), whlch i~ porformad by EDC/CRC
generator 1850. ~DC/C~C ~enar~tor 1850 L~ al~ coupled to intern~l firawall bu 1815.
. ! ~DC/C~C gen~rator 1850 generat~ and check~ the ~ama : I EDC/CRC code th~t i3 u~ed by ths I/O d~ice. Preerably, I/O
. modula 100 gen~rate~ two ~DC. One, which can al00 b~ a I ~ EDC/C~C, i~ used for an interf~ce to a network, ~uch a~ the : I Ethernet packet network to wh~ch module 100 i3 coupled (~ee I slam~nt 1082 ln Fig. 17). Tha other i8 u8ed for a disk ¦ interface ~uch as d~k 1nter~ace 1072 i~ Fig. 17.
I EDcJc~c coveragQ i~ not requ~x~d betwee~ CPU module 30 I I and ~/o module 100 b~c~u~e the modul~ int~rco~nectæ ar~ ¦
. I dupli~ ted. For exa~ple in CP~ ~odule 30, cre88-link 90 co~munlcat~ ~ith flrewall 1000 throu~h ~odule in~erconnece 130, and cro~-link 95 communicat~ with firewall lOlO
through modul~ interc:onnect 132.
~ A m~3s~ag~3 res:eiv~l from E:q h~ t nstwork 1082 i~ checked ¦ for a ltalid ~DC/CRC by nat~ork control 1080 3ho~ in P.ig. 17.
~he data, co3~ ta with ~DCJcR~ written to a local RAa~
1060 al~o ~hown in Flg. 17. All d~ta in 1OCA1 RAkL 1060 i3 transf~3rred ~co ~ory module 6û u31ng Dila. A ~N~ control 1890 coordl~at~ the traslsf~r and dir0~t~ /CRC genera~or 1850 to ch~ek th~ ~ralidity o~ th0 ~DCJCRC ~ncodsd d~a beinq ~ran0ferr~
:~ Mo ~ d~t~ ~r~nafer~ with ~n I/0 d~v~G6s a~ don~ wlth DMA. Oata i~ mo~7ed b~tw~en main ~a~mory and IJo buffer L~wOrrlc~ m~mory, When d~ta 1~ mov~d rrom ~che m~ain me~ory to an ~/0 INNECAN, HENDER50N
F.~R.~30W ~ARRE7r li Dl,'!`:!`IER
177~ ~ 9rl~r,~ W. I -- 59 GTO~. O C ~oooa ~zo~93 a~o . I
.~ l ' ~:

I ~ ~ 2 2 ~
I buffsr memory, an EDC/CRC may be appende~, Hhell th~ d~ta ia mov~d from I/O buffer memory to m~in ~emo~y, an ~DC:~CRC may be chacked and moved to main m~ory or may be #tripp~d. When data is mo~ed froD:I the I/O bu~er memory throlagX an ex~ern~l . I device, uch a~ a di~k or Bthern~t adaptor ths EDC/C~C may be , .
checked locally or at a distant r~ceiving node, or both. The memory dat~ pack~t~ may have their EDC/CRC generated at th~
d~ 3tant node or by the local interXace on th~ I/o module.
This opsxation en~ure~ that data re~ding in ox being tran~ferred through a 3insle rail ~y~te3~ li3c9 I/O module 100 is covered by an error det~ction code, whlch i8 pre~exably at least a~ reliable a~ the ~o~ualcat~ ona ~edi~ the da~ca will eventually pa~s through. Diff0~erlt I/O ~, for exa~ple tho~ wh~ch handl~ ~ynchxonous pro1:o~ols, preferably h~ an EDC/CRC generator which g~nf~r~t~s ~nd che~ks th~ BDC:/CRC
codas of the appropriate prot~col~.
In general, D~ control 1890 ha~dle3 the portion of a DMA o1?eration ~pecific to the ~harecl memory controllsr lOS0 and local RAN lû60 being addr~s~3sd. ~he 32 bit bu~ 1020 i~
driven in two dlf ~erçmt m~des . During DP~A setup, DMA control lB90 u~ bu~ 1020 A~ a ~andard asyslchronou~ mlcroproce880r bu~. ~h~ addre~N ln local RAM lOfiO wher~ the~ DMA operation wt~l oc~ uppli~d by ~har~d memory controll~r 1050 and Dl~ control 18gO. During ~ch~a ackual ~l~A tr~n~felr, D~
control 1830 direct~ Dl~ control lin0~ l~9S to drilra bus 1020 in a ~ynchxonous fa~hion. Sharacl ~ory ~ontroller 1050 will tran8fer a 32 bit data word wlth bu~ 1020 ~very bus cycl~, FINNEC~N. HE!iDER50N
F.~R~EOW G~RRE~r ~I DUNNER
177~ T~rT. N W. ¦ -- 60 w~ To~. O ~ .
~70~ 30 :, ' and DM~ control 1890 ~Eeeps track o~ how m~ny word~ are left to be transf~rr~d- Shared mQm~ry control 1050 al50 controls local RAM lOS0 and cr~ate~ the naxt DMA addre88.

I The I/0 modules tlO0, 110, 120~ are respon~ibl~ for con~
; trolling the read/wTita op~ration~ t9 their own local RAM
1060. The CPU module 30 i~ re~pons~blQ for controlliny th~
transfer operations with m~mory arr~y 60. The DNA engine 800 o~ memory controller~ 70 a~d 75 t~hown in Fig. 8) direct the ~: I DMA operation~ on the CPU module 30. Thl~ divi-~ion of labor pre~nt8 a fault i~ th~ DNA logic on any module from degrading th~ data integrity on any other module in zones 11 o~
Th~ fu~ction3 of tr~c~ 1872 and tsac~ R~ controller 1870 ar~ descri~d in greater deta~l b~low. Brie~ly, when a f~ult i~ detect~d and tha CPU~ 40~ 40', 50 and 50' and CPU
module~ 30 and 30' a~e no~ifi~d, v~rioua txace RAM~
throughout co~putar ~y~tem lO ar~ caus~d to perfor~ cer~ain functions described b~low. The communic~tion~ with the trace RAM~ takas placo over ~rac~a bu~ 1095. ~rac~ RA~ control 1870, in r~pon~ tO E~igllal8 from trace bu~ 1095, c~u~
tr~c~ ~ 18~2 eithor to ~top storiIIg~ or to du~Ap its cont~nts OV0E tr .ce bu~ 1095 .
I/0 ~ le bus 1020, which i~ preforAbly a 32 bit parallel btla~, co~ples to fir~wall lOOQ and 1010 as w~ll a~
to other ele~ nt~ of thc I/0 modul~ 100. A ~har~d m~ory con~roll6~r 1050 i alao coupl~d to I~0 bu~7 1020 i~ I/0 module wOr~lc ~ 100. Shar~d m~morS7 controller 1050 i~ coupl~ to a local FINNECAN. HENDERSON
F~RABOW C.~P.RETr 6 DUNNE~
,77, ,. ,.~.. ,.. ~ -- 6 1 ~
oTo~. o. C ~oOO--~2021 1~ 0 , ' .

. .
memary 1060 by a shar~3d m~mory bus 1065, which preferably carrie~ 32 bit data. Preferably, local m~ory la60 is a RA~S
wlth 256 Rbyte~ of memory, but th~ e of RAM 1060 iB
I di~cretion~ry. The ~h~red memory controller 1050 and loc~l i~ 1060 provide memory capability for I~:) modul3 lOOo ~I Disk controller 1070 prs)vide3 a standard interface to a ,¦ dlsk, such as d~sks 1075 and 1075' in PigO l. Di~k I controller 1070 i~ al80 coupled to 3h red m~mory controller 1050 either for uæo of local R~ 1060 or for com~unication 1 with I/0 module bu~ 1020.
.¦ ¦ A networlc controllor 1080 provids~ an inl:~rface to a ¦ standard network, such a thQ Eq~HER~I! n~tw~rk, by way o~
ne~work interface 1082. Network controllsar 1080 i~ also coupled to sharQd mQmory controller lOSû which acts a~ an int~r~S~c~ hoth to local RlUI 1060 and I/O nodule bu~ lû20.
Th~re i~ no requir~Aent, how~ver, for any one 9p~1CifiC
organization or ~tru~'cur~ of I/O modul0 bu~ 1020~
PC~lq (pow~r and coolin~ int~rface modul~) iupport ele3ll~n~ 1030 i~ eo~ecte~l to I~O module bu~ 1020 and to an ASCII inter~c4 1032. PCIM ~upport element lO30 allo~
.; pre: caaBin~ sy~t~ 20 ~-o ~aonitor ~hc~ ~ta'cu~ o~ the power syst~a I L.e., oa~t~rie~, ra~ulAtor~; e~c. ) and th~ cooling 3y~a~ (l.e., f~n~) to en~ur~ ~cheir prop~r operation.
Pr~3f~rably, PCI~ l~upport ~ nt lO30 only receive~ ~e3~age ¦ when ~h~re i8 ~o~e fault or poc~ntial faul~ indication, 3uch as an un.cc~ptAbly low b~tt~ oltag~. It i ~l~o po~slble . LAW o,rlC., to u~ P~ l 0uppor~ el~ment 1030 ~o monitor all tha pow2r ~nd .'! E(;~`J. HENDER50N
:`F.~R~OW G~RRETr ;: 6 D~ZER
"s ~ ecT ~ ~ I 6 2 0~ ~ c ~000- 1 ~2021~9~ G~0 .~ I ~, ~ l , "
', '~ . , , ' ' cooling ~ubsy~tema periodic~lly. Alternativ~ly PCI~ ~upport el~m~nt 1030 may be connectei directly to firewall S lO00 and 101~-Diagnostic~ microprocs~sor llO0 18 al~o conn~c~ed to th~
¦ I/O module ~u~ 1020. In generall diagno~tic~ microproc~or 1100 is used to gathar error checking information from trace S, such as trace ~N 1872, wh~n fault~ are d~cted. Tha~
I d~ta is gathered into tracQ bus~s 1095 and lQ96, through i firowall~ 1000 and 1010~ respecti~ely, through modula bus 1 1020, and into microproce~or 1100.
:~ ! D . I~ERl?ROCE~SSOR AND INT~ODUI~ O~IJNICATIO~
1. ~h~ ' The elemetnt9 of computer ~yl3taa lO do alot b~ th~sl~e~
con~ti~ut~ ~ ~ault toler~r~t ayst~. Ther~ ne~d~ to b~ a co~-municat~ or~ p~hway and p~otocc~l wh~ ch allowo co~munlcatlon during nonDal oper~tiona arld op~r~tion du~lng ~lt detection and corr~ction . ~ey to 3uch co7~u~ication i~ Gross-l~ nX
pathway 25. Cros~-link pathway 25 con~priP7e~ the parallel l link~ rial l~ , and clock ~ignal~ alre dy ds~crib~d.
I Tho~3 ar~ ~hown ln F~s7~ 19. Tho p~r~llel link lnclud~as ~c~o identi~al 8~1tl~ of data and addres~ llne~7, control l~nes, lnt~rnl~t lin~, codQd error linQ~, and a soft r~t r~qu~st lin~. q!he data and addre~ line~ and the control line~ con ta~n iRfor~tion to b~ ~xchanged b~t~n th~ CPU modul~7, ~uch a~ rom th~ modul~ int~r~:onnec~7 130 and 132 (or 130 and 132 ' ~ or from m~3aory module 60 ( 60 ' ) .
L~ o~r~ I
NEC~N. HENDEA50N
FARADO~. G~RRETr ~ DU~;ER
~ ,77~ a~c~ ~ w -- 6 3 ~YI~IG~O~I. G C ~000--1~0~1~0~01~4 ' , , '' ~

i! I

The interrupt lin~8 preferably contain on9 line for each of the interrupe, level~ availabl~ to I/O 3ub~ystem (madul~s 100, 110, 120, 100', 110' and 120' ) ~ Tha~ line~ are ~hared i by cros~-link~ 90, 95, 90 ' and 95 ' .
The cod~d error line~ preferably in~lude code~ for synchroni~ing a con~ale "~LT" reque~t for bo~h zonQ~, one for ~ynchrorlizing a CPU ~rror for both ~one~, one for indicating the occurr~n~e o~ a CPU/mQmory failure to the other zone, one for synchronizin~ ~ error for both zones, and one f or indLcating clock pha0~ orror . Th~ error iin~
f rom each zone 11 or 11 ' are lnput~ to an QR g~t~, ~uch 28 OR -gato 1990 for zone 11 or O~ g~ 1990' for zone 11'. The output at ~ach O~ g ~e proqid~ an lnput to th~ cro~
of ~cho other zone.
¦ Tho f~ult toler~nt proce~ing sy~lt;em 10 i~ dlE3R~ gned to continue op~ratlng a~ ~ dual ra~l ~yote~ de~pite tras~ nt faults. The I~C) aubsy~t~m (mo~ul~ 0~ 110, 120, 100~, 110', lZO') call also ~xpsrionce ~ran~i~n~ erross or fa~llt~
and ci~ntinu0 ~o op~r~ . In the pref e~rr2d ~bodim6~n~, an error det~c~ed by flr~wall comp~ri~on c~rcuit 1840 ~ill cause a qynohroni~ed ~rror r0port to b~s mAd~ through l?athw~y 25 for C~?~J dlrected oper~lon~. E~rdw~re in C:PU 30 and 30' wiLl ~au~ ynchror~qsod soft r~et through pathway 25 and will ret~y th~ fault~ ol?~ration. ~or DI~A dir~ct~i opera~ioTI~, ~ ~he 9~ error d~tection re~ul~s in ~ynchronous in~errupt.

through 1?at~way 25, ~nd sof~we~ 0 in CPUc 40, 50, 40 ' and 50 ~W O~IC~9 ¦ will re~tart the DMA opera~ion .
Fl"l~ieGA~`I . HE~DeRSO~J I
F.~RABO~ CARReTr li Dl,l`;~ER
1~7~ T~ T ~ W. 6 4 W~irlll 5TO~. O C ~000--~0~ 0 ~

2 ~
Cer~ain tran~ient error are not ~diately recoYarable to allow continued operation in a full-duple~ ynchronized fashion. For example, a corltrol error in m0~ry module 60 can re~ult in u3~known data in memory modulc 60. In this situation, the CPUs and mamory elements can no longer ~ function reliably a~ part of a fail afe 8y8telll ~0 thQy are ¦ removed. Memory ar:r y 60 mu~t then undergo a m~ory res}rnc before the CPU~ and me~aory ele~nent8 can re~oiII the ~y~tem.
Th~ CPU/memory f ault code of the ~oded error line~ in pathway 25 indlc~ta~ o CPU 30 ' thac thQ CPlJs ancl m~o~ey slement4 of CPU 30 h~ve be~n faulted.
The control line~, whlch repr~ t a coD~bin~tion of cyclQ type, error typ~, zmd ready ~ond~ r:i~, provide th~ ¦
handshaking betws~ CPU ~odulQ~ (30 ~nd 3û') and the I/O
modula~ . Cyclc typ~, as explainec'A above, d~f ine~ the type of :~ bu~ oper~Ation being perA4Orm~d~. CPU I/O r~3ad, D~ l:ransfer, ~MA setup, or in~rrupt vec~or r~uQs~. Error ~ d~flnss ei~her a firew~ compare or a CRC arror. ~aady~
me~s~g~ ar~ ~nt bGtw~en thR C~U ~nd I/O modulea to indica~e the compl~tion of requa~l:ed ope~atlorl~.
Th~ ~r~l cros~-1ln}c include~ two s~t~ of two 111191~ to :~ pro~rid~ ~ s~rlal da~ tran~f~r for a ~ u~ r0~d, loopb~ck, and d~ta tr~n~f~r.
.~ The clock ~ignal~ a~cchan~ad ar~ ~he pha~e looked clock signal~ C~C Y ~nd CLKC~ H ~de1~y~d).
F~ . 20A~ how block dias~r~s oiE th~ nts of CPU
wOrrlc~ modul~l~ 3û arad 30~ ~and I/O modul~ 100 and 100' through which ECAN, HENDER50N;
FAAA80W CARAE~r ': 177~1 ~ ST11C7:T. !I W. j -- 6 5 o~Or~.O c ~ooo-2 0 ~ C ~ 0 , ' ' .

d~ta pa~ses during ~h~ difforenl: oper~tions. Each of tho~e element~ ha~ each been de~cribed previously.
Fi~. 20A shows th~ data pathw~y~ for a typical CPI~ I/O
! read oporation of data from ~n I/O module 100, ~uch as a CPU
I /o register r~d operation of regi~tex data f rom shaved m~mory controll@r lO~0 (1050' )~ Such an opexatiosl will be ref~rred to as a re d of loc~l data, to di~tingui7h i~ ~rom a :~ DMA read of data from local memory 1060, which usu~lly contains data from an internal de~ice controller. The local data ar~ pre3umed ~co be Ator~ in lo~l ~N ln60 (1060' ) for tr~nsfer through ~hared m~msry controlle~ 1050 (1050' ) . For ona p~th, the data p~81~ through ~irewall 1000, modul-intarcsnnect 130, to cro~-linls 90. ~ 3een in Fig. 12, cro~s~link 90 del~ys the datl~ frozll f i~e;w~ll 1000 t9 m~mory controller 70 ~o that th~ data to c~oæs-llnk sa~ snay be presen~ed to memory controller 70 at the ~ame tima th~ data are pre~ented to memory cc~ntroller 70~ thu~ allowing procs~ing ~y~tem~ 20 and 20 ' to r~in ~yn~hronized. The .~ da~a then proce~d out of m~ory controllara 70 and 70 ~ in~o CPUs 40 and 40' by way of lnternal bu~sei3 46 and 4S'.
~ si3~1ar p~th is taken for raadlng data into CPV~ 50 and 50'. Da~ ro~ ~che shar~l sn~ory controll~r 1050 proe~lo through firewall 1010 and lnto cro~ k 9~. ~t .~ th~t ~c~o, th~ dat~ o.r~ rou~d both to cros~ k 95~ and throus1h a del y unit in~ide~ cro~-link 95.
I CPU I/O r~ad operations ~y 180 be perform~ for data .w o~lC59~ ¦ receivQd ~rom thla I/O d~vices of proc0~sillg 3y$tam 20 via a INNECAN. HENDER50tJ ¦
F.'~RA90W C~RRE~r 6 DE:~;NER
177~ ~ srl~s~T, Y w ~, ".~-oY. O c , ~, 120~1 ~D~ 0 ~' I .', _ I¦ Rhared memory control.ler 1050' and local RA~ in I/0 device ! loo.
Although I/O modulea lO0, llO, and 120 ar~ ~imilar and correspond to I/O madule~ 100', llO', and 120', re~pec~1valy, the corre~pondin~ I ~O rQodul~s are not in lock~tep ~ynchxorlization . U~ing memory controller 1050 ' and local RAN
1060 ' for CPU I/O read, th~ data would firs~ go to cro~-links 9 0 ~ and 9 5 ' . The re~ining daka path i~ slluival~nt to th~ path from memory controll~3r 105û. The data trav~l from the cro~salink~ gn~ and ~5' up through Dl~mory contruller~ 70 and 75~ and finally to CPUa 40' a~d 5û~, re~p6lc~ively.
Simult~neou~ly, th~ data tr~v~l acro~ to cro~s~ ss 90 and 95, re~pectlvaly, and then, without p~aing through a del~y el~m~nt, th~ dat~ contlnue up to CPU~ 40 and 50, r~p~ctiv~ly.
Fig. 201~ hows ~ CPU I~O wslt~ op~r~tlon o~ lo~al da~
Such loc~l data ar~ tran~ferr~d fro~ th~ CP~ 40, 50, 40~ and 50~ to an I/O modula, such as I/O ~odule 100. ~n example of uch ~n op~ratio~ write to a re~i~ter i~ ~h s~d memory c~n~ ^7r~ 10S0. The d~ta tr~sferr~d by CPU 40 proce~d alsng tha ~a~ p~th but in ~ direotion oppo~ite to th~ o~
th~ d~ta dusln~ tho CPU I/O read. Specifically, ~uch data p~8~ t~rou~h bu~ 46, memory co~troller 70~ various latche~
(to per~t ~ynchronl~ation)~ fir~ll 1000, and ~amory controll~r 10S0. D~ta Ero~ CPU sa ~ ~ls70 follow the path of the CPU I/O re~d~ in ~ rever~ d$re~tion. Specifically, such ~ ~Aw or~lce~ data pa~7a through bus 56~, me~ory controller 7S', cro~7s-link 'INNEG~N, HeNDeRSON
FAAA30W CA~RETr li DU~ iER
~l~7~7R~T ~ w. 67 -R~lOT5r~. O C ZOOO--~O~ o 1~

~ ' :: :

2 2 ~
I 95~, cro~-link 95, and into flr~w~ll 10~0. As indicat~d elhave, firew~ 1000 and 1010 ch~cls the dat~ dur~ng I/O
I write operation3 to check ~Gr errors p~ior to 3to~ag~, ~ han write8 are pe~formed to an I/O module in tha other ~onc, a similar operation i~ perormed. 80wever, the d~ta from CPU~ 50 and 40' ara u3~ad inatead of CPUs 50' and 40.
The data f rom CPUs S O and 4 0 ' aro tr n~mit~ed through etrical p~ths to sh~red memory controll~r 1050 ' O The da'ca from CPUa 50 and 40 ' aro compAred by firawalls 1000 ' and 1010'. Irhe rea~on different CPU pairs are u~sd to ~enric~ I~
O ~rrite data i~ to allow checklng of 11 da~a paths durlng .
nor~l u~e in a full duplex sy~te~. ~r;ter~il check~ for each zono were pre-,riously perfo~d at m~nory controll~r~ 10, 75, 70' and 75'.
Fig. 20C ~hOWB tha data path~ fox D~ raad operations.
The dat~ from mamory array 600 p~ simultaneously into memory con~rollers 70 and 75 and ~hen to crc~s~ ks 90 and 95. Cro~ 90 delay~ the da~ ~ransmi~ d to fi~ewall 1000 ~o that th~ data from croa~ 95 and 95 reach f ~ rewz~ 1000 and 1010 a~ ~ub~ ntially th~ 8~ timo.
SLmilar to i:h~ CPU I/O writ~ op~rativn, th~re are fous:
coplo~ o~ dat~ of data to the~ v~riou~ cro~ . At ~he ~Lr~l, only two copie~ are r~caivsd. A dlf~ererlt pair of d~t~ ~rell us~ ~h~n p~r~orm~ng ra~d~ to ~ona 11. T'ne d~ ~
path~ for the D~ writ~ op~raticn ~o ~ho~n i~ Fig~ 20D and ~r~ r to tho3e ~or a C~?U I/O re~d. Spe~iicelly, dat~
from ~h~red m~mory controll~r 1050' proce~d through firewall '~NECA~ HE~DERSON
FARA~OW ~ARRETr ~ DL~ ER
177~ ~ ST17~T, N W --' 6 8 MIMOTOM. O C ~OOO--1~0~)~9.11-01~50 ..

' . .", ' .~ .

';
:
' '' ' . 1000', cross-link 90' (~ith a del~y), mQ~o~y controller 70', I and into mQmory array 600'. Si~ultaneously, ~h~ da~a paB8 through firewall 1010', cros~-lin~ 95' (with a delay)~ and I memo~y controller 75', at which time it i8 compar~d with the data from mamory controller 70' during an int~rrail error chec~. As with the CPU I~O read, the d~ta in a DMA write ¦ operation may alternatlvely ~e b~ought up through ~hared memory controllor 1050 in an equivalent opera~ion.
The data out o~ cros~-llnk 90' al80 pa~ through cro~s-link 90 ~nd memory controller 70 and into ~mo~y arr~y 600.
The data f~om cro~s-Link 95~ p~B~ through cro~s-link 9S ~nd memory controller 75, at which time they are co~pared ~ith the data from m~m~ry controller 70' during ~ ultaneoua in~errail ch~ck.
Tha data path for a memory r~0ynchronization [re3ync) op~rRtion L~ shown ~n ~ig~ 20E o In th~ f~ oper~tion th~
con~ent~ of both m~wory array~ 60 ~nd 60' mu~t be ~et equal ¦ to each other. In m~ory res~nc, data fro~ msmo~y array 600' p~ through m~ory con~rollers 70' and 75' under D~
control, then throu~h ~ros~-link~ 90' and 95', rospecti~ly.
Tho dat~ then antar~ croas-linXs 90 and 95 and m~ory controller~ 70 and 75, respectively, bafor~ b~ing ~torad in ~o~y ~rr~y 600.
2.
Th~ pr~c~dlng di~cusalona of 8y8t~ 10 have made reXer~nce t~ many dlfer~nt ne~d~ ~or res~t~, ~n ~srt~in ~w orrlcr~ in~tanc~ no~ dL~cu~sed, r~ets ar~ u~d for ~tandard InlNEC.~, HENDERSON
F.~RA~OW CARRETr I; 3UNNER
,", n sr~eT~ N. ~1. 1 -- 69 W~l.lOT0~ 1 C ~0000 ~20~1~9~ 0a~0 ~ ' ' ` "

', ll ~22~
functions, ~uch aR wh0n power i9 Lnitially applled to 3y~tem 10- Most ~y~tem8 have a ~ingle r~et which alway~ ~et~ the proc~sor back to ~om~ predetennLned or initial qtate r and thu~ di~ruptq the proc~3sor~ ~ in8truction f low. Unlike most . other yQt~3ms, however, reset~ in ~ystem 10 do not aff~c~ tho Il flow of in~truction execution by CP~ 40, 40~, 50 and 50~
I ~ unle~ absolut~ly neces~a~r. In addL'cion, re~e~ in sy~tem 10 af f ect only those portions th~t n~sd to b~ r~t ~o ' re~tore normal operation.
! ~other a~pect of th~ re~'cs in sy~t~m 10 L~ th~ir containm~nt. On~ of the prime con~idsration~ in ~ f2lult toleran~ 8y9tam i8 thAt no function ~hould be allowed so 8top the 8yate3~1 from operating ~hould that f~un fail. For chi~ r~son, no ~ingle ro.et in sy~te~ lû control~ nt~
of both zon~ 11 and 11 ~ without direct coop~ration betw~en zones 1$ and 11'. ~hu~, ln full duplex mode of oper~ ~on, all rcs~t~ ln ~one 11 will be lnd~p~ndent of re89tl~i in zone 11~. When syst~ 10 i~ in mastar/slave mode, how~ r, tha 31ave zone u~e~ the re~ts of the ~tex zoJIe. In addition, no ra~et in ~y~te~3 10 a~fec~s th~s co~t~rlts of memory chip3.
Thu~ m3ikh~r cach~ ~emoxy 42 and 52, ~crat~h pad mQmory 45 and S5 nor m~mory module 6û 10B~ any data du~ to a re~e~.
~ h~r~ ar~ pr~f erably thr~e cla~ e3 o~ reoe~ in ~y~te~
10; ~clock re~t, ~ nh~rd re~et, n and ~oft re~3t. ~ ~ clo~k reset realigns all th~ clock phas~ g~Dn~rator~ in a zone. A
clock re~ot in zon~ 11 w~ll also inltlal~s~ CPUs 40 and 50 ~AW or~lcr ~ and msmory ~odule 6 0 . A c loc~ r~t do~ not af f e~t th~
ecA~, H~D~R50 FARA30W~ RRE~r ~3 D ~ ; E R ' 1~711 11 ~TR~T N ~ -- 7 O
aTO~ D. C ~ 000 -1~0~ 0 . ~ .

~,- ' ' .' .
.. ' ' . , .

':
' ~:

2~2f~ 1 mo~ule interconnect~ 130 and 132 except to realig~ the clocX
pha~e gerlorator~ on those module3. Even when ~y~tem lO i8 in ma~ter/~lave mode, a clock re8et in the slave zone will not i di~turb da~a transfer~ from the ma~tsr zona to th~ slave zone I module interconnect . ~ c3 ock re3elt in zon~ 11 ', how~ver, will initialize the corre~ponding ~lements in zone 11 ' .
In genarfil, a hard re et return~ all state de~ic~s and regi~ters to ~ome predetennined or ~nlti~l ~tate. A ~oft re~et only returns state enginas ~nd te~por~ry storage registers to 'cheir predete~min~d or in~ti~l ~tate. Tha ~tate ~ngin~ in a modul~ i~ the circuitry that defin~ ths ~t~te of that modul~. RaSTi~1 er8 cont~ninq error ir~formation ~r~d ;~ ~on~iguratLon data will not be afe~ctl3cl b~ a sof t re~et .
Addltlonally, ~y~tem 10 wLll e~eleeti~r~31y apply both hard re5~3t8 and ~oft r~et~ ~t the ~a~0 tilae to re~6lt only tho~e ele~3ntg that noed to be relnitializ~d in order to Gontinue procas~ing .
The hard r~t~ clear syst~ 10 and, a~ in conventional 9yatQlQ8, return æy~t~ 10 to al lcnown ~onfi3ura~ion. Hard r~et~ are u~d after power i~ appliç~d, when zone~ ~re to be ~ynchxonl~KI, or to initializ~ or dl8~ an I/O mo~ule. In sy~ 10 th~r~ ere pr~'0rably four hard re~e~$s "powar up res~t," ncPa hard r0~e~t,n nmodul~ reæ~t," and "devic~ re~t.'' E~rd re~ s c~n b~ furthsr bro~en dowrl ln~co loc~l ~nd sy~tem hard re~ . A lo~al hard r9~ on1y affect~ logio that r~pond~ when the C:PU i8 in the ~lave mcade. ~ ~y~ m h~rd w O~IC~
CINNECAN. HeNDeR50N
FARA~OW GARRE7r ~ DUNNER
w ~ 71 ~
TO!~. O ~ oooa ,~ 120~1~e~ aD~O

' . ` :
, 2 '~ ~
reset i3 ~ ed to th~ logic that L~ comlected to cros~
cablou 25 and module interconnect~3 130 and 132.
The power up reset i~ u~ed to initialize zones 11 and 11 ' immediately af ter power i~ ~upplied . The power up re et forces an auto~tic re3et to all parts of the zone. A power Up re~8et i neYer connected batween th~ zona~ of ~y tem 11 becau~e ~ach zone has its own pow3r supply and w$11 thu3 exp~rience dif f orent length ~ psw~r-on ~ e~2nt~ . ~he power up reset i~ implem~nted by applyirlg all hard reset~ and a cloc3c re et to zone 11 or 11'.
The C~U hard rs~et is u3~d f or dlagnostlc purpos~ in order to return a CP~J module to ~I known st~tq~. ~he CPU ha~:d reset clear~ all infoxm~tion in the CP~, mesll~ry contl:oller~, and memory module statux regi~ter~ ln th~ affe~ted zone.
Altho~gh 1:h6t cache memorie~ and -m~o~:y ~odulea ar~ di~abled, the content~ of the ~rat~h pad ~a 45 and 55 and of the memory ~odule 60 ar~ rlot change~:l. In addition, wllike the power up rs~et, the CPU h~rd r~t ds~ not modify the zone iderl~iflcation of the cross-links nor th~ clock ~astership.
The CPU h~rd rel~et $~ the s~a of all lo~l h~lrd re~et~ that can be appll~ to a CPU modul~ and fi ~lock re~et.
~ 3 ~ul~ hardl re~ç~t i0 u~d to s~t th~ I~0 modulQ~ ~o a kno~n ~ta~o, ~uch a~ durlng bootstrapping, a~d is ~19e used to re~ov~ a fa~ g I~0 modul0 from thts ~y~t~m. Th~3 I/0 module hard re~0t clear~ eY6~ hing on the I/0 modula, l~ves the firewall~ ln a di~gno~lc mod~, and di~able~ the driYers.
L~W O~IC~ I
-INNEC~N. HENDER50N ~
F.~R,~O~ CARRETr ¦
~ DUN~ER
~7~ Tn~,~T. N W. j 7 2 ING--01 . Cl C ~000--2 0 ~ 1 2 9 ~ 0 ~ I
'' : . ' ' . .' ' ' .'' ' " , .

~ 2 2 ~
A device re~et icl u~ed to re8et I/O device~ C0~16Cted to the I/O module8- Tho re5et3 ara d~Yice dependont and ars provid~d by the I/O module to which the devic~ i8 connected.

, The other class ef reset~ i3 soft reset~. A~ explained I above, qof t r~t~ clear the ~tate Qngineq and temporary i regi~te~s in sy~t~m 10 but they d~ not ch~nge configur~tion i information, such a~ ths mode bit~ in th~ oroe3-links. In addition, soft reYet~ al~o clear the error handllng mechani~m~ in the module~, but they do not change srror regi~ters such a3 ~ystem error reg~st~r 898 and y~te~ falllt addre~s r~gi~tor 865.
Soft reset~ are targeted so th~t only the nece sary portions of the ~y3te3n are re~et. For e~ple, if modulel . intsrconn~ct 130 ne~d~ to b~ ss~et, C~U 40 i~ not resE3t nor are the devlcos colmoc:ted to ~/0 modul~ îOO.
q~here are threa uniqu~ a~pect~ o~ 80~t r~het80 On~ i~
I that each zone i3 ra~ponsiblQ for genar~ting its own re~at.
¦ Faulty error or reset logLc in on~ zona i8 thus pravented from causing re~et~3 in th~ non faulted zone.
I Th~ 3acond a~pect ls that the 80ft re~et doe~ no~
¦ di~rupt the ~ nee of in~ruceion execu~ion. CPUs 4a, 40', 50, 50 ' ar~ rs~0t on a comb~ned clock and hard re~et only.
Addlt~o~lly DA~ACrY controller~ 70, 75, 70' ancl 75' have eho~ ~tate angina~ and ragisters nece~sary to ~ervica CA~U
in~truc~i on~ attached to hard ra ~t . Thus the soft re~g~ i~
transpRrsnt to ~of twar~ exacution .

~AW.O~rlC~ ~
Fl`.'NECAN. HENDERSO~ ¦
F.~R~aOW G~RRETT
6 DU~Ne~ I
,"~ ~ ~r~.. N.~ I -- 73 ~51'1NOTON. O C ~OOO~I
~20~ h~0 ' ~J~22~ ~
The third aspecl: i that the range o~ a soft re3et, that is the numb~r of ele~lentQ in 3y9teZm 3.0 th~t i affected by a ~oft ra~et, i8 dependant upon the mode of sy~tem 10 and the original re~et reque~t. In ~ull duple~ mode, the so~t reset reque~t o~ginating in CPU module 30 will l~ue a ~ot re~et ~o alL element~ of C~U module 30 a~ w~ll a8 all i~ewall~
1000 and 1010 attach~sd to ~odul~ interconnect 130 and 132.
Thu~ all modules ~enric~3d by module lntorcormect 130 and 132 Will haYe thair state engins~ and t~porary ragister~ re~et .
This will clear the ~ystem pipelin~3 o any problem caus~d by a ~ranslent error. Since 3y~te~ 10 i3 ln d~plex mode, zone 11 ' will be doing ev~rything th~t zone 11 i~ ~hus CPU
module 30' will, at th~ aamel tim~3 as CPU module 30, i3~ue a Ro~t re~et r~e~t . The ~o~t re~t irl zon~ ll ' will hav~ the ~ efect a~ the ~oft reset in zone 11.
When -~y~t~ 10 18 in a ~a~t~:r/slave mode, however, with CPU modulo 30' in the 31~ve ~ode, A ~oft re~et re~u~st oriqinating in CPU ~odul~a 30 will, a~ axp~cted, i~~UQ a ~oft re~e~ ~o all element~ of CPU rsodule 30 as well a~ all ~irew~118 1000 and lO10 attachecl ~o madule in~rcoAnect~ 130 and 132. A~ ionally, the ~oft re~t r~que~t wlll be fo~w~rd~d to CP~ mf:~dulQ 30' via cro~-link~ 90 and 90't croJ~ nk cabl~ 25, and cro~links 9û' and 95~. Pa~t~ of l~dU13 lnt~rconnect~ 130 ' and 132 ~ will receive ~he ~o t r~set. In thi~ same corlflguration, a ~cft r~ss~ roque~
originating froJa CPU module 30' will only re~ t memory w O~lCt-. ~ CINNECAN, HENDER5oN
F.~RA~OW GAAAETr li DUNNR
'177~ ~ Sr~Ctr ~ ~ -- 74 'I' w~S~ Cro~ O C 2000-(20~92 ~ 0 ,``
' , I ¦ 2 B 2 ~ 2 ~
I controller~ 70' and 75' and portion~ o~ cro~-links 90~ and ~5~.
Soft re3Qt3 include l'CPU soft re8et~ and "9y8tem ~oft . resetz.~ A CPU ~oft r2set i9 a soft reaet that affects the state engine~ on tha CPU module that originated the r~que~t.
I A ~yRtem soft res~t i~ a qoft reset ovf?r ehe module l ~ in~orconnect and t~ose el~m2nt~ dir~ctly ~ttached to it. A
I CPu msdule can always requ38t a C~U soft re~et. ~ ~y~t~m I o~t re3et can only be request~d i~ ~h~ cro~s-link o~ ~he reque~tlng CPV i~ i~ dupl~ mode, mast~r/slave mode, or off mode. A cro~ link in th~ 8~ve mode will take a sy~tem soft roset from the other zone and gen~r~te . yst~m ~oft re~et to its own modul~ intorconnecta. ~
CPU so~t res~t~ claar the CPU pip~l~ne following an error conditlo~. Th~ CPU pipelino include~ m~mory interconnec~ B0 and 82, l~tche~ (not ~hown~ Ln me~ory controllor~ 70 and 75, DMA 3~gln3 sao, and cros~-links 90 and 9S. The CPU ~oft rss~t can also occ~r following a DMA or I/0 time-out. A D~A or IJo tim~-out occur~ whe~ th~ I/0 deYice doe~ not ro~po~d within a speslfi~d ~im~, period to a D~A or an IfO r~que~t.
~i P~g. 21 ~how~ the r~et li~es ~ro~ ths CPU module~ 30 and 3a~ ~o the ~/0 module~ 100, 110, 100~, and 110~ and to the ~ory ~odul~5~ 60 and 60~. ~h~ ~PU module 30 receives a DC 0~ nal indica~ing when the power ~upply ha~ t~led.
i It 1~ thi~ ~lgnal which initialize~ tha power-up re~et. CPU
L~W05~1Ct~ modul~ 30~ r~ceives a similar 3$gnal from its pow~r ~upply.
INNEC~W HENDER50N
F.'~RA~OW ~P.ETr ~, 6 DUNNER
177~ 1~ gTC~T, ~1 ~1. -- 7 S
IIIOTO- O C 2000~

:;

~, .

I ~J~t~
one ~y~tem hard res~t line LR ~9nt to ~ach I /0 module, and on~3 ~y~tem soft rQ~et i~ 8~1nt to eVQI~ three I/0 modules.
The r3Ason that ~ingle hard reR~ needed for each module i3 becau e the 3y8tem hard reset line are used to remov0 I individual I~0 modul~ from l2y~t~m 10. The limitation of thre~ I/0 module~ for each ~y8t'i~ ~oft re~et i~ merely a i loading con~ideration. In addition, on~ clock reset line i~
¦ ~ent ~or every I/0 module and m~mory m~dule. The ra~son for u~ing a Qingls linQ p5r modul~ i8 t O conerol the sk4w by controlling the load.
Fig. 22 ~how~ the ~l~ents of CPU ~odul~ 30 whi~h relata to r~3sets. CPU~ 40 and 50 co~tain clock gQn~rator~ 2210 and 2211, r~p~ct~vely. Memo~ controll~rs 70 and 75 cont~
:~ clock gen0r~tor3 2220 and 2221, r~pectit,r01y, and cro~a-link3 90 and 9S contain ¢lock generator~ 2260 and 2261, re~p0c~ively. The clock g~ner~tor~ di~lde down the ~y3ti~
cloc3c ~ for u~e by th6~ indivldual modllle#.
l~ory controller 7 0 contain~ r~t con rol circuitry :; 2230 and a soft rese~ reque~t reg;stiar 2235. Me3l1ory controll~r 75 contain~ reset con~rol c~rcuitry 2~31 and a ~oft raset reqlleot r~gister ~236.
C~o~olink 90 cont~in~ both ~ lo ::~l resa~ gener~tor 2240 and ~ ~y~t~ r~t g~nera~or 2250~ Cro88~ k 9S corltain~ a local r~llot g~loxntor 2241 ~rld ~ 3y~t~ re~et gen~r~or 2251.
Ths " local " portios of a cro~-link ~ hat portion of th~
cxoa~-link which xa~is~ with the CP~I modul~3 wh~n that cro~-w or~lce~ lirlk i~!3 in the 81~Y~ mod0 ~nd th~refore inelud~ the ~erial -rl~NEGA~, riE~DE~O~J
FARA~OW C,~RRETr ~ DUI~;~ER
,,7, ~ ~T~ ~ -- 7 ~; ~
~A5111~CTO~, O C tOOO~I
011 ~q~ 0 .' ., .~ .

2 ~
regis~e~ and 90me of the parallel registQr~ . The 1` ~ly9'Cem'' portion of a cross-link i~ that portion of the cro~-link . that 1~ neaded for acce~Y to module interconn~cts 130 and 132 (or 130' and 132' ) and cro~-link cables 25.
ll¦ The local re~et generator~ 2240 and 2241 generate resets for CPU module 30 by ~ending h~rd and ~ot reset signal~ to the local re8et control cLrcuits 2245 and 224fi of cros~ link~
, 90 and 95, r~spectively, ancl to the re~t control ci rcult~
;~ I 2230 and 2231 of memory controll~r 70 and 75, re~pectiv~ly.
I Local cross link reset control c~xcllits 2245 and 2~46 re~pond ¦ to the ~o~t resel: 3ign~1~ by re~tting th~ir ~tate engin~, the latche~ storlng data t9 b~ tran~rred, ~md their error - regl~ter Those circuil:8 reapond l:o tha hard re~et 9ignal8 by taking the 3a~a~ action~ &~ a~a t~ken ~o~: the so f ~ re~et~, and ~y al~o resettin~ the crror reg$~ter~ and the configur~tion r~giotar~. Re~e cont~s)l circuits 2230 alld 2231 re~pond to har. and soft r~set 8igsul8 1s~ a similar manner .
,~ . In addition, the local rese~ ganer~tor 2240 send clock raset Sig~lal5 to the IJO module~ lQ0, 110 and 120 via module intercon~ t~ 130 and 132. ~he I/0 modul~s 100, 110, and 120 u~ th0 ~locls re~e~ 8iglU!Il~ to res~ their c10~3u in the ~n~ao~ d0~crib~d b~slow. Sof1: reset reque~t regi~r~ 2235 I ~nd 2236 ~n~ soft r~ es1: signal~ to local r~s~t genorator~
,; 1 2240 and ~241, reap@cti~rely.
! Sy~t~m r~et gerlç~rator~ 2250 and 22Sl of cros~ 90 :~ ~An O,~,ci., and ~5, raspac~ely~ ~end sy~t~a hard re~6~t signal~ and INECW, He~lDERSON
F.~R~DOW C ARRETr ,", ~ ~Tn~T. ,. W. -- 7 7 ' . W~ OTO~ D. C ZOOOC
~0~

, .
, ,, , ' ' ' ' ' ' , ~ J~2~2~
I ~ystem ~o~t res~t ~ignal3 to I/0 modules 100, 110, and 120 'i via modulR interConn~\ct9 130 and 132, re8p~ctlvely. I/0 I module~ lO0, llO, and 120 re8pond to the soft reset ~ignal8 by rQ~etting all regi9t~r9 that ar~ depend~nt on CPU data or command~. Tho3e module~ r~pond to the hard re~et signals by resetting the ~ame r~gi~t~r a~ ~o~t r~s~t~ do, ~nd by al80 resetting any confi5~ ation regi5ter8.
In addition, the 91y3t;Bm re~et ~enerators 2250 and 2251 al~o ~end the ~y8tem 80f t and ~yst~m hard ra8et ~ign~18 to '. the 8y8tqla ro~et control circuit 2255 and 22S6 o~ e~ch cros~
11 lin~c. 5ystem re~t control circuit 2255 ~nd 22S6 respond to , tho Yy~tem aoft re~t 5i9~ 8 and to the 8y81~ hard r~3~et - I ~ign~l~ in ~ n~nnsr silll1l~r to the re~pon~3 of th~ local il r~sot control c$rcuits to th~ lot~ oft ~nd local hard r*se~
3igllal~ .
~ emory cc:ntroller~ 70 ~nd 75 c~u~e cro~ 90 and 95, re~pectivsly, to gener~te tho ~ot re~ok~ when CPUs 40 and 50, r~spe~cti~ely, w~ite th~ appropriate codeR into ~of~
reaet raqua~t reg$~t~r~ 2235 and ~236, ra~pectL7rely. Soft ro~t ~equ~ rog~tsss 2235 ~nd 2236 ~and ~of~ ra~t reque~
~ign~l~ to loc~l re~e~ gen~rator~ 2240 and 2241, r~ c~ 3ly. l!he cos:l~d error slgslal i8 Q9nt fro:m mamory I con~roll~ar 70 to local re~at gen~l3ratoræ 224û and 2241.
System aoft r~6~ta are Yent b~tw~e~ zon~ along the~ same d ta path~ data and control 9i~18 are ~3ent ~ Thu~ ~ the same philo~ophy o equalizing delay~ i8 u80d for ~ee~3t~ a~ for ~w O~V~
-I~NC~:~I. HENDERSON
F.~R.~30~ cARRE~r ~ Du~ eR
1~5 .~ ~T~ 7 ~ --OTO~. a c 2000111 zo21 z1~a r ~o .1 .

data and addres~e~, an~à re~et~ reach all of the elements ln bo~h zone~ at ~pproxim~te1y the s~e t~
Hard re~etq ara generated by Cl?U8 40 arld 50 writing the appropriatq code into the local hard r~set register3 2243 or by the re~3ues~ f or a power up reset cau~ed by the DC OR
signal .
Syrlchroniz~tion circuit 2270 in cross~ k 90 include~
appropriate d~lay elem~llt8 to en~uxa that the~ DC 0~ ~ignal go~39 to all o~ the local and r~t gener~tors 2240, 2250, 2241 and 2~51 a~c the ~me time.
In f act, q~ chror~ization of r98`3t8 1Z~ V0ry important in 3y8tom 10 o rrh~lt i~ w~y the reset signal~ origin~te in the~
cros~-1ink~. In that w~y, ths res~t~ can be ~ent to arrl~oe at differsn~c li;odu1~s and ~ ts i~ th~ ~odules ¦ approsci~taly ~ynchronou31y.
Wlth the und~3r~t~nding o~ the structure in Fig~. 21 as~d 22, ~che ex~cution o the d~ ferent hard re~ets can be better ~ under~tood. Tha power up rss~3t g~nerata~ both a 9y tem h~rd I re~et, a loca1 hard re~et and a clock r~set. G~nerally, cross-lLnk~ 90, 95, 90' and 9S' are~ lnitially in iboth the croa~-lis~ o~ nd re~ync o~f ~o~, and with bo~h zon~3 a~er1:in~ c~13c3c ma~tsrship.
CPlJ/i~ ~aul~ r~l3e~ iL~ au~co~tically 2cti~
Wh~lnl2~Yer FAemWrY controll~3r~ 70, 7~, 70~ and 75~ detec~ a ! ! cPU/~ fau1t . ~h~ coded ~rror lo~ i8 ~ent ~rom erxor 10gic 2237 and 2 38 to both cro~0-llnk~ 90 and 950 The CPU
.AwOr~e~ ' modu1e whieh g~ner~ted the faul~ i~ th~ r~movscl fro~ 3y~tem l`iNECAN. H~!`iDER5oN
FARA~OW GARRF~Tr !
eR
A S~RtC~ 1. ~ 7 9 ~A51~ 5TO)~ O. C ~000~ I
1~0 ;~ 50 1~ ' ' .

~ . ~
:. :
. .
:,;

2 '~ ~ 1 10 by ~etting it9 cro9~-link to the ~l~ve a~ate and by 1, ~etting tha oro~s-link in the other CPU ;nodule ~co the m~ster 3tate. The non-faulting CPU module will not experience a re~et, however. ~n~taad, it will be notifi~d of. th~ fault in the other mcdule through a code in a ~erial cross-link error rQqi~ter (no~ ~hown). ThQ CPU/l~l fault rl~8et con5i8t8 of a clock reYet to the zone with th~ failing CPI~ modul~ and a local ~of~ re3et to tha~ module.
A re~ync reset i~ s~6entially a 81yBtem 80ft r~set with a local h~rd re~et and a clock resQt. The r~ c r~et i~ used to bring two zone~ into lock~tQp ~ynchronization. If, after a period 1~ ~hiGh zs~nas 11 an~ ~re not ~ynehronizod, ths contants of the m~ory modul~s 6D ~nd 60 ~udlng the~
stor~d ~tAt4~!} of the CPV regi~ters, a~ce set ~ual to e~ch other, th~ re~ync ras6tt is u~ed to bring the zon~ into a compatible conf iguration 80 they can re~tart in a duplex mode.
~ he re~ync r~t i~ sentially ~ C~ h~rd r~et and a cloc~s r~et. Th~ re~ c re~et i8 acti~vate~l by ~o:twars writing the re~ync re~t ~ddre~s into or~e o the parallal cro~R-link r~gi~ters. ~t that ti~a~, on~ zone ~hould ~ in th~ cro~ t~r/re~ync ana~te~ mod~ and the oth~r in tha cro~ Je/se~ync ~ mod~ . A si~ultanQcsu~ reset will th~ra b~ pe~îos~ed on both th~ zone~ which, amonq othar ~hingo, w~ ll 0~t ~11 four CS0813~-1islk8 lnto the duple:~ mod~?.
Since the ~esync re~et $~ s~ot a ~yst~ soft re~t, the I/O
module~ do not rs~celve reset.
I~NEC,~N. ~E~DERSON
F.~RAEOW C~RRETT
li DU~l`iER
17~ ~ sT~rrT~ rl w i ~ 8 O
s701~, 2 c ~OOo~ I
120.~...0 !

;l ~ 2 2 '~; ~
~¦ The preferred embodLment of system 10 al~o en~ure~ that !¦ cLoe3c re~e~ 3igs~ do not re~et con~or~n~ cloc3c~, only non~
¦ cenform~ng c1wk~. The reason for thi~ i8 th~t whenevor a clock i8 reset, it alter~ the t~ming of the clock~ which ln turn af fect~ the operation of the raodule~ with such clo~ks .
If the modu1e wa~ performinçl correctly ant lt3 clock wa~ in he proper phase, the~ a1taring its op~r~tion would ~e bo~h unnecessary and wasteful.
Fig. 23 ~how~ a pr~err~d ~ bodimont of cir~uitry which ! will en~ure th~t only nonconfor~ing cloGks ar~ re et. The 'I circuitry ~hown in Fig. 23 preferably re~de~ in the clock :~ gener~tors 2210, 2211, 2220, 2221, 2260, and ~261 of ~he cor-- ' re~ponding modu1e~ ~hown ~n F~g . 2 2 .
,1In the preferred es~csdisil0nt, th~3 dlff~rQnt clocX .
genGratOr~ 2210, 2211, 2220, 2221, 2260, and 226~ include a .
~i ri~ing edg~a dete~ctor 2300 and a pha~e genarator 2310. The ri~ing ~dgs d~tector 2300 r~ceive~ the clock rese~ sign~ls fro3ll tho c~eosæ-11nk~ 90 ~nd 95 ancl g63n~rat0s ~ pul~e o Xnow ¦ duEation con~urr0nt w1th the ri~ing ~ o$ th~ clock re e~
.1 signal. ThAt pul8e 1~ in an lnpu~ to ~he pha~e~ g~ rator 2310 a~ ~re ~h~ int~rnal clock ~ignal~ for t~e pzirt~cular . ~odul0. ~h0 1n~0n~al clock aignals ~or th~t modul~ ar~ clock . signal~ which ar~ derived ~ro~ g h~ ~y~tem clock ~ al~ that hav~ ~en dls~ributed fro~ o~cill~tor ~y~t~ 205 and 200'.
Phace ~esne~:cator 2310 i~ pre~r~bly a divid~-do~ circuit :I whic:h ~orm~ di~fe~rent ph~ for th~ elock ~ignal~. Othar ~w Or~c~
rl~lNeCA~I. HEtlDERSO~J I
RAYOW ~RRe~r ~i DU~I~IER
5T;l~ 8 1 TO~.O C 2000~ i ~2021 ~0~ o .

.'~ ' .~ :

?J 2 ~; o~
design~ for phaso generatox 2310~ sueh a~ recirculating shif~
register9, can al~o b~ u~d .
Prof~rably, the rising edgæ pul~e from ri3ing edg~
d~tector 2300 cau~e~ phaee generator 2310 to output a preselected phase. ~hus, for example, if phase genera~or 2310 were a d1vide-down circuit wLth æ~v~rsl 8ta~e8, ~hQ
clock r~et ri~in~ ~d~Q pul~Q c~uld be a ~et input to the stage whlch g~n~rata~ the pre~el~ted pha~e and a resQt input to all ather ~t~ge~. If ph~ss gen~rator 2310 were ~lready gener~ting that ph~sa, then the pres~nca of th~ ~ynchronizad clock resQt ~ignal would bs e~3Qn~ially transpar~nt.
The re6et~ thus org~nizod are des~gn~d to provi~e the minimal di~ruption to the ~ormal execution of 8yxte~ 10 t and only cau~e tha r~st~c actlon of int0rrupting the noxmal sequence~ o in~truc~ion e~eGUtio~ ~han ~uch dr~stic actlon i8 re~uirad- Thi~ i~ par~iGul~rly Lmportan~ in a dual or multipls zone er~qJixonment b~cau~e of ~he problem~ of re~ynchroniza~ion whiCh con-irentional re8et8 cau~e. Thu~, it i~ pre~erabl~ to n~ 0 ~che numbor o~ h~rd re~et~, a~ i~
done in 8y8tl~m 10.
It ~ill ~e app rent to those ~klllsd ~n th~ art that ~arlou~ modific tions ~nd Yariation~ ca~ b~ ~d0 in th~
~e~o~y of th~ px~ent invention w~thout departin~ fro~ t~h ~cop~ ox api~it ~ ths 1~o~tion. ~hu~ in~ended that th~ pre~ent inv~nt~on cover the ~odification~ and variations or thi~ inv~ntion prov~d~d they com0 within the 8COp~ 0~ the ap~end~d claim~ ~nd their equivalen~.
IEC/W. HE~IDERSON
F.~ OW. C~RRETr & D ~ E Q
17~ 5T~C~T. N W. 82 ~ NOTON. O C ~000-- ¦

.
' :, ;~ :
' ~

Claims (3)

1. A memory for storing data in a computer system, wherein the computer system includes a memory controller means for writing or reading data during a memory transfer cycle, and a memory interface bus for coupling the memory to the memory controller means, wherein the memory interface bus includes a plurality of bidirectional data lines, a plurality of time division multiplexed bidirectional lines, and a cycle timing line for providing cycle timing signals to the memory, and wherein the memory comprises:
memory array means including a plurality of addressable storage locations, for storing data and ECC signals;
sequencer means coupled to the cycle timing line, for generating a plurality of sequence timing signals in accordance with a clock signal and the cycle timing signals;
data buffer means, coupled to the plurality of bidirectional data lines, to the memory array means, and to the sequencer means, for transferring data between the bidirectional data lines and the memory array means, when enabled by the sequence timing signals during the memory transfer cycle;
control buffer means, coupled to the plurality of time division on multiplexed bidirectional lines, to the memory array, and to the sequencer means, for transferring address signals, provided on the time division multiplexed bidirectional lines by the memory controller means, from the time division multiplexed bidirectional lines to the memory array means, to access the addressable storage locations in the memory array means specified by the address signal, when enabled by sequence timing signals at the beginning of the memory transfer cycle; and for transferring ECC signals, between the time division multiplexed bidirectional lines and the memory array means, at times when data is transferred between the bidirectional data lines and the memory array means, when enabled by sequence timing signals after the beginning of the memory transfer cycle.

2. A memory for storing data in a computer system, wherein the computer system includes a memory controller means for writing or reading data during a memory transfer cycle, and a memory interface bus for coupling the memory to the memory controller means, wherein the memory interface bus includes a plurality of bidirectional data lines, a plurality of time division multiplexed bidirectional lines, and a cycle timing line for providing cycle timing signals to the memory, and wherein the memory comprises:
memory array means including a plurality of addressable storage locations, for storing data and ECC signals;
sequencer means, coupled to the cycle timing line, for generating a plurality of sequence timing signals in accordance with a clock signal and the cycle timing signals;
data buffer means, coupled to the plurality of bidirectional data lines, to the memory array means, and to the sequencer means, for transferring data between the bidirectional data lines and the memory array means, when enabled by the sequence timing signals during the memory transfer cycle;
control buffer means, coupled to the plurality of time division multiplexed bidirestional lines, to the memory array, and to the sequencer means, for transferring cycle type signals, provided on the time division multiplexed bidirectional lines by the memory controller means, from the time division multiplexed bidirectional lines to the control buffer means, to enable reading or writing of data in accordance with the type of memory cycle specified by the cycle type signals, when enabled by sequence timing signals at the beginning of the memory transfer cycle; and for transferring ECC signals, between the tome division multiplexed bidirectional lines and the memory array means, at times when data is transferred between the bidirectional data lines and the memory array means, when enabled by sequence timing signals after the beginning of the memory transfer cycle.

3. A memory for storing data in a computer system, wherein the computer system includes a memory controller means for writing or reading data during a memory transfer cycle, and a memory interface bus for coupling the memory to the memory controller means, wherein the memory interface bus includes a plurality of bidirectional data lines, a plurality of time division multiplexed bidirectional lines, and a cycle timing line for providing cycle timing signals to the memory, and wherein the memory comprises:
memory array means including a plurality of addressable storage locations, for storing data and ECC signals;
sequencer means, coupled to the cycle timing line, for generating a plurality of sequence timing signals in accordance with a clock signal and the cycle timing signals;
data buffer means, coupled to the plurality of bidirectional data lines, to the memory array means, and to the sequencer means, for transferring data between the bidirectional data lines and the memory array means, when enabled by the sequence timing signals during the memory transfer cycle;
control buffer means, coupled to the plurality of time division multiplexed bidirectional lines, to the memory array, and to the sequencer means, for transferring cycle type signals, provided on a first portion of the time division multiplexed bidirectional lines by he memory controller means, from the time division multiplexed bidirectional lines to the control buffer means, to enable reading or writing of data in accordance with the cycle of memory transfer cycle specified by the cycle type signals, when enabled by sequence timing signals at the beginning of the memory transfer cycle;

for transferring address signals, provided on a second portion of the time division multiplexed bidirectional lines by the memory controller means, from the time division multiplexed bidirectional lines to the memory array means, to access the addressable storage locations in the memory array means specified by the address signals, when enabled by sequence timing signals at the beginning of the memory transfer cycle; and for transferring ECC signals, between the time division multiplexed bidirectional lines and the memory array means, a times when data is transferred between the bidirectional data lines and the memory array means, when enabled by sequence timing signals after the beginning of the memory transfer cycle.