Fault Tolerant Routing Switcher Topologies for Centralized Distribution Systems

Progress and Pragmatism

Fault Tolerant Routing Switcher Topologies for Centralized

Distribution Systems

by

Charles Meyer and Robert Hudelson

NVISION, Inc.

Introduction

Advances in digital storage technology have made it possible to store vast amounts of program material in a central location. In addition, advances in digital distribution technology have made it possible to more efficiently move this material between facilities and, ultimately, to deliver it via a multiplicity of channels to a mass audience. At the heart of the network, switching and routing must be deterministic, error free, and fault free.

In the centralized location, be it a network operations center, centralcasting hub, or transmission uplink, the core router is likely to be quite large. A fault in even a single path could affect one or more stations and millions of viewers. A fault affecting a large portion of the router could have even more dire consequences. Therefore, protecting the integrity of all critical signal paths is extremely important. Routers must be highly reliable with a very low failure rate. They must be resilient. Any single failure should not multiply in a catastrophic fashion. Any failures that do occur should be easily repairable. Reliability, resiliency, and repair-ability are the cornerstone features of any fault tolerant centralized routing system.

Router Topologies

There are many different router architectures that may be considered for large matrix

implementations. For a broadcast facility, the following feature set is typical. These routers are non-blocking. They must be able to route any input to any output, to any arbitrary set of outputs, or to all outputs. Every crosspoint needs to switch at an identical and pre-determined instant in time and each output of the router must be able to change state at a single switching instant. The router must keep all its signal paths in phase. There are other features, such as dual reference switching, and multi-format operation, but the set just described is sufficient. The key is understanding how the unique characteristics of a given topology affect its reliability.

XY, or Space, routers are composed of arrays of crossbar elements. These elements may be integrated circuit multiplexers or discrete semiconductor switches. Interface circuitry surrounds the switch core providing required signal processing such as cable equalization and data re-clocking for signals entering and exiting the switch array. In one common router topology, inputs to the router are processed on input modules. Each of these modules provides fan-out to a number of crosspoint modules. Each crosspoint module feeds a number of output modules. Figure 1 shows such a topology.

SMPTE 144th Technical Conference and Exhibition - 1 -

October 23-26, 2002 Pasadena, CA

In 1Input ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleIn nInput ModuleIn 1 Progress and Pragmatism

Out 1Out 1Output ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOut mn X oCrosspointModuleIn nIn 1Out oOut 1n X oCrosspointModuleIn nIn 1Out oOut 1n X oCrosspointModuleIn nIn 1Out oOut 1n X oCrosspointModuleIn nOut oFigure 1: A Traditional Multi-Card Crossbar Router

By using multiple stages of crossbar arrays, it is possible to build non-blocking routers that use reduced numbers of crosspoint elements. Clos[1] proved that such a topology is non-blocking for one to one connections if a minimum number of middle stage crosspoint arrays were used. Karp[2] went on to prove that if additional middle stage arrays are added, such a router is also

conditionally non-blocking for 1 to many, and 1 to all applications. The condition is that some, or many, of the paths already established in the router must be re-configured. This rearrangement can cause unacceptable transient distortion to the signal. For signals which can be time aligned at the bit level, such as digital AES3 audio, it is possible to synchronize the switch array,

eliminating any such transients.[3] A block diagram of Clos’ three-stage switch is shown in Figure 2.

SMPTE 144th Technical Conference and Exhibition

- 2 -

October 23-26, 2002 Pasadena, CA

N

Arraysn Progress and Pragmatism

k Arrays

N

Arraysnn X kn X kNNXNNnnXnnk X nn X knN

Inlets

n X kn X kNNXNNnnXnnk X nn X knN

Outlets

n X kn X kNNXNNnnXnnk X nn X knFirst

StageSecondStageThirdStage

Figure 2: Clos Three-Stage Routing

Space routers typically switch serial data or analog signals. However, early digital routers for ITU-R BT.601-5 data, or SMPTE 292M data used multiple layers of space routing. Each layer corresponded to a single bit of the parallel data word. This reduced the digital serial rate through the crosspoint to 13.5 Mb/s or 74.25 Mb/s respectively. Input and output modules contained the necessary serial to parallel and parallel to serial conversion circuits.

Time domain routers usually require a data bus. In the simplest example, each input to the router is assigned a unique, fixed time slot on a distributed data bus. Routing occurs when an output signal latches data off the bus at the time slot allocated to the desired source. It is a requirement that the bus bandwidth is equal to or greater than the product of the bit rate of the input signals and the number of input signals. Figure 3 shows a simple timing diagram of such a router.

SMPTE 144th Technical Conference and Exhibition - 3 -

October 23-26, 2002 Pasadena, CA

Channel ClockClockDATA

0

1

2

3

4

5

6

Channel n

Progress and Pragmatism

n+1n+2

70123456701

Figure 3: Conceptual Bus Time for a TDM Router

Figure 4 shows the signal flow within the router. It is important to notice that there is not a crosspoint module, or chip, in this type of router.

DATABUSInput ModuleSlow Shift RegisternCLK

ChCLKOutput ModuleFast Shift RegisterFast Shift RegisterSlow Shift RegisterInput ModuleSlow Shift RegisterOutput ModuleFast Shift RegisterFast Shift RegisterSlow Shift Register

Figure 4: Time Multiplexed Data Bus Routing

SMPTE 144th Technical Conference and Exhibition - 4 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

A modern variant of time domain routing called Shared Memory Routing uses memory circuits rather than a bus. Input signals are converted from serial to parallel, and then written into

memory. Input 1 is written into address 1, input 2 to address 2, and so on. Routing occurs when an output reads data from the address corresponding to the desired input of the router. As with time routing over a bus, the read/write bandwidth of the memory must equal 2 times the product of the word rate of the input data and the number of inputs to the router. Data may be converted between serial and parallel on the input and output modules, or on the crosspoint module. Figures 5 and 6 show block diagrams of this type of router in an AES3 data application.

AES Input 1AES Input 1Subframe ASubfame ASubframe BSubframe BInput signals are split intoAES subframes.Converted into parallel data.Written sequentially intomemory.Input n-1 AAES Input 1Input n-1 BSubframe AInput n ASubframe BInput n BMemory LocationsInput 1 AAES Input 1Input 1 BSubfame AInput 2 ASubframe BInput 2 BAES Output 1AES Input 1Subframe ASubfame ASubframe BSubframe BOutput signals have randomread access from memory toperform 'switching'.Data is packed into AESpairs and serialized.AES Input nAES Input 1Subframe ASubfame ASubframe BSubframe BAES Output mAES Input 1Subframe ASubfame ASubframe BSubframe B

Figure 5: An AES3 Shared Memory Switching

In Figure 5, the sequential source data memory is shown. Each AES output is then created by a non-sequential read from two independent addresses. Shared Memory routing greatly facilitates routing of AES3 monaural signals.

Figure 6 shows how the shared memory approach can be used in combination with traditional Time Division Multiplexing Techniques to construct expandable routing systems.

SMPTE 144th Technical Conference and Exhibition - 5 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

NV7256 FrameInput Modules1TDM Crosspoint Modules1256Internal1-256AES Inputs51211Internal 1-256 AES Inputsexported to other framesOutput ModulesOuput ModuleOuput ModuleOuput Module2561512513512Ouput ModuleSignals from other framesExternal257-512AES Inputs10241025Read BusExternal513-768AES Inputs15361537External769-1024AES Inputs2048

Failure Modes

Figure 6: Shared Memory Switching with Expansion[4]

Most larger routers are composed of a number of printed circuit assemblies, such as input modules, output modules or crosspoint modules. In some topologies, modules may contain multiple functions. For simplicity, failure of power supplies, or modules which interface to

control and automation systems will not be considered. Failure of an entire module is major. As an example, an entire group of inputs may be lost to all outputs of a facility. Losing a crosspoint module is even more severe since every output it feeds no longer receives a valid signal.

Fortunately, the likelihood of these failures can be minimized if the modules are well designed and have undergone burn-in and factory test. Adequate training of service personnel is also

important. Errors during maintenance and service have resulted in removing the wrong module.

When a single component fails, the symptoms can be subtle, and also more confusing to debug. Single component failures can occur for several reasons, the first being improper handling. Proper care in handling circuit modules is becoming critically important. Smaller transistor geometry and higher electron mobility provide the high speed performance needed for routing 270 Mb/s to 1.5 Gb/s signals. These faster chips have lower breakdown voltages and more static

SMPTE 144th Technical Conference and Exhibition - 6 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

sensitivity. It is much easier to damage, or stress these chips in what used to be acceptable, safe environments. Centralized equipment rooms should maintain adequate humidity levels and

utilize anti-static flooring as a minimum. Ground straps should be considered as part of servicing equipment as well.

Accelerated aging in semi-conductors is the next most likely cause of failure. This can result from excessive thermal rise in the equipment chassis, poor thermal conduction of heat away from the package, metal migration inside the chip aided by moisture build up, or a number of other causes. If a device was stressed due to static discharge, operation at normal thermal profiles can result in early failure.

In a Space router, single device failures have a number of symptoms. A typical failure may occur when one output will cease to work. No input can be routed to it, but another output, even on the same module, will faithfully pass every input of the router. In another example one input may be lost to every output. If the failure occurs in the switch core, one or more inputs may be lost to one or more outputs. Isolating this error to a single module requires a good knowledge of the router architectures. The cause of this type of error is much more easily isolated and identified in the classic XY structure of figure 1 than in the Three-Stage structure shown in figure 2.

In a Time router, the same failure modes associated with a Space router can occur. But, there are other more interesting symptoms. Consider the bus structure shown in figure 4. Imagine that a driver on the MSB of the bus fails in such a way that the logic level is forced to a fixed state. Every output of the router will now receive incorrect data samples when the actual logic level of the MSB is opposite the level at which it failed. If the signal is an audio sinusoid, the output could look as shown below in Figure 7.

Ideal SignalFailed Bus Effect

Figure 7: Distortion Caused by MSB Bus Failure

SMPTE 144th Technical Conference and Exhibition - 7 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

Now imagine that this same error occurs in a low ordered bit. Total harmonic distortion will

increase less dramatically than the example shown. It is unlikely that this distortion is perceptible in normal program material. This type of error can be very difficult to find.

A shared memory router can exhibit all the failure symptoms of space and time routers plus two more. Once again, assume the input to the matrix is a digital sinusoid. If one bit in the memory array fails, then only one source to the router will exhibit the distortion at any router output. This type of error could go undetected until such time as the router is used to pass non-audio data. Specifically, the error could be in the LSB of a linear AES3 digital audio signal. It is extremely unlikely that this type of error will be detected with any analog audio test gear. However, if a compressed audio signal, such as Dolby E is passed through the router, the failure will become immediately apparent. While it is highly unlikely, Shared memory routers are also susceptible to soft errors in their memory core. These would manifest themselves as unexpected, untraceable spot noise.

It is important to note that in the case of either shared memory or bus oriented TDM routing, spreading the outputs of the router across more than one memory, or bus, will reduce the impact of a failure on the total output space of the router. It is also possible to design a Shared Memory Router such that the interconnect between the modules is identical with that found in a space router. In fact, for any router topology, it is possible, and typically desirable to manage the signal interconnect between the inputs and outputs so that the resulting impact of any single failure is minimized with respect to the total number of affected outputs. Combining these techniques, as is the case with the NV7256[4], mitigates many of the possible failures associated with basic Shared Memory Router implementations.

Impact Block

Impact Block is a number used to represent how many outputs of a router will be lost assuming a single module fails completely. The router is assumed to be configured in a diagonal with input 1 connected to output 1, input 2 to output 2, and so on.

Figure 8 shows a classic, 128 x 128 XY router with input, crosspoint and output modules. There are 16 input modules, each with 8 inputs, 16 output modules each with 8 outputs, and four crosspoint modules that are 128 x 32. If an output module fails, 8 outputs are lost. If an input module fails, 8 outputs will also be lost. The impact block size for the output module is 8. Obviously, in normal operation this number could vary, but the input module is considered to have an impact block of 8 as well. If the crosspoint module fails, 32 outputs of the router are lost, and so its impact block is 32.

Clearly, routers with smaller impact blocks are more resilient, providing more robust facility operations. In the limit, an impact block of 1, would be ideal. Unfortunately, Routers with smaller impact blocks are also more expensive.

Modules with multiple input circuits have lower overhead per circuit costs. Technology continues to put more and more functionality in less space. Crosspoint chips for 2 Gb/s

applications are readily available at 68 x 68 sizes, and will soon be available at 144 x 144 sizes. The density of interconnect components is also increasing. High speed connectors offer 4 twisted pair connections per 2 mm of card edge length, again at lower per pin cost.

SMPTE 144th Technical Conference and Exhibition - 8 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

The least expensive approach to build a 128 x 128 square router would be to use a single crosspoint chip including all the input and output circuits on the same circuit module. This

implementation provides a very compact chassis and the highest possible impact block, which is also the worst possible impact block for a 128 x 128 router, specifically 128.

In 1Input ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleInput ModuleIn 128Input Module88888888In 1Out 18888888888888888Out 1Output ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput ModuleOutput Module128 x 32CrosspointModuleIn 128In 1Out 32Out 1128 x 32CrosspointModuleIn 128In 1Out 32Out 1128 x 32CrosspointModuleIn 128In 1Out 32Out 1128 x 32CrosspointModuleIn 128Out 32Out 128

Figure 8: A 128 x 128 Router

Engineering is the practice of balancing technical performance for price. Robust plant

implementations require smaller impact blocks, and typically, slightly more physical space, but affordability prevents an impact block size of 1. Obviously, to reduce the size of the impact block, the crosspoint implementation must be addressed.

SMPTE 144th Technical Conference and Exhibition - 9 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

The N on 1 Redundant Crosspoint Topology

A novel compromise to this design dilemma is to provide redundancy for the Crosspoint section of the router. Consider a topology where a redundant crosspoint module is added to the system. Figure 9 below shows a router constructed with P input modules, each with p inputs, M output modules, each with m outputs, N crosspoint modules and 1 redundant crosspoint module, labeled N+1. It should be pointed out that the number of input and output modules need not be equal, and that the number of signals on each input, output and crosspoint module will vary as needed to provide the optimal tradeoff of price, performance and reliability for a given application.

Each input module feeds a copy of its signals to each crosspoint, including the redundant module. The signal paths between inputs and crosspoints may be point-to-point for 1.5 Gb/s signals or bussed for applications where bandwidths are sufficiently low. Each crosspoint feeds a fixed number of output cards. Assume that each crosspoint module is 256 x 128 and each output

module supports 16 signals. Therefore, each crosspoint module is dedicated to 8 output modules in a 256 x 256 router. In this scheme, the connections between crosspoint and output are point-to-point. The redundant, or N+1th, crosspoint feeds each group of output cards in common. Just like the connections between the inputs and the crosspoints, this topology can be point-to-point for 1.5 Gb/s signals, or bussed for lower bandwidth requirements. In effect, the redundant

crosspoint may be thought of as an identical crosspoint module with each output comprising a fan out distribution amplifier.

Input Module 1 Input Module P XPT Module 1 XPT N+1 Module XPT Module N Output Module 1 Output Module 2 Output M-1 Module Output M Module

Figure 9: N+1 Redundant Crosspoint Router Signal Flow

SMPTE 144th Technical Conference and Exhibition - 10 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

The topology looks surprisingly like the three-stage router of figure 2 however, the fact that each input feeds every crosspoint differentiates the two architectures and guarantees that the N-on-1 approach is unconditionally non-blocking for any arbitrary configuration, including broadcast.

In figure 10 below, the method of implementing the N-on-1 system is quite clear. Each

crosspoint module is 256 x 128 and feeds 8 output modules each with 16 outputs. Each output card receives two inputs for each output. One is from the crosspoint card that normally feeds it. The second input comes from the redundant crosspoint card.

In 1In 1256x128InputModuleIn 16PrimaryCrosspointIn 256In 1256x128Out 1Out 1Out 128Out 1OutputModules1-8Out 128RedundantCrosspointOut 128In 256In 241In 1256x128InputModuleIn 256PrimaryCrosspointIn 256Out 1Out 128Out 129OutputModules9-16Out 256 Figure 10:

If the redundant crosspoint card is to be used, it must be given the necessary matrix map for the card it is replacing, and then the 2:1 selector on the output cards must also be switched. It is possible to update every crosspoint element in less than one vertical. Then, all crosspoints may switch during the vertical interval switch lines defined by SMPTE RP168 and SMPTE 299M resulting in a transparent switch.

There are many methods to determine when the switch to the stand-by crosspoint should be made ranging from a manual switch to automated detection of signal loss. The key point is that the switch should be vertically accurate, so that those outputs that are functioning correctly do not suffer adverse effects.

Once the switch-over has occurred, the Crosspoint module may be removed for normal service. The result is that the impact block for the system is now reduced to that of the input or output modules in the frame. In this example, they both have an impact block of 16.

The N+1 Redundant Crosspoint Implementation

SMPTE 144th Technical Conference and Exhibition - 11 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

While the N-on-1 approach does not provide an impact block of 1, it does provide a very cost effective way to implement an extremely robust facility. The Mean-Time-To-Repair ( MTTR ) of such a router is extremely small. In a semi-automatic mode, a plant alarm can be used to signal an operator to activate the change over switch, a process which can be executed form a remote location. In a fully automatic mode, MTTR can be as short as a few frames of video.

This approach has definite speed and cost advantages over the common practice used today. Midnight drives to the facility, rummaging around in a spares cabinet for modules, and patching around critical paths prior to swapping modules are all unnecessary actions.

The Hidden Economic Advantage of N-on-1

Note that with additional output busses represented by the dotted lines shown in figure 11, and the use of a 3 x 1 switch on the output modules, two, 256 x 256 square, crosspoint modules can be used in a 1-on-1 redundant scheme. Since the redundant backup eliminates the crosspoint module from impact block consideration, the crosspoint can now be made as big as reasonably possible. This ensures that the router will have the highest possible reliability at the lowest possible cost. Furthermore, this architecture provides the ability to incorporate new, larger crosspoint chips as they become available prolonging the lifetime of the installed product in the facility and reducing the future cost of the product. In 1In 1256x256InputModuleIn 16In 256Out 1Out 128Out 1OutputModules1-8Out 128PrimaryCrosspointOut 129Out 256EmptyCrosspointIn 241In 1256x256InputModuleOut 1Out 128Out 129OutputModules9-16Out 256RedundantCrosspointOut 129In 256Out 256In 256 Figure 11: One Extra Bus Enables Redundant 256 x 256 XPT Cards

SMPTE 144th Technical Conference and Exhibition - 12 -

October 23-26, 2002 Pasadena, CA

Reliable Plant Implementations

Progress and Pragmatism

A number of methods exist to build large, resilient centralized switching plants. The optimal impact block size for a router depends upon the size of the facility, the type of work done at the facility, and how the facility is configured. It also depends upon the total time it takes to repair a failure, cost of capital and cost of operations. In nearly every case, the N-on-1 crosspoint architecture provides valuable advantages.

Using Patch Bays

Patch bays offer a proven technique that is widely deployed as the first line of defense against failure. One particular attraction of patch is the lack of electronic components. The implication being that patch is more reliable, and hence the best way to back up the electronic switch. Patch bays can still lay claim to this difference, but the advantage is debatable. Electronic components and circuitry have become far more reliable. In fact, they are nearly as reliable as the mechanical surfaces and leaf springs used in patch bays. The cost and size of electronic routers have been dramatically reduced at the same time reliability has increased. Therefore, the cost of patch, as a percentage of the total system, has increased as a method to provide back up to operational capability.

When a failure occurs in a full patch system, a human operator must manually patch across the desired input to output connection. Then, using the patch bay as a debug tool, a number of other signals must be patched around the system to provide continued operation while the facility

engineer troubleshoots the system. This is a time consuming process and prone to human error.

Assuming a single point of failure, routers with redundant crosspoint topologies only need patch to provide protection for an input or output module. In some facilities, mission critical inputs and outputs may occupy only one or two modules, therefore, it is possible to provide patch for just these modules, greatly reducing the cost compared to a system with full input and output patch bays. Consider a router that has K total inputs and M total outputs. Key router inputs ( k ) and outputs ( m ) are cabled through patch to adjacent input and output modules, respectively. An additional input module and output module are added to the frame providing hot standby protection.

If an input module fails, patching each corresponding input of the failed module across to the standby inputs restores the path. Likewise, should an output fail, patching once again restores the path. At this point however, the router must be reconfigured. Using a simple, but special panel, it is possible to create a “Smart Salvo”. This smart salvo re-maps the input addresses associated with the faulty input module, to those of the standby input card. So, even though inputs 1 through 16 have been patched to inputs 241 to 256, for example, the automation and router control systems still respond to the operator as if inputs 1 through 16 are still operational. To simplify the change over procedure even more, a small button panel with “Take” and “Release” keys can be used to execute the Smart Salvo. Each button corresponds to an input module that is cabled to patch. Pushing the button and Take, will re-map all signals of that module. Pushing the same button and Release returns the router to the normal state. Outputs are managed in a similar fashion. Since only a small number of signals are patched, this process is very rapid and less prone to human error. Figure 12 shows this router topology.

SMPTE 144th Technical Conference and Exhibition - 13 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

Patch Cord Patch for k+16 Inputs Standby Input Module, 16

Critical k Inputs K Inputs Critical m Outputs Standby Output Module, 16

M Outputs Patch for m+16 Outputs Patch Cord

Figure 12: Partial Patch with Partial Matrix Backup

The key advantage with an N-on-1 redundant crosspoint router is that crosspoint failure can be repaired with one button push. Then, input or output module failure can be bypassed with a small number of patch cords. With full patch, a number of manual router connections will need to be made. In the case of partial patching, shown in Figure 12, a single router salvo can be executed to simplify the recovery after the patches have been made. N-on-1 redundancy insures that the number of patch connections in either case is very small, typically 16. Repair is then reduced to exchanging modules in the frame with spares. The additional cost of N-on-1 crosspoint

technology is typically much less than the cost of patch bay hardware it replaces. If the cost of lost service during prime time is considered, the cost of N-on-1 crosspoint technology is very inexpensive insurance.

Signal Interleave Techniques

Because N-on-1 technology reduces the impact block of the router to a small number, it is now possible to distribute the outputs for a particular device, or functional area of a facility, across a number of output modules, thereby reducing the impact to plant operations even further. A good example is master control. If four video signals are fed to one station, then one output on each of 4 different modules can be used. If any one of those modules is lost, only one signal is lost to that location. This technique is very widely adopted in industry today.

SMPTE 144th Technical Conference and Exhibition - 14 -

October 23-26, 2002 Pasadena, CA

Progress and Pragmatism

When combined with augmented patching, a very robust facility is created. Additionally, the time to restore full On-Air operations is very short since it is only the time necessary to install one patch cord. After the rest of the augmented patch cords are installed, the router may be repaired on line. Later, at a convenient time, the salvo may be released, and the patch cords removed, restoring normal operation.

2x1 Protection Switching

Redundancy for a router system can be constructed using two identical routers and a number of 2 x 1 switches as shown in Figure 13 below.

In 1In 1Out 1Out 1In MIn MOut NIn 1Out 1Out NIn MOut N

Figure 13: 1-on-1 Full Redundancy

In this topology, two M x N non-blocking routers are used to provide 100% switching backup. The outputs of these routers are then fed to N, 2 x 1 switch elements, each individually controlled. If each of these switch elements were on an individual card or module, and system power and control were effectively managed in a redundant fashion, this would be a very robust system indeed. It also would be exceptionally expensive costing approximately three times as much as the base router. Using N-on-1 crosspoint protection, the number of router outputs requiring 2x1 switches instead of patch can be greatly reduced offering considerable savings, and a high level of automated recovery from error.

SMPTE 144th Technical Conference and Exhibition

- 15 -

October 23-26, 2002 Pasadena, CA

N-on-1 Plant Topology

Progress and Pragmatism

Post production facilities have implemented this topology for a number of years. A number of edit suites, maybe 3 or 4, are built with identical equipment in the suite and in a central equipment room. If one client session runs long, or a production control surface fails, the identical router configuration and control settings can be called up to configure an open suite for the same

purpose. This same model can be applied to central-casting. Assume that 5 separate channels are broadcast from one location. In all likelihood, the critical path hardware associated each channel will be identical. In an N-on-1 redundant topology, one more set of hardware is installed in the main equipment room. In case of failure, it is switched in behind the control surface of the master control room.

The smaller the impact block of the router, the less likely it is that a router failure will affect critical operations, particularly when signal interleave techniques are used to cable the facility. Therefore, the ratio of N to 1 can be quite large and still afford a very low probability of plant failure. Or, stated differently, for most facilities only one additional set of critical path hardware is required.

Characteristics of Reliable, Resilient Routers

There are a few simple rules for evaluating routers for long term reliability.

1) Make sure that all moving parts ( parts with low MTBF ) can be repaired. Fans are the

prime consideration. Avoid fans on circuit board assemblies if at all possible. If an active module must be removed to repair or service a fan, a significant portion of the router may be taken off line. Fans are commonly found on power supply modules, but those modules are typically redundant, so one may be removed for repair without affecting the system.

2) Be sure that adequate cooling exists in the router chassis. Low cost infra-red

thermometers are available that can measure the surface temperature of components

while they are operational in the frame. Ideally, components should be in the range of 40 to 50 degrees C ( 100 – 125 degrees F ) with an ambient air temperature of 68 to 70 degrees F. This allows for 15 to 20 degrees F of thermal rise without accelerating the aging of most semiconductors.

3) Avoid high humidity environments. It is not uncommon for chip components such as

resistors and capacitors to fail from metal migration at their electrodes in high temperature, high humidity environments.

4) Be sure that no active components are captive in the back of the frame. MTTR for these

modules is excessive.

5) Look for regulatory compliance marks, such as CE and UL.

SMPTE 144th Technical Conference and Exhibition - 16 -

October 23-26, 2002 Pasadena, CA

Conclusion

Progress and Pragmatism

It is guaranteed that at some point in time an electronic circuit will fail. When it does, it is

essential that a plan for recovery and repair exists. The high value of data in a centralized routing facility precludes long periods of down time for repairs. But, each plant has a different operating model, and so each plan will be different. These plans may be based on a highly automated system structure and plant specific software, or they may utilize human resources and a well organized, documented set of procedures. A number of techniques for robust plant design have been described, and some, or all, of them may be included in your facility and its recovery plan. And, in every case, the N-on-1 redundant crosspoint architecture has been shown to offer significant benefits in rapid recovery, and reducing MTTR for the system. N-on-1 crosspoint backup provides very affordable insurance against failures by protecting the router module with the largest possible impact block, thereby significantly reducing the impact block of the router. Given the value of commercial air time during any major broadcast, the economic advantage of this novel architecture is clear.

SMPTE 144th Technical Conference and Exhibition - 17 -

October 23-26, 2002 Pasadena, CA

References and Notes:

Progress and Pragmatism

[1] “A Study of Non-Blocking Switching Networks” , Charles Clos, Bell Systems Technical Journal, pg 406-pg 424, March 1953.

[2] “Three-Stage Generalized Connectors”, Richard M. Karp, Report No. UCB/CSD 90/558, January 1990, Computer Science Division, University of California, Berkeley.

[3] “Three Stage Router for Broadcast Application”, Charles Meyer, US Patent #6,430,179, August 2002.

[4] Foreign and Domestic Patents Pending.

SMPTE 144th Technical Conference and Exhibition - 18 -