Can open-loop control algorithms solve the data center switching problem?

: There is a world-wide push to create the next generation all-optical transmission and switching technologies for exascale data centers. In this paper we focus on the switching fabrics. Many diﬀerent types of 2D architectures are being explored including MEMS/waveguides and semiconductor optical ampliﬁers. However, these tend to suﬀer from high, path dependent losses and crosstalk issues. The technologies with the best optical properties demonstrated to date in large fabrics (>100 ports) are 3D MEMS beam steering approaches. These have low average insertion losses, and equally important, a narrow loss distribution. However, 3D MEMS fabrics are generally dismissed from serious consideration for this application because of their slow switching speeds ( ∼ few milliseconds) and costs ($100/port). In this paper we show how novel feedforward open loop controls can solve both problems by improving switching speeds by two orders of magnitude and costs by one order of magnitude. With these improvements in hand, we believe 3D MEMS fabrics can become the technology of choice for data centers.


Introduction
The electrical power consumed in exascale data centers is large and growing. The largest data center today uses ∼ 150 MW [1] of power and 100 MW centers are not uncommon [2]. As the world attempts to reduce its carbon footprint and use electrical power more efficiently, the size and number of data centers continues to grow exponentially [3]. In response to this threat, there is a world-wide effort to develop all-optical technologies to replace electronic transmission and switching/routing, two major consumers of power in modern data centers [4,5].
In this paper we focus on the switching/routing aspect of the problem. In the telecom world 20 years ago, there were circuit switches and packet switches. The circuit switches brought connections up for long periods of time and the temporal requirements on those switches were driven only by the desired restoration times, typically 50 milliseconds, the SONET specification [6]. Packet switches were electronic and had switching time requirements of microseconds to nanoseconds. The desired optical switch fabric for today's data center application is a hybrid between these two traditional categories [7].
The typical architecture used today comprises thousands to tens of thousands of racks with multiple servers per rack. The top of rack (ToR) is typically a node in the switching network [8][9][10]. This drives the requirement of thousands of ports per fabric. As an example, the DARPA PIPES specification is 1000 ports [11]. The traffic between racks consists of packets of varying lengths, from nanoseconds to days. The longest of these packets are "elephant flows" and while they make up a small percentage of the total number of flows, they can contribute a relatively large portion of the total data traffic volume [12,13]. The desire is to offload this traffic to an all-optical network which can route and carry the information much more efficiently than can be done electronically [2,4,5,7]. While there is no hard temporal specification, faster optical switching leads to a larger portion of flows being routed through the all-optical network and greater energy savings. The DARPA PIPES program, for example, has a specification for a 10 µs switching time [11].
In the envisioned architecture, the switching fabric would be surrounded by optical elements of varying powers ranging from short reach (SR), long reach multimode (LRM), long reach (LR), extended reach (ER, ZR). These elements are high speed and expensive with a steep financial penalty for more optical power. This drives the need for low average losses in the fabric and equally importantly, a narrow distribution of loss for all connections. High speed optical systems suffer from limited dynamic range and a path length dependent loss is difficult and expensive to deal with. This drives the DARPA PIPES specification of 3 dB insertion loss [11]. Finally, cost drives everything in the data center. In the telecom world of yore, per port costs of $100 -$1000 were acceptable; whereas the target cost today is closer to $10/port [4].
3D MEMS switching architectures are a mature, commercially available technology, developed almost 20 years ago, which meet all the relevant optical specifications. Systems have been built with an average loss of 1.3 dB for all connections in a 238x238 port system with a narrow loss distribution (+/-.3 dB), low polarization dependent loss of <40 dB, a spectral bandwidth of >100 nm, low (unmeasurable) crosstalk between channels, fully optically transparent supporting both classically and quantum coherent transmission [14][15][16]. This same technology has been expanded to create switching fabrics with 1,100 inputs and outputs [17]. The dual Achilles Heals for this technology have been speed and cost. Traditionally, 3D MEMS devices have had millisecond switching times [18][19][20][21] and cost ∼ $100/port [4]. A major contributor to the high cost are the electronics to control each mirror. The typical 3D MEMS fabric is an analog device, requiring fast analog control of high voltages on the order of 100 V. This is usually done with the combination of a high speed DAC and a high speed, high voltage linear amplifier (HVamp); however, achieving high enough slew rates is a challenge. Each port in a 3D MEMS fabric requires 8 of these DAC/HVamp channels. At ∼ $10/channel these systems cost $80/port, well above the $10/port target. Therefore, any solution must address the electronics cost per channel.
In this paper we discuss how 3D MEMS beam steering technology augmented with modern open loop controls can improve the step and settle response times by orders of magnitude. The theory for these methods was laid out by Imboden et al. [22] who demonstrates the feedforward double step drive in capacitive, electromagnetic, and thermally actuated MEMS mirrors. This technique improves the settling time of a high quality factor (Q) system from ∼ T 0 Q (ringdown time) to T 0 /2, where T 0 is the period associated with the system's natural frequency, ω 0 = 2π/T 0 . Here we expand on that work by experimentally demonstrating the previously predicted overdrive methods using a LambdaRouter electrostatic tip-tilt mirror. The LambdaRouter is a well known device with good optical properties, allowing us to focus on improving the switching speed [14,15,17,23]. We then use simulations to explore the impact of lower Q values on the response. From a combination of experimental and simulated results, we discuss how a micromirror could achieve a 10 µs switching time. We also demonstrate the use of pulse width modulcation (PWM) methods to drive these devices, which can reduce the control costs by a significant margin. We have previously shown how one can use PWM to control MEMS in an analog way, which allows the DAC/HVamp to be replaced with a simple switching circuit [24]. Timing replaces the need for analog control. Our thesis is that modern control theory approaches for driving 3D MEMS devices can eliminate the two bottlenecks that limit their general application in data centers. We believe control theory, appropriately applied to a mature, commercially available technology, can solve the data center switching problem. (c) Colorized SEM image of a LambdaRouter mirror [23]. (d) Schematic of electrostatic tip-tilt mirror. Applying a voltage to V 1 will accelerate the mirror in one direction, whereas applying a voltage to V 2 will deflect it in the other direction.

Theory
In many cases, MEMS mirrors can be modeled as an underdamped system defined by the linear differential equation where I, γ, k t , τ(t), and θ are the system's moment of inertia, damping coefficient, torsional spring coefficient, applied torque and the mirror's angle respectively. Such a device will ring when driven by a step input and the settling time is related to the Q of the device. However, we have shown that by applying precisely timed steps or pulses to a high Q system, one can improve the settle time by orders of magnitude [22]. There are three open loop techniques discussed in this work: (1) double step (DS), (2) unipoloar overdrive (UOD) and (3) bipolar overdrive (BOD), all of which can replace a single step (SS). Intuitively, all three work off the idea that an underdamped system will overshoot when given a step input. The precisely timed pulses/steps manipulate the mirror's momentum such that it reaches the desired angle, θ 0 , with zero velocity. At that point, the torque required to counter the restoring spring torque (τ 0 = k t θ 0 ) is applied resulting in zero motion and zero net torque on the mirror. The double step, illustrated in Fig. 1(a), consists of a step input of torque τ 1 for time t 1 , at which point the torque τ 0 is applied. A full derivation for the timing t 1 and torque τ 1 are presented in Imboden et al. [22]. They are originally derived for a translational system but the solution is general and applies to any linear, second order differential equation, such as Eq. (1). As the Q increases, t 1 and τ 1 approach T 0 /2 and τ 0 /2. It should be noted that for an electrostatic mirror like the LambdaRouter, τ ∼ V 2 for small displacements, where τ is the torque and V is the applied voltage. The overdrive method, illustrated in Fig. 1(b), consists of a step input τ max until time t 1 , then a step to τ min until time t 2 , at which point the final torque τ 0 is applied. For brevity, we limit the discussion to only two versions of the overdrive (1) unipolar overdrive where τ min is defined as 0 and (2) bipolar overdrive where τ min is defined as −τ max . Similiarly, the derivations for t 1 and t 2 for both overdrive methods are provided by Imboden et al. [22]. The analytic derivation assumes zero damping which significantly simplifies the derivation and solution. This is a good assumption when the Q factor is high (Q > 100). These three drive techniques, double step and the two overdrives, can be implemented with analog controls or digital controls using PWM [24,25]. Using PWM, the torque discussed above is related to the PWM duty cycle. For small displacements, PWM linearizes the forcing function with respect to the duty cycle such that τ ∼ R DC τ PW M where R DC is the duty cycle (0-1) and τ PW M is the max torque being switched on and off [24]. There are twelve different drives discussed throughout this work and summarized in Table 1. Ten of the drives are analog and two are PWM. The normalized t 2 timing for a high Q system is shown in Table 1. The bipolar drive is not directly tested with the PWM method. However, previous work has shown that given a high enough frequency, PWM can be directly substituted for analog control and implies that the bipolar drive could be implemented with PWM [24]. Likewise, the t 2 values are based on the analog driving condition, but since PWM and analog are interchangeable given a high enough frequency, we assume the same analytic solution for both.

Experimental setup and device
This work focuses specifically on electrostatic mirrors since they are currently the most popular micromirror used by commercial optical switches as they are relatively fast and low power [4,19]. We use the same two-axis gimbal design used in the LambdaRouter mirror array, shown in Fig.  1(c) and a simplified schematic of the setup is shown in Fig. 1(d). To generate the analog drives we use a combination of a pulse generator, summing amplifiers and high speed, HV amplifiers.
To generate the PWM signal, we modulate the signal from the summing amplifier prior to the HV amplifier. Although we use expensive components for the PWM drive in this work, these can be replaced with a relatively inexpensive controller and HV switching circuit [24,25]. Generating a voltage difference (V 1 ) between the mirror and pad1 produces an electrostatic force and subsequently a torque on the mirror, tilting it in one direction. Similarly, applying a voltage difference (V 2 ) between the mirror and pad2 the mirror produce a torque in the other direction. We use two identical mirrors for the experiment. The first mirror uses the inner gimbal and experimentally demonstrates the double step and unipolar overdrive methods. The step response for a system defined by Eq. (1) is known. By fitting the response of our system to this known response, we can determine the natural frequency and Q factor. The first mirror has a natural frequency of 834 Hz (T 0 = 1.2 ms) and a Q factor of 2.48, well below the condition for which the previously derived overdrive theory is valid. The double step and the unipolar overdrive do not require negative torques, therefore they can be implemented using only pad1. The first mirror broke during the experiment so a second mirror was used for the bipolar overdrive. Unfortunately, the inner gimbal of the second mirror does not work so the outer gimbal is used. Due to the changes from the inner to the outer gimbal, the second mirror has slightly different dynamics with a natural frequency of 522 Hz (T 0 = 1.9 ms) and a Q factor of 1.27, again well below 100. Throughout this work the timing is normalized with respect to T 0 so the difference in frequency is not important, however the difference in Q does have a noticeable effect. The bipolar overdrive requires both positive and negative torques which is not possible with a single electrode. Pad2 is implemented to provide the negative torque required for the bipolar drive. We experimentally demonstrate all twelve open loop methods and Table 1 labels which mirror is used by specifying whether the inner or outer gimabal is used. To determine the optimal timing, we start with the theoretical t 1 and t 2 (τ 1 for double step), then iterate the drive parameters (i.e. t 1 , t 2 , τ 1 ) until we minimize the mirror's ringing.

Simulations
We use Euler's method to numerically solve the mirrors' responses to the overdrive conditions using phase plane trajectories [26,27], assuming the mirror's motion is defined by Eq. (1). We start by solving the τ max step response given the initial state, zero angle and zero velocity. Instead of plotting angle versus time however, we plot angle versus velocity, giving us the phase plane trajectory for τ max . We then simulate the τ min step in a similar way except that we start the system at the final state (θ 0 and zero velocity) and simulate backwards in time. Plotting the τ max and τ min trajectories together, we can find where the two curves cross and calculate the timing t 1 and t 2 necessary for the overdrive. Using this approach, we can think of moving the system from the initial state to the final state with our controls. We then verify the timing by simulating the overdrive response with τ max , τ min , and τ 0 . The main benefit of the simulation is that it provides insight into using the overdrive methods on systems with non-zero γ values. When simulating the conditions in Table 1, we use the same ω 0 and Q factor for the respective mirrors. As shown in Table1, there are no simulations for the double step or the two PWM methods. The reasons is that double step already has an analytic solution so the simulation does not provide additional insight. Similarly, we have previously shown a direct correlation between analog and PWM control, so we assume the same timing applies [24].

Analog control
The experimental responses for the conditions in Table 1  Normalized Response (A.U.)  Fig. (2a). d) Voltages corresponding to the 9x and 4x responses in Fig. (2b). The solid and dashed lines represent pad1 and pad2 voltages respectively. Pad2 voltages are shown as negative to more easily distinguish from the pad1 voltages and to better represent the torques acting on the mirror.
presented here, V 0 is 50 V and θ 0 is 5°optical. The single step response for the first mirror, Fig. 2(a), settles to within 5% of the final value at t = 2.2T 0 . The results in Figs. 2(a) and 2(c) demonstrate not only the improvement in performance but also the simplicity of these drive techniques. The second mirror investigates the bipolar drive and similarly, the responses are shown in Fig. 2(b). We show the SS response, which settles at t = 1.2T 0 , for comparison to the first mirror. The difference between the two SS responses is consistent with the difference in the Q factors. The driving voltages for BOD 4x and 9x are shown in Fig. 2(d), the rest of the drives are omitted to simplify the plot. Here the solid lines represent the voltages applied to pad1 and the dashed lines are the voltages to pad2. The pad2 voltages have been plotted as negative voltages to more easily distinguish the different curves and better represent the torques acting on the mirror. This plot demonstrates the improvement in speed over single step, with t 2 for BOD 9x at 0.11T 0 . One consequence of increasing the torque and speed of the pulses is that we start to excite a higher order mode of the mirror, shown in the BOD 9x response. Imboden et al. [22] showed that for the two step drive, adding a second step into the pulses can suppress such higher order modes. It is clear from Fig. 2 that both overdrive methods have a speed advantage over the single step and double step. What is less obvious from Fig. 2 is the comparison between the unipolar and bipolar drives. If we continue to increase the τ max /τ 0 the bipolar overdrive will continue to settle quicker; whereas, the analytic solution shows that for a high Q system the unipolar overdrive is limited to 0.25T 0 as the braking torque is limited to that of the restoring torque of the spring [22]. show that the Q has larger effect on t 1 than on t 2 , with the discrepancy between the experimental and theoretical t 1 greater than that of t 2 for all V max /V 0 values. The disagreement between the data and theory is due to the high Q factor assumption discussed earlier. We see that the simulations, which account for the Q of the device, have good agreement with the experimental findings.
Using the simulation, we further investigate the effect of damping and the overshoot error due to errors in t 1 and t 2 for the bipolar overdrive. Fig. 3(c) plots the ratio of a damped versus undamped t 2 value for three different forcing conditions. This plot essentially quantifies the accuracy of an infinite Q assumption, but more importantly shows the effect of damping on the final settling time. At higher values of Q (≥ 100), the assumption is good with the simulated and theoretical timings agreeing. As the Q value decreases the timings diverge such that a critically damped system (Q = 0.5) will settle 34%, 13%, and 4% slower than an undamped system driven by a bipolar overdrive with τ max /τ 0 = 2, 4, and 10 respectively. Fig. 3(d) investigates the effect of timing error, t err or , on the angular error, θ err or , for the same forcing conditions as Fig.  3(c). Here we define t err or as the deviation from the optimal t 1 and t 2 timing and θ err or as the normalized overshoot of the system ( θ ma x −θ 0 θ 0 ). We use a Q factor of 100 for this study. As we expect, the larger the timing error, the larger the error in the response. Similarly, higher torques allow the mirror to move quicker, but at the expense of more error for a given timing error. For example, at t err or /T 0 = 0.001, the BOD 2x drive will have less than 2% error, whereas the BOD 10x drive has ∼6% error. Fig. 3(e) explores the effect of the Q factor on θ err or . For this plot we use a BOD 10x drive. With more damping, the system is less sensitive to error in the timing; however, Fig. 3(c) shows that this also means a slower settling time.
Throughout this work we have normalized the settling time and displacements, therefore generalizing the results for other linear systems. One system in particular that is of interest is a micromirror capable of switching within 10 µs [11], and Figs. 3(b)-3(d) help build a roadmap for creating such a device. An electrostatic MEMS mirror with a Q > 3 driven by a bipolar overdrive with an τ max /τ 0 of 10 (V max /V 0 = 3.16) will settle within 0.1T 0 . Therefore, a micromirror with a natural frequency of 10 kHz can switch in under 10 µs. These kinds of frequencies are relatively common for a scanning electrostatic mirror [28,29]. Figure 3(d) shows that in order to keep the error under 1% (θ err or = 0.01) one needs timing precision of 15 ns (0.00015T 0 ). Again, this is well within the range of modern controllers.

PWM control
Switching time is an important parameter for these mirrors; however, as discussed previously, providing a cheaper method of control is equally important, and PWM is a more cost effective method. Each one of the analog drives in Table 1 can be implemented as a PWM drive by varying the duty cycle to control the torque. Fig. 4a and 4b illustrate how the analog drives for DS and UOD 1x drives are converted into a PWM signal. For the double step, instead of using a τ 0 /2 step, the PWM version uses a 50% duty cycle signal. The UOD 1x drive is more trivial because it is stepping up and down between 0% and 100% duty cycle. However, this same technique can be used to make smaller steps by adjusting the 100% duty cycle pulses with different duty cycles. Fig. 4(c) compares the response of the PWM double step and PWM 1x unipolar overdrive with their analog counter parts. The same analog response from Fig. 2(a) is included for comparison. Instead of controlling the electrostatic force by adjusting the voltage level, we adjust the duty cycle of a PWM signal. For example the drive for PWM DS, consists of a 68% duty cycle signal until t = 0.51T 0 at which point the duty cycle in increased to 100%. Both of these parameters agree with the theoretical value within 3%. It should be noted that the PWM method is implemented with no discernible performance difference compared to the analog method.

Conclusion
As data traffic continues to grow exponentially, engineers and scientists around the world work towards the next generation of all-optical transmission and switching technology. 3D MEMS beam steering is a mature, commercially viable option with some of the best optical properties to date. However, MEMS micromirrors are typically dismissed for their slow switching speed and high cost per port. In this work we have demonstrated that a modern, simple to use, open loop controls technique can improve the switching time of an existing commercial mirror by a factor of 10. These controls in conjunction with PWM provides a fast, cheap method to control electrostatic micromirrors. We further discussed how a mirror with a 10 kHz natural frequency could achieve a 10 µs switching time. We believe that an appropriately designed 3D MEMS switch with modern controls can be competitive for the optical switching technology of the future.