Episode 132: 1.3 Millimeters (Hubble Mirror Flaw)

On the latest episode of The Space Above Us, we learn about the major systems of the Hubble Space Telescope and take a deep dive on what was wrong with the mirror, how it happened, and how it wasn’t caught.

Episode Audio #

Photos #

Transcript #

NOTE: This transcript was made by me just copying and pasting the script that I read to make the podcast. I often tweak the phrasing on the fly and then forget to update the script, so this is not guaranteed to align perfectly with the episode audio, but it should be pretty close. Also, since these are really only intended to be read by myself, I might use some funky punctuation to help remind myself how I want a sentence to flow, so don’t look to these as a grammar reference. If you notice any egregious transcription errors or notes to myself that I neglected to remove, feel free to let me know and I’ll fix it.

---

Hello, and welcome to The Space Above Us. Episode 132: 1.3 millimeters

Last time, we talked about the 15th flight of Space Shuttle Columbia: STS-58. The second flight of the Space Life Sciences payload delivered a wealth of new data about how living systems adapt to microgravity, using space to shed light onto complex biology. We also talked to some Russians, learned some leadership lessons from John Blaha, and kept our piloting skills sharp.

The next shuttle flight is STS-61, the first servicing mission to the Hubble Space Telescope, but as I mentioned last time, we’re going to take a quick detour before following Endeavour uphill. The mission itself will involve four spacewalkers performing five back to back EVAs, surpassing even the Skylab-repair-spacewalks in terms of overall complexity. With so much going on, I didn’t think I was going to be able to do justice to the mission itself, as well as the famous mirror flaw, all in one episode. So while I generally resist the temptation to do this, we’re essentially splitting STS-61 into two episodes. Yes, I’m sure you’re all so upset.

Today we’re going to go on a tour of the observatory, where we’ll learn about all the major components, and how the Hubble works overall. We’re also going to talk about what, exactly, went wrong with the mirror, and we’ll try to get to the bottom of how it happened in the first place. What we won’t be discussing is the early development for the Hubble as well as its deployment on STS-31, which was covered back on Episode 106. So if you really want to be fully up to speed, it might be worth relistening to that episode.

Since we’re going to be spending a lot of time there, in this episode, the next episode and four more later on, let’s get to know the Hubble Space Telescope. At a first glance, the Hubble looks like a big shiny cylinder about 13 meters long, or 43 feet. On closer inspection, the one big cylinder is revealed to actually be two smaller ones. The shorter cylinder in the back is wider, with a diameter of 4.3 meters, and the front cylinder is more narrow, with a diameter of 3.1 meters. As a mental shortcut for imagining the size, think of those big cylindrical tanker trucks that you sometimes see on the highway. The surface of the front cylinder is silvery and crinkled, thanks to the aluminized mylar foil coating the main structure, which assist with thermal control. The surface of the back cylinder is silvery metal, with seams, rivets, and hatches visible throughout. Flanking this central shiny structure are two large brownish orange solar arrays, measuring 2 by 12 meters each. Sticking out of the central structure, perpendicular to the solar arrays, are two black dishes, perched at the end of appendages that for some reason I simply cannot find the length of. These are the high gain antennas used to send science data over TDRSS. The entire surface is riddled with over 68 meters of bright yellow handholds. Emblazoned near the top of the narrow cylinder are red letters that any space nerd would recognize: the NASA worm logo.

Each of these segments plays a central role in the overall operation of the observatory. We’ll start with the easy ones: the solar arrays provide electrical power and the high gain antennas provide communication links. Though there are also some smaller low gain antennas on the back of the telescope, which are typically used for command and control, which requires a lot less bandwidth.

At the front part of the telescope, the narrower cylinder looks like one long tube but is really two parts: the light shield in the front and the forward shell in the back. At the front of the light shield is a large opening, called the aperture, and a similarly large door. The opening allows light to enter the telescope, and the door is there to prevent light from entering when mission operators don’t want it to. For instance, if pointing control was lost and the telescope was about to point towards the sun, or if a shuttle servicing mission was arriving, the door would close. The light shield is actually mostly just an empty tube, with its inside lined with ridges, called baffles, which help to prevent stray light from getting too far into the telescope. If you were to enter the aperture at the front of the telescope and head down into the light shield, you would soon arrive at the forward shell, which is where the secondary mirror is housed, but we’ll get to that in a minute.

Heading back outside, let’s take a look at the wider cylinder on the bottom. Again, this looks like one big cylinder but it’s really broken into two parts: the Equipment Section in the front, and the Support System Module in the back. The Equipment Section is basically just what it sounds like. It’s a series of 12 bays that line the exterior of the telescope, forming a big ring that goes around the central structure. Two of those bays are dedicated to the support structure used to carry the Hubble into orbit, but the other ten bays are filled with various equipment required to operate the spacecraft. Since Hubble was always intended to be serviced by humans, these bays have outward-opening doors, allowing easy access by astronauts. Some of the equipment here includes reaction wheels (which are used to point the telescope), data management hardware, communications hardware, command and data hardware, stuff like that.

Moving all the way to the back is the Support System Module, which houses the core of the telescope. Around the upper edges are more doors, which provide access to the radial science instruments. These instruments are positioned alongside the beam of focused light, and have light bounced over to them, hence “radial” instruments. In the bottom, behind large access doors, are the four axial science instruments, so-called because they align with the axis of the telescope.

To better understand how the telescope operates, let’s follow the path of a photon from a distant star. After traveling for sometimes billions of light years, the photon suddenly finds itself entering the aperture of the Hubble Space Telescope. If it doesn’t strike the back of the secondary mirror, or the thin spindles that hold it in place, or the baffles lining the interior of the light shield and forward shell, the photon will make its way back to the heart of the observatory: the primary mirror. This is mirror is gigantic. It’s 2.4 meters across, just shy of 8 feet, and shaped like a donut, leaving a 60 centimeter hole in the center (about 2 feet). The photon hits this mirror and is reflected. If it was a flat mirror, it would just bounce right back out, but instead, the mirror is curved, so the photon gets bounced up towards the secondary mirror. The secondary mirror is significantly smaller, only 30 centimeters across, and it too is curved. The reflected light is focused closer together, forming a sort of cone, with the base a the secondary mirror, and the point entering the hole in the primary mirror, where it can then be distributed to the scientific instruments. By bouncing the light back and forth in this way, the telescope can perform equivalently to a much longer telescope. I haven’t seen this reason listed, but I would also guess that it’s much easier, logistically, to receive the focused light behind the primary mirror, where all the scientific instruments can be grouped together, out of the path of the incoming light.

So just in short: light enters the aperture, bounces off the primary mirror, which directs it onto the smaller secondary mirror, which bounces it back down through the hole in the primary mirror, where the focused light can be sent to various radial and axial scientific instruments.

Also, fun fact: if you’ve ever looked at a photo and noticed those four big spikes that come out of every star, those are actually interference patterns introduced by the presence of the support structure holding the secondary mirror in place.

After years of planning, fabrication, assembly, and testing, Hubble was finally lofted into its 600 kilometer high orbit in the back of Space Shuttle Discovery, on STS-31. Again, we talked about STS-31 back on Episode 106, so for now, it suffices to say that the deployment was a success, and as Discovery returned to Earth, scientists got to work on calibrating the telescope.

Nobody expected the telescope to return crystal clear images right off the bat, that’s just not how it works. Between assembly, the rocky ride to orbit, the change from a 1-G environment to a 0-G environment, and massive swings in temperature, something was bound to be slightly out of position. But when I say “slightly”, I really mean “slightly”. The structure holding the secondary mirror in place was made out of carbon fiber, to reduce thermal expansion and contraction, since its position relative to the primary mirror needed to be controlled to within 2.5 millionths of a meter, which is about 100 times smaller than the width of a period at the end of a sentence. So yeah, something was going to be off.

But that’s ok, the telescope was planned with that in mind. It was possible to slightly move the secondary mirror. Scientists on the ground could move it a little closer, a little further away, move it side to side, or even give it a little tilt. These would all be excessively small motions, but it would be enough to focus the telescope and get everything working. But the thing was.. it wasn’t working. Try as they might, the scientists could simply not get the telescope to focus. But what was weird was that the problem they were seeing wasn’t one that they expected. If the mirror was off the light axis, they would expect to see cones with sharp points. If the mirror was too close or far away, they would have expected moving it back and forth to solve the problem. Instead, when inspecting images of distant stars, they were seeing concentric rings: a clear indication of something called spherical aberration. But that simply can’t be right. The only way for there to be spherical aberration would be if– actually, let’s take a quick second to learn about spherical aberration.

First, let’s clear one part up: aberration. This just means that there is some sort of deviation from what was expected. In the context of optics, you can more or less replace the word in your head with “flaw”.

So what about “spherical”? Imagine that you took a metal ball and pressed it partway into a block of clay. When you took the ball out, you’d see a divot left in the clay. That divot would only be a piece of a sphere but it would have the same curvature. If you were to then somehow coat that divot in something shiny, you would have a spherical mirror on your hands. Spherical mirrors are nice because they’re easy to make, easy to verify, and they’re just simpler overall. However, if your goal is to make a telescope, they have some problems.

Telescopes are devices to take light from a wide area and focus it down to a small area. With the same photons over less area, the resulting image will be brighter, and you’ll be able to see stuff that you normally couldn’t. And by tweaking the sizes and positions of optical elements, you can also magnify the image, allowing far away stuff to seem closer. But there’s a key word there: focus. When light comes from a very far away source, the light rays are essentially parallel, and we call this columnated light. If you were to take your fancy new spherical mirror and shine some columnated light at it the light rays would bounce off the mirror, but would not all converge on one spot. Instead, it just sort of gets smeared around somewhere near the middle, which is no good for making a clear image, which is no good for astronomy.

If you remember your elementary geometry, you might know where we’re going next. If you were to replace that spheroid section with a paraboloid section, we’d get much nicer results. No matter where on the paraboloid your columnated light hits, it’s going to bounce off the mirror and fly through a point above the center point: the focus. So if you put your instruments at the focus of a parabolic reflector, all the light hitting the mirror will hit your instruments, which is exactly what we want. The spherical mirror smears out the image, the parabolic mirror collects it all in one point.

Well, for reasons that I don’t quite understand, since optics is hard, the mirrors on the Hubble were both designed as hyperboloids, that is, they took a hyperbola, spun it around in 3D, and the resulting shape is what they wanted the mirror to be.

This is all a very long way of saying that there exists a class of shapes that will collect light and focus it all into one point. But if you don’t build the shape just right, different parts of the mirror aren’t going to reflect light exactly where you want it, and there’s gonna be trouble. So with that ominous foreshadowing out of the way, let’s learn how they made gigantic mirrors in the early 1980s. Specifically, we’re going to find out how Perkin-Elmer, the contractor for the job, made the Hubble mirror.

To start, Perkin-Elmer ordered a big giant slab of glass from Corning, who make all sorts of specialty glass products. It’s high quality glass, but it’s just a big slab, so Perkin-Elmer had some work to do. The process, in short, will be to grind the glass into the right shape, and then carefully coat it in aluminum so that it’s nice and shiny. But as you might expect, the devil’s in the details. Grinding the mirror was a painstaking and extremely time consuming process. Even with a computer to help out, the hours were long and the work required unending focus and concentration. After a round of grinding, the mirror would be measured to see how the overall shape was evolving. Armed with this new information, the computer would be updated, and the grinding would continue. Continue that process long enough, and you’ll arrive at the desired shape.

But, you might be wondering, how do you know if it’s the right shape? The tolerances on the mirror’s final shape are so small that measuring it becomes a severe challenge. You can’t just take a ruler to the thing or eyeball it. But lucky for us, even the rough version of the mirror is precise enough that we can use a handy bit of light physics to assess the overall progress: interferometry.

I’m not going to teach you interferometry today. It’s too big to just throw into an episode, and while I can hear my math and physics mentor complaining that he already taught me all about it, I don’t always understand it super well. But I can tell you the basics. In short, interferometry uses the wavelike properties of light to very precisely measure something.

Visualize in your mind’s eye what happens when you drop a ball in some water. The ball displaces the water, creating waves which radiate away in all directions along the surface. Now what happens if you drop a second ball nearby at the same time? You’ll end up with two sets of waves. When the two sets of waves run into each other, things start to get complicated. If you look carefully, you’ll notice that sometimes the waves combine and make bigger waves, and sometimes they cancel each other out and the water stays flat. In more scientific terms, the waves are undergoing constructive and destructive interference. And just like the water waves, light waves do the same thing. In normal life you don’t really see this effect with light very often since there’s a ton of light flying around in all sorts of directions at all sorts of wavelengths. But if you control conditions a bit, the examination of these interference patterns, or interferometry, can be an incredibly useful tool.

The simplified version of how the mirror is measured is that laser light is beamed down onto the mirror. It reflects back and is collected in such a way that it creates interference patterns. By examining these interference patterns, the optical technicians can learn about the shape of the mirror. I’m tempted to try to go into more detail with this, but this episode already has a lot to take in without me trying to describe light physics in an audio-only medium. The important thing is that by using special techniques, it’s possible to shine light on the mirror and generate a specific pattern which tells you the state of the mirror.

But we have another problem to solve. If you were to do this trick with a spherical mirror, the resulting patterns would be a nice clean and orderly set of horizontal lines. But we don’t have a spherical mirror, we have a hyperbolic mirror. It would still be possible to interpret the resulting pattern, but it would require more effort and introduce more chances for mistakes. It would be great if we could instead do this test in a way that made the hyperbolic mirror behave like a spherical mirror. That way we could have a nice unambiguous interferogram to study. And as you may have guessed by my leading tone, this is totally possible, by using a device called a null corrector.

Again, I’m going to gloss over some details here, but you can always shoot me an email at jp@thespaceabove.us if you want me to point you in the direction of further reading. Anyway, a null corrector is a device that is inserted into the light beam in order to make our hyperbolic mirror give test results as if it were a spherical mirror. The upshot of all this is that if we use a null corrector, even with a non-spherical mirror, as long as our interferograms show straight horizontal lines, we know everything is great. Perkin-Elmer knew that the scale and tolerances of the Hubble mirror were such that their usual null corrector was not going to be sufficient, so they developed a new design called a Reflective Null Corrector. It’s not really important to know all the physical details of the Reflective Null Corrector, just know that it’s a device about the size of a mini-fridge that contains a few mirrors and lenses, and allows us to test in the way that we want.

So after grinding and polishing the mirror for a while, it would be wheeled over to this reflective null corrector, an interferogram would be generated, the optical team would analyze it, and grinding would continue. Eventually the final shape was achieved, the mirror was coated in aluminum, the null corrector once again confirmed that it was correct, and it was shipped off for integration with the overall telescope. This was in May of 1981, right after the first flight of the shuttle.

Nine years later, after all sorts of delays both in the Hubble program and the shuttle program, the 35th flight of the space shuttle carried the telescope, and its gigantic mirror, into space and scientists got to work on calibrating it. Two months later, they discovered a problem.

As I mentioned earlier, they tried adjusting the optics to account for minor, but expected, imperfections, and nothing was working. I can only imagine the cold sense of dread that must have fallen over the project scientists when they were forced to admit that the problem was being caused by spherical aberration. And the spherical aberration they were seeing could only mean one thing: the primary mirror was the wrong shape, and no motion of the secondary mirror could fix it.

But how could this be true? The mirror had been tested hundreds of times during and after manufacturing and while no mirror could be completely flawless, it was easily within the needs of the mission. Had something damaged it during assembly? Had it warped in the extreme temperatures of space? Had the transition to microgravity not been properly accounted for?

An investigation was immediately launched which soon got to work gathering as much information as possible from Perkin-Elmer, but nine years is a long time. People move on, people forget, people die, things get lost. Heck, Perkin-Elmer didn’t even really exist anymore, it had been bought by another company years ago. But the investigators did what they could with the information they had and suspicion soon fell onto one artifact. The one piece of this puzzle that everything else depended on: the reflective null corrector. If the reflective null corrector was wrong then while that would launch its own raft of questions, the immediate facts would at least make sense. If they could inspect this device, the current mystery could be solved. Amazingly, after all these years, the null corrector still existed, and it was exactly where it was left nine years ago.

Using extreme care to not disrupt this suddenly all-important instrument, the investigative team were able to use the reflective null corrector to recreate some earlier tests and confirm that it was in the same state now as when it was used when measuring the Hubble’s mirrors. This meant that whatever they could learn about it now should apply to what went wrong then. And what they learned now.. was that the reflective null corrector.. was wrong. A single lens was 1.3 millimeters out of position. 15 years of effort, over a billion dollars, state of the art instruments, international collaboration, and a space shuttle flight, and it all came down to this one lens being 1.3 millimeters out of place. Go grab a book, pull together 13 pages, and look at it edge-on. That’s roughly the size of the error we’re talking about here. But that 1.3 millimeters made all the difference because this instrument, the reflective null corrector, was the source of truth. It was the instrument that told everyone if the mirror was the right shape. But since it was wrong, that meant that the Hubble mirror was very very precisely ground into an incorrect shape. It would be like very carefully building a house using a measuring tape that told you 10 inches was really 15, and then wondering why you bumped your head when you walked through the door.

You might be imagining the mirror with weird ripples and lumps in it right now, but no, it was still a nearly perfect mirror. It was just very very very slightly too flat. If we start in the center and work our way out, the amount of incorrect flatness gets bigger and bigger until we reach the edge and find that the mirror is too flat by 2200 nanometers. 2.2 microns. 0.00008661417 inches. I tried to find a good scale comparison for this and felt like I’d fallen into a Terry Pratchett book. The size of the error was about half the diameter of spider silk, or just a little smaller than a particle of smoke, or two-and-a-half times the size of a typical red blood cell.

OK, but how? How could this mistake have possibly been made? As usual, it’s important to have some empathy here and put ourselves in the shoes in the people who made the blunder. Not necessarily sympathy, though maybe some of that too, empathy. In 1980, NASA was a long way from the endless streams of money that supported the Apollo program. And they were also suffering from a Congress that was sick of being asked to kick in more than expected for programs that were in trouble. So the result was an agency, and thus a project, and thus a contractor, that was out of money and out of time. The mirror needed to be finished as soon as possible with no extra resources and anything that got in the way was viewed with suspicion.

That’s factor number one. Factor number two is that some of the people involved were, to be blunt, arrogant and difficult. The mirror polishing team distrusted outsiders and made independent verification difficult. They wanted to be left alone to their work, which they assured the world they were the experts in.

Another factor was that this work was being done in a facility that was also doing classified work just down the hall, which limited the number of people who were allowed access to the site in the first place.

Yet another factor was that there was a separation between the experts who were designing the process and the workers who were actually grinding the glass. Mistakes were made that would’ve grabbed the attention of any optics expert, but passed unnoticed by the people who on hand at the time.

And lastly, for our discussion at least, was siloed communication. In a pattern that will seem all too similar to another aerospace disaster from the 1980s, there were clear warning signs that something was wrong. But these signs were either not recognized, not communicated up the chain, or not well understood.

When you combine all of these factors together you have an on-the-ground workforce that is largely shielded from scrutiny, failed to communicate between designer and doer, and who were working grueling hours under immense pressure to finish the project. The conditions were ripe for something to go wrong.

OK, I think we can understand what the environment was like at Perkin-Elmer at this time. We don’t have to accept it and we don’t have to condone it, but we can understand it. Now that we do, let’s learn how this lens came to be out of position.

The reflective null corrector had actually been built for another project, sort of. As part of the process of winning the contract for the Hubble mirror, Perkin-Elmer made a smaller mirror that used the same reflective null corrector. The project demonstrated that they were capable of doing the highly technical work and that their innovative new tool worked as expected. But after they got the contract and began work on the larger mirror, the null corrector had to be updated to match the new dimensions. The lenses and mirrors inside would have to be moved around.

Much like the primary mirror itself, with an instrument this precise you don’t just take out a ruler and start moving things. We’re actually going to use interferometry again. Analysis determined the required dimensions and special rods were constructed out of a material that undergoes very little thermal expansion and contraction. They were then independently verified to be the correct lengths. These “metering rods”, as they’re called, were then loaded into the null corrector. The ends of the rods were made to be reflective, so that a laser could be bounced off of them and interferometry could be used to precisely position them. Again, there’s a lot going on here so I’m going to restate that: in order to correctly position the lenses in the null corrector, special rods were built with the precise lengths required, and the rods were then inserted into the apparatus. Then lasers were bounced off of their shiny ends, which allowed technicians to place the rods with great precision, and the lenses were placed based on the positions of the metering rods.

The rods were the correct length and they were installed correctly. But when the time came to bounce the laser light off of them, something went wrong. In order to actually insert the rods, they had to be fitted with special caps on the ends. I believe this is just so it there’d be some place to grab onto them and it could be attached to the apparatus. In the center of the cap was a hole so that the laser could get to the shiny end of the rod. And just to be sure, the end cap was coated with a non-reflective coating. So even if the laser was off by a little bit, it wouldn’t reflect off the end-cap. Well……. almost. There was just a little bit of anti-reflective coating missing. So when the test was done, the laser actually bounced off of the end-cap, which was (wait for it) .. 1.3 millimeters out of position.

With that, we’ve reached the bottom. I mean, I guess we could pursue how the anti-reflective coating came to be flaked off, but I don’t have that answer and I doubt there would be a paper trail to support one anyway. The fact is that the flaked off coating allowed the laser to bounce off of the wrong thing, which dictated that the technician move the lens to the wrong place, which meant that the primary mirror was polished to the wrong shape, which means the Hubble wasn’t able to do what it was designed to do.

Well now we have to ask ourselves how this wasn’t caught. How is it possible that such an enormous error, and in this context 2200 nanometers is an enormous error, went undetected all the way through launch? One natural question might be how it wasn’t caught when they performed a full end to end test of the optical telescope assembly. That one’s easy enough to answer: there was no end to end test of the optical telescope assembly. In order to do this, you’d have to build a whole second set of giant precision optics and would basically double the cost of the project.

But tests were done, and there were warning signs. Even in the moment, as the reflective null corrector was being readjusted and the laser bounced of the metering rod end-cap there was a chance to realize this error. When the laser measurement was made, the technician began moving the lens in question. The apparatus was not designed to move the lens as far as he thought he now needed to move it. Rather than wondering “wait, why is the machine literally stopping me from doing this” or checking with the designer of the null corrector or anyone else, he inserted a couple of washers in to serve as spacers. Ultra-precise optics… hardware store washer. It boggles the mind.

And there were chances. In order to precisely align the mirror, the reflective null corrector, and the device that recorded the interferograms, another tool was built: the inverse null corrector. I know that sounds really similar to “reflective null corrector”, the instrument with the problem, but I don’t make the names. The INC’s job was to simulate a perfectly spherical mirror during the alignment process. The INC was used to line everything up successfully completing its job, but at the same time, the resulting interferogram clearly indicated that there was a problem. Instead of nice straight lines, some lines were curved, which is a sure sign of spherical aberration. Astoundingly, the error was attributed to the inverse null corrector, not the reflective null corrector, and there was no further investigation. The reasoning was that the INC was less precise than the RNC, so of course if they were seeing an error now, it must be the less precise instrument.

I can kind of see how this mental trap would happen. The RNC seems to be working great, but when tested with an instrument designed to simulate a perfect mirror, an error appears. It’d be natural to assume that the new instrument was wrong. And with all the schedule and budget pressure, I can see, even if I strongly disagree, how a dedicated investigation might be skipped. What I’m unclear on is it seems like this also wasn’t written up in a report for NASA, or even run by the designers of the reflective null corrector. So I think this is just unambiguously bad. And it wasn’t the only case like this.

Near the end of the overall process, it was important to find the exact center of curvature of the mirror, in a test called the vertex radius test. Somewhat maddeningly for someone trying to tell a coherent story, this test was performed with something called the refractive null corrector. I can’t even just abbreviate that one to reduce confusion. But it doesn’t matter, because all you need to know about this latest null corrector was that it was being used to find the center of curvature of the mirror. In this test, yet another interferogram was made, and just like the INC test, instead of straight lines throughout the mirror, near the edges there were big swoopy lines, a clear sign of spherical aberration.

This was shrugged off in a similar manner to the INC. The point of this test was not to judge the the quality of the mirror, and it was being performed with a less precise instrument anyway. In fact, the refractive null corrector was actually what the reflective null correct was supposed to be improving on. Instead of mirrors it used lenses, which somehow made it less precise, again I don’t understand optics. The point is that while testing something unrelated, this older version of the primary measuring instrument dedicated a massive flaw. It was shrugged off as being an imprecise instrument, and anyway, if there was a problem, the mirror team expected that it would be caught in a final review.. a final review that was ultimately canceled for budget reasons.

But there’s one more wrinkle to this vertex radius test. The interferogram with the wavy lines was sent to NASA. But since the focus of the test was the center of the mirror, only the center of the interferogram was sent to NASA. They basically cut out the center, which only had straight lines, and sent that just that part along. The way I see it, there are two ways to look at this: either somebody innocently cut the relevant part of the interferogram down to the size of a normal sheet of paper so that it would fit in a memo, or someone intentionally cut out the bad parts of the interferogram in order to hide them from NASA.

Briefly, I can see either scenario being plausible. I dug up a document that showed the extent of the cutout. When I measured it, it had the same aspect ratio as a standard 8.5 by 11 inch sheet of paper. In another document, I learned that these interferograms are big, nearly 2 feet across. With that knowledge it’s clear that the cutout was not completely arbitrary. It was as much of the relevant part of the interferogram as could fit on a normal sheet of paper.

But.. the cutout also begins at almost precisely the point where the lines start to curve. So if someone was trying to hide the curvature, they picked the perfect place to put the cut. Now, I find it extraordinarily difficult to believe that somebody would knowingly cover up a flaw in the mirror. After all, there would be no hiding it. It would be found eventually. But I can believe that somebody thought the test instrument was wrong and didn’t want to deal with a bunch of questions from NASA, which would take more time and resources to answer and put them further behind schedule.

If the cutout was designed to intentionally hide bad data, I’m not a lawyer but I’m pretty sure that’s criminal fraud. If it was an unlucky blunder.. then it was one heck of an unlucky blunder.

Alright, so let’s see if we can play a game of “Why”s and work our way down and back up:

• Why were the images returned from Hubble blurry? Because the mirror was too flat at the edges, causing a spherical aberration.
• Why? Because the instrument designed to precisely measure the mirror was constructed with incorrect dimensions.
• Why? Because a technician mistakenly bounced a laser off the end-cap of a metering rod instead of the rod itself.
• Why didn’t anyone notice when other instruments showed a problem? They simply believed that the other instruments were wrong and did no further investigation.
• Why didn’t NASA stop this? The data given to NASA showed everything was fine and they were not aware of the changes made to the reflective null corrector or the problems discovered with other instruments.
• Why? NASA’s QA effort was limited by the proximity to a classified project, a cantankerous team lead, and an overall lack of resources.

I think we could probably go further in either direction but I think that pretty concisely tells the tale.

OK so.. what are our lessons learned? I’ve got a few that are sort of a mix of my personal opinions as well as the recommendations from a few sources. In no particular order:

First, it is essential to maintain test equipment and a robust paper trail all the way through launch and on-orbit checkout. I know that in my day job I have certainly found this paperwork to be onerous, but I understand why it’s there and after this episode I certainly have a newfound respect for it. After nine years, with an inadequate paper trail, it was extremely difficult for NASA to figure out what happened with the Hubble mirror. They did get it, but it seems to me like in some respects they just got lucky. And they never were able to answer every question completely.

Second, independent quality assurance personnel must be given the access required to do their jobs. Nobody likes having their work second-guessed and nobody likes taking time for meetings that feel useless when you’re oh-so-sure that you haven’t made a mistake. But an independent set of eyes can make all the difference in the world. They can break group-think, leverage different experiences, and stop a disaster before it really gets going.

Third, single points of failure are dangerous, and not to be trusted. Sometimes you can’t avoid a single point of failure. But in that case you have to throw as many resources as are required to verify that this critical system works the way you think.

Fourth, pay attention to what the tests tell you, even if it’s not related to the question you’re trying to answer. If you wanted to check if your oven was on, so you opened it up to check if it was hot, only discover a strong smell of natural gas, you wouldn’t discard that information as unrelated to the test. You’d grab the cat and sprint out of your house.

Fifth, limiting communication is dangerous. This one is tricky because there is obviously a line to be drawn somewhere. It’s probably not great for everyone with a concern to immediately call the administrator of NASA. After all, sometimes a perceived major problem really is a misunderstanding. And maybe it really is a better use of resources to not call a giant meeting and launch an investigation into why your unit test failed, only to discover you just forgot a semi-colon. But I think it’s really really easy to take that line of thinking too far. And suddenly your team is insular, distrustful, and impenetrable. Nobody wants to admit that they made a mistake. But admitting that a mistake was made early and fixing it is a whole lot better than allowing it to grow to the point that other people are telling you about it.

Sixth, perform sanity checks. If a relatively cheap test had been done to look for massive errors in the mirror, this whole crisis could have been averted. It wasn’t, because nobody thought it was necessary since the very precise instrument said it was fine. But if you’re totally completely sure that you’ve assembled the car right, then you shouldn’t have a problem when somebody wants to kick the tires and see if they fall off. And remember that precision and accuracy are different things. I could tell you that I am 12.211968 meters tall and that would be a very precise number.. but it certainly wouldn’t be an accurate number! The other null correctors were less precise, but they were more accurate. And since the error was so large they were easily detectable by these other instruments.

I could go on, but this episode has really gotten away from me, so I think that’ll have to do for now.

In the end, NASA determined that Perkin-Elmer’s egregious errors had caused around $87 million dollars of direct damages, not adjusting for inflation. Mindful of the difficulty of proving their case with a spotty paper trail and the long gap between the error and the its discovery, NASA and Perkin-Elmer settled out of court, with Perkin-Elmer agreeing to pay $25 million dollars. Rather than actually cut a check, NASA was given a discount on ongoing work for other missions. Nobody went to jail, nobody got fired. Whether they should have or not, you’ll have to decide for yourself.

But I think it’s safe to say that the concrete costs of fixing the problem were far exceeded by the damages done to NASA’s reputation. The Hubble became a joke, literally used as the punchline in movies and comedy acts. It was compared to the Titanic, and to the Hindenburg. Critics pointed to it as yet another sign of a space agency past its prime and clearly in decline. Or as yet another example of government waste.

But.. that’s not really fair. The Hubble actually still mostly worked, even with the flaw. It was still doing valuable science. And sure, NASA was a different agency than it was in 1968, but the world was different too. NASA was still the agency that put Alan Shepard on top of a Redstone missile, the agency that sent humans to the moon and gave them a car, the agency that sent robots to Mars, that brought Apollo 13 home, and that made launching and landing a 110 ton orbital space plane look so easy that people got bored. We can fix this. And next time, we’ll do just that.

Ad Astra, catch you on the next pass