Episode 165: STS-83 - Substack Attack (MSL-1)

New episode! The crew of STS-83 is ready for a 16 day flight aboard Space Shuttle Columbia, with Spacelab packed full of materials science experiments. So why did the flight end up only being 4 days long?

Episode Audio #

Photos #

Post-Flight Presentation #

You can see the mission in motion in the post-flight presentation right here:

Transcript #

NOTE: This transcript was made by me just copying and pasting the script that I read to make the podcast. I often tweak the phrasing on the fly and then forget to update the script, so this is not guaranteed to align perfectly with the episode audio, but it should be pretty close. Also, since these are really only intended to be read by myself, I might use some funky punctuation to help remind myself how I want a sentence to flow, so don’t look to these as a grammar reference. If you notice any egregious transcription errors or notes to myself that I neglected to remove, feel free to let me know and I’ll fix it.

---

Hello, and welcome to The Space Above Us. Episode 165, space shuttle flight 83, STS-83: Substack Attack

Last time, we rode along with the second crew tasked with upgrading and repairing the orbiting Hubble Space Telescope. The mission didn’t quite have the same stakes as its predecessor, STS-61, but that didn’t make it any easier, or the consequences for failure any smaller. The intricate choreography between the two EV crew members outside on any given spacewalk and their robot arm operator inside the crew cabin was a sight to behold, with minds melding to act as one highly capable unit, entirely focused on the task at hand. Today, like all shuttle flights, we’ll see a similar level of intense teamwork as the seven person crew worked to squeeze all the science they could from the mission despite constraints that started out as challenging, and unexpectedly became impossible.

Today we’ll be climbing aboard the original spaceworthy orbiter of the fleet, Space Shuttle Columbia, and taking the short trip down the tunnel from the middeck into Spacelab, for the Microgravity Science Laboratory 1 mission. This name sounded a little familiar, so I took a quick glance down the list of Spacelab flights, and sure enough, a lot of them end in the word “Microgravity Laboratory”. On STS-42 we had the International Microgravity Laboratory 1 mission, then on STS-50 we had the US Microgravity Laboratory 1 mission. On STS-65 we hit the IML sequel, International Microgravity Laboratory 2, and similarly on STS-73 we find the US Microgravity Laboratory 2 mission. I’m not exactly sure why this flight didn’t end up being the International Microgravity Laboratory 3 or US Microgravity Laboratory 3 flight, but it didn’t. Instead, we have the Microgravity Science Laboratory 1, or MSL-1. So now you all get to have IML-1, USMP-1, IML-2, USMP-2, and MSL-1 rattling around in your heads like I do.

I suppose that now that I’ve made sure any combination of the words “laboratory”, “microgravity” and “mission” have no meaning, I should probably explain what’s actually on tap for today’s flight, which in a way was intended to be the end of an era. With the International Space Station not just on the horizon, but practically on our doorstep, this flight was intended to be the last shuttle-based materials science flight. We’ll still see the occasional material science experiment in the future, but this is the last time that the entire mission is really oriented around it first and foremost. Though even this dedicated mission has its eye on the future, making ISS hardware and procedures a centerpiece of the flight. Let’s meet the crew, get to orbit, and get into it.

Commanding the mission was Jim Halsell. When we last saw Halsell he was riding on the right-hand side of the flight deck as Pilot on STS-74, the second Mir docking mission. This is his third of five flights.

And for the first time in almost ten episodes, we have some rookies! Riding shotgun is the Pilot for today’s mission, Susan Still. Susan Still was born on October 24th, 1961 in Augusta, Georgia. She earned a Bachelor’s Degree in Aerospace Engineering from Embry-Riddle University before continuing on to a Master’s in the same subject from the Georgia Institute of Technology while working as a Wind Tunnel Project Officer for Lockheed. After that she joined the US Navy, becoming a naval aviator a couple of years later. She flew the TA-4J Skyhawk and the EA-6A Electric Intruder before attending Test Pilot School and later learning to fly the F-14 Tomcat. In December of 1994, NASA came calling, scooping her up in Astronaut Group 15, aka the Flying Escargot. With this, her first of two flights, she becomes one of only three women to serve as space shuttle pilots.

Sitting behind Still was Mission Specialist 1, and today’s Payload Commander, Janice Voss. The last time we saw Voss it was on STS-63, the so-called “Near Mir” flight. On that mission, Space Shuttle Discovery approached to within only ten meters of Mir, setting the stage for the docking on STS-71. This is Voss’s third of five flights.

Moving to the left, we find Mission Specialist 2, Mike Gernhardt. When Gernhardt last flew it was on STS-69, the second flight of the Wake Shield Facility, alongside a different astronaut named Voss: Jim Voss. No relation. Since he’s today’s flight engineer, Gernhardt will be backing up Halsell and Still, keeping an eye on the orbiter when the Commander and Pilot were asleep. This is his second of four flights.

Moving downstairs, we find Mission Specialist 3, Don Thomas. Thomas last flew the inky skies on STS-70, the final shuttle-based TDRS deploy mission, waving goodbye to TDRS-G as it made its way up to the GEO ring. Fun fact, Thomas flew with Commander Halsell once before, on both of their rookie missions, STS-65. This is his third of four flights.

Joining the main crew were two Payload Specialists! Payload Specialist 1 was Roger Crouch. Roger Crouch was born on September 12th, 1940 in Jamestown, Tennessee. He earned a Bachelor’s in Physics from Tennessee Polytechnic Institute the same year that John Glenn was lofted into orbit, before moving north to Virginia Polytechnic Institute to earn a Master’s and doctorate in Physics. At a post-flight event, Commander Halsell introduced Crouch as being quote “in charge of the NASA materials science program for about 100 years.” which isn’t all that far off. Crouch started working at NASA Langley in 1962 before continuing on to be the principal investigator or Program Scientist on numerous Spacelab missions and serving as the Lead Scientist for the Microgravity Space and Applications Division from 1985 to 1996. And after his brief astronaut career he’d continue to work on space science, serving at NASA HQ and as a Senior Scientist for the ISS. This is his first of two flights.

And last but not least, Payload Specialist 2, Greg Linteris. Greg Linteris was born on October 4th, 1957, which probably sounds familiar to most listeners here. You have until the end of this sentence to remember why October 4th, 1957 sounds familiar. Yep, it’s the day that the Soviet Union launched Sputnik into Low Earth Orbit, kicking off the space age. Linteris earned a Bachelor’s in Chemical Engineering from Princeton, a Master’s in Mechanical Engineering from Stanford, and a doctorate in Mechanical and Aerospace engineering from Princeton. Linteris spent his career delving into the mysteries of combustion, helping to better understand everything from internal combustion engines, solid rocket propellants, and fire suppression. This is his first of two flights.

Early in the first attempt at launch, well before the crew boarded the spacecraft, the attempt was scrubbed due to a minor thermal issue with a floodlight in the payload bay. But 24 hours later everything was going smoothly. Well, more smoothly. There was a small issue with one of the fuel cells. Without getting into eye-watering detail, a number that represented the performance of Fuel Cell 2 that was supposed to be low was a little higher than expected. And rather than dropping quickly, it was sort of hanging around at elevated levels. The fact that it had been so high violated the launch rules, but as the planned liftoff time approached it was within limits and was continuing to move in the right direction, so a waiver was issued that allowed the launch to proceed.

There was also an issue that was popped up when performing the cabin leak checks. At the end of the test, a seal just came off the orbiter entirely, sticking to the pressurization probe. This necessitated replacing the seal and redoing the test, which meant that there was more oxygen than usual lurking around, which meant they had to wait around a little longer for levels to return to normal, resulting in a 20 and a half minute delay.

But ultimately, the countdown continued, the main engines lit up, the stack rocked forward and back, the SRBs ignited, and on April 4th, 1997, at 2:20 and 32 seconds PM Eastern Time, for the twenty-second time, Space Shuttle Columbia roared into orbit, experiencing an uneventful ascent and kicking off a planned 16 day mission.

The early mission proceeded exactly how it was supposed to. A little under two hours after arriving in orbit, the payload bay doors were opened, and around three hours after that, Spacelab was activated and the scientific investigations could begin. The crew also split up into two shifts, as was typical for a Spacelab flight. On the red team was Commander Halsell, Pilot Still, Mission Specialist Thomas, and Payload Specialist Linteris, and on the blue team Mission Specialist Voss, Mission Specialist Gernhardt, and Payload Specialist Crouch.

As I mentioned at the top, the goal of this mission was sort of a last hurrah for shuttle-based material science, while also providing a convenient shakeout flight for validating ISS procedures and hardware. Case in point, mounted in the walls of Spacelab was an EXPRESS Rack, which is another tortured acronym standing for: EXpedite the Processing of Experiments to the Space Station. EXPRESS. The idea here was to provide a standard interface that all experiments going to the ISS could use. This would make it easier to plan on both the science and operations sides, while also making it possible to develop hardware and procedures dedicated to the handling of these racks. So as the name implies, by getting everyone to decide on a common interface, they could.. y’know.. expedite the processing of experiments to the space station.

But more than just a common interface, the rack provided the support that any experiment would need, making it so that each experiment didn’t have to figure it out on its own. Each of the up to ten experiments mounted in the rack would benefit from electric power, water and air cooling, data connections, and even a nitrogen hookup. All of this was combined into a big box that was about the size of a bathroom shower, which just so happened to be about as big as possible while still easily fitting through the various hatches and around the various corners it would have to traverse in order to be installed in the ISS. What a coincidence!

In addition to the usual material science focus on stuff like crystals, and the return of some experiments like TEMPUS, which used electromagnetic levitation to super-cool materials, there was a big focus on combustion. Combustion is one of these things that humanity has been using since basically forever, but still doesn’t quite fully understand. Part of the problem is that when you study combustion on the ground, gravity exerts a massive influence on the result. When the gas around a fire gets hot, it gets less dense, and in gravity that means it gets more buoyant, rising about the flame. More gas gets pulled in from the bottom to replace the gas that left, and now you’ve got a convective current. It’s why flames are pointy on top. These gravity-related effects could be masking smaller and more subtle aspects of combustion, leaving gaps in our understanding. So while humanity has a pretty good handle on combustion and burning, the hope was that by carefully burning things in the absence of gravity, we could learn just a little bit more. And with so much combustion happening in the world, in everything from cars, trucks, airplanes, cargo ships, and massive power plants, a very small improvement could save a lot of fuel, a lot of money, and a lot of greenhouse gases.

But before we really dig into the science, there’s something else vying for our attention. Remember that pre-launch fuel cell problem that I made sure to draw a lot of attention to? Yeah.. it’s back. And it’s not good. So, let’s refamiliarize ourselves with the space shuttle orbiter’s fuel cells.

First, what even is a fuel cell. Well, we know this from all the way back on Gemini V: a fuel cell is a device that ingests hydrogen and oxygen, causes them to react, and produces heat, water, and, critically, electricity. In short, it allows a spacecraft to generate electricity by carrying a bunch of hydrogen and oxygen around instead of solar arrays. Both approaches have their pros and cons: fuel cells offer more control but only last as long as your supply of reactants last, solar arrays essentially last forever but are physically large and now your mission is dependent on your particular orbit, the spacecraft attitude, and even the time of year. Since the shuttle was always primarily envisioned as having short missions, fuel cells made a lot of sense, so fuel cells are what it used.

The orbiter had three fuel cells, which were all under the liner of the payload bay, right behind the crew cabin. Fuel Cell 1 was on the starboard side and Fuel Cells 2 and 3 were on the port side. For dimensions, imagine a typical kitchen microwave oven, and add another one right next to it, so you have a weird double-wide microwave, and that’s pretty close. Half of the fuel cell did the actual chemistry, while the other half contained support electronics and other equipment.

Inside each fuel cell were three substacks, each composed of 32 cells. Each cell had a porous matrix of electrolyte in the middle, separating an oxygen side and a hydrogen side, each with their own electrodes. Super cold hydrogen and oxygen would be run through a pre-heater to warm them up a bit and then enter the fuel cell, where they would react across the electrolyte, generating electricity, water, and heat. The heat was just sort of a nuisance that was dumped overboard with the radiators, but the water was used by the crew, and the electricity powered the spacecraft.

The nightmare scenario with a fuel cell, because anything on a spacecraft has gotta have an associated nightmare scenario, is the uncontrolled mixing of the two reactants. Hydrogen and oxygen are nasty substances to deal with. Hydrogen is extremely difficult to contain, oxygen reacts with everything, and worst of all, when you mix them together they tend to catch fire and explode. It was super duper important that the fuel cells not experience uncontrolled mixing of the reactants.

One way this could happen would be if a tiny hole were formed in the electrolyte matrix. In a process that sounds a whole lot like blow-by eating through an o-ring, the tiny hole would allow a small amount of uncontrolled mixing, which would result in more heat, which would grow the hole, which would result in more uncontrolled mixing, which would result in more heat, and so on. At a certain point this could start happening fast enough to cause a fire or explosion.

With this in mind, the fuel cells came equipped with a Cell Performance Monitor, or CPM. The CPM would monitor the voltage on each end of the three substacks of each fuel cell. If everything was working great, there would be very little difference between the voltage on either side of a substack, ideally zero millivolts. If the difference reached 300 millivolts, the crew would have a critical problem on their hands with fuel cell failure, fire, and explosions in the near future. So if you were to open your copy of the Space Shuttle flight rules (which I know you all keep handy right?) and flip to page 1432, you’d see flight rule A9-1-J, which states that among other conditions, a fuel cell will be considered lost if quote “The FC substack delta volts changes by 150 mV from the baseline preflight value, is increasing, and measurement of single cell voltages has not been performed.” 150 millivolts, that’s the red line. So you can imagine why the ground and crew were not thrilled to discover that the Cell Performance Monitor was reporting that substack 3 in fuel cell 2 had an elevated voltage differential, and it was increasing at a rate of 5 millivolts per hour. Let the troubleshooting commence! Oh, and thanks to Wayne Hale for clarifying how rule A9-1-J would be referred to verbally.

First, around 20 hours into the flight, while the science crew toiled away inside Spacelab, the orbiter crew were asked to perform a manual purge of all three fuel cells. Nothing was wrong with fuel cells 1 and 3, but it was desired to keep everything in a similar state as fuel cell 2 so that their output could be compared. This was a sort of cross-check against the possibility that the Cell Performance Monitor itself was broken. Plus, the fuel cells were purged pretty regularly even under nominal conditions.

To execute the purge, let’s pull out another voluminous NASA document, the Shuttle Crew Operations Manual, or “SCOM”. the crew would’ve floated over to the starboard side of the flight deck, just behind the pilot’s seat, to panel R11U. From there, they would flip the switch for the purge heater, which kept the purge lines nice and warm, preventing any extra water from freezing and blocking things up, which could cause the purged fuel cell to fail. After half an hour of warming, the crew would flip the “FUEL CELL PURGE VALVE 1” switch and watch the display showing oxygen and hydrogen flow rates. During a purge, the reactants are flowed through the fuel cell at a higher rate than normal, but still slow enough that power could be produced and the reactant pre-heater could keep up. The hope was that the increased rate of flow would flush out any contaminants that might be gumming up the works. After leaving the valve open for two minutes, the crew flipped it shut again, then flipped the switch for the fuel cell 2 valve, waited ten minutes, closed it, opened fuel cell 3 for two minutes, closed it, waited another half hour for things to clear out, and then turned the purge heater off again. So fuel cells 1 and 3 got the typical two minute purge, but fuel cell 2 got an extended ten minute purge.

If you’re curious, the crew could have also left this purge up to the computer by using the same control panel. Flip the fuel cell purge heater switch to GPC, along with the switches for FUEL CELL PURGE VALVES 1, 2, and 3. “GPC” stands for General Purpose Computer, so we’re essentially handing control for these switches over to the computer. Now that the computer is in control, flip the “FUEL CELL GPC PURGE SEQ” switch to “START” and hold is there for at least three seconds after you see the talkback indicator switch to gray. The computer will turn on the heater, wait for the proper temperature to be reached, and open and close each valve in sequence for two minutes each. You can also skip a fuel cell by leaving its valve off of the GPC setting. 30 minutes after the last purge, the computer will turn the heater back off and be done. Such convenience!

OK, that was a lot of detail on how to physically execute the fuel cell purge.. but did it work? Sort of. The rate of increase in the voltage delta slowed from 5 mV an hour to 2 mV an hour, which is good. But the overall delta was sitting at 134 mV, very close to the 150 mV limit. Not great. The crew began to shut down non-critical systems to reduce the load demanded of the fuel cells.

Around seven hours after the first fuel cell purge, with the CPM indicating a 140 millivolt delta on the problem substack, the crew tried again, this time performing a 30 minute purge, on fuel cells 2 and 3. The delta decreased to reasonable levels during the purge itself, but after the purge ended, it popped right back up. The delta eventually settled in at around 130 millivolts, 20 millivolts shy of the flight rule limit, but this was a little misleading. Since the current was also dropping, it was masking the fact that the voltage delta was actually still increasing. At this point there was nothing else to do. The crew disconnected Fuel Cell 2 from the main electrical bus, and at a mission elapsed time of 2 days and 9 seconds, it was shut down and safed, an action that could not be reversed. Columbia was now running on two fuel cells. They didn’t have to initiate an emergency deorbit or anything, but STS-83 had now switched to what’s called a Minimum Duration Flight. Some top priority science would continue, but the rest of the flight was essentially dedicated to wrapping things up and coming home as soon as was reasonably possible.

In reality, the orbiter was perfectly happy to run on two fuel cells. In fact, it could run just fine on only one, though I believe the power available to payloads would be reduced. But since the fuel cells were the only source of electricity, and since electricity was required for, y’know, life, the flight rules did not mess around. As STS-83 CAPCOM Chris Hadfield describes in his book “An Astronaut’s Guide to Life on Earth”, the flight rules exist for this exact purpose. The crew were obviously devastated to be facing an early return only four days into this 16 day mission they had trained so long and hard for, but equally devastated were the scientists who were counting on this data and the ground crews who kept everything running. It would be very easy to say “wellll, the problem is really just with that one fuel cell and the other two are working just fine. Let’s keep pushing it.” I’m sure some people would say that’s even the correct decision. After all, that’s what redundant systems are there for. But mission controllers learned long ago that it was critical to take emotion out of the decision making process. They had already sat down and thought about this very scenario, along with thousands of others, and determined that losing one fuel cell dictated a minimum duration mission. So that was it. Period.

The clock was now ticking, but some science was actually able to continue. In fact, in a section discussing some of the combustion experiments, the official mission report declared somewhat hyperbolically quote “The operations on flight day 3 are regarded as the most important day in combustion history.” I’m not sure who was making such proclamations, but I’m happy to give credit for the most important day in combustion history to the day that Unnk the Caveman picked up a burning branch and ran off with it. That was a big day.

The mission report isn’t all that far off though. Among other things, the combustion experiments succeeded in achieving a soot-free flame by precisely dialing in the flow rate of the fuel, and they also succeeded in making free-floating spheres of flame that were bigger than ever before. Both of these accomplishments provided valuable insight into the fundamentals of the combustion process.

Though, with the power budget reduced, we get another one of those great juxtapositions that the space age seems to love presenting us with: crew members working on sophisticated experiments in the belly of the space shuttle orbiter.. by flashlight. Yep, no power budget for the lights in Spacelab, so flashlight it was. Other shut-down systems included stuff like the Regenerative Carbon Dioxide Removal System. With the flight now much shorter than planned, old fashioned canisters of lithium hydroxide were perfectly capable of handling the carbon dioxide scrubbing needs.

For the next 30 hours or so, the crew and ground support teams scrambled to re-prioritize the onboard experiments, completely rewrite the flight plan, and get as much done as possible, all while observing the new power limits. But only three days and nine-and-a-half hours after lifting off, Spacelab was deactivated, the vast majority of scientific objectives not completed. There was still some flying to do, but the mission was essentially over.

With such a depressingly short flight, the Florida weather decided to add insult to injury by being completely clear, resulting in the deorbit burn taking place right on time. Even during entry, the fuel cell situation affected life for the crew. Some navigation aids were not activated until Columbia had slowed down to 4.6 kilometers per second, around 60% of orbital velocity. But everything worked as it should, and after an uneventful entry, Space Shuttle Columbia touched down safely at the Kennedy Space Center rolling to a stop and closing out STS-83 after only 3 days, 23 hours, 12 minutes, and 39 seconds.

The post-flight investigation provided some frustrating news and some grimly satisfying news. First, the root cause of the fuel cell failure could not be determined, which is pretty annoying. However, upon examination, 20 adjacent cells in fuel cell 2 were found to be degraded, which meant that the Cell Performance Monitor was telling the truth after all. I call that satisfying because as bad as the crew felt about their truncated flight, imagine how bad they’d feel if they discovered that it was all due to a faulty sensor.

Future flights would benefit from STS-83’s lessons learned, with the Launch Commit Criteria updated to be on the lookout for the anomalous fuel cell behavior that was spotted and dismissed in the buildup to launch. But I’m sure that was cold comfort to the crew.

All that said, don’t feel too bad for them. We’ll get into more detail about this later, but in an unprecedented move, just a few months later, NASA would fly the same crew on the same orbiter with the same payloads to perform the same mission. STS-83 would get a do-over! And in doing so, it would provide some fascinating insight into space shuttle processing and budgets. So hang tight, STS-83 crew. We’ll see you again soon.

Next time.. while the STS-83 crew were disappointed to return to Earth so much sooner than expected, I can think of someone who would eager to swap places with them. Jerry Linenger has been on Mir since January, and after an eventful few months is eagerly checking out the window for Space Shuttle Atlantis to arrive. Let’s go get him.

Ad Astra, catch you on the next pass.