The Right Key

Abstract

The four space walks of astronauts Luca Parmitano, ESA astronaut, and Drew Morgan, NASA, took place from November 2019 to January 2020 to extend the lifetime of the 9 years old AMS (Alpha Magnetic Spectrometer) cosmic ray detector. This is a poignant overview of the technical difficulties and engineering challenges, including the development of special tools, overcome by a team of engineers and physicists from CERN and NASA over the 4 years planning and the challenges of the Extra Vehicular Activities (EVAs) for the astronauts. This chapter is based on interviews carried out by Paola Catapano and Claudio Bortolin to astronaut Luca Parmitano and the scientists behind the mission; these are guiding the reader through the most stressful moments of the four EVAs. The success of the EVA’s program allowed AMS-02 to be back to fully operational in space, poised to explore our Universe for many years to come.

At the beginning of 2014, right after the end of my first mission, a NASA engineer asked for my opinion about the feasibility of carrying out a complex mechanical repair on the AMS-02 experiment. She showed me some slides where the problem was described, asking for my opinion. I replied that, since it was an engineering intervention, we had to adopt an engineering approach, so with the right tooling we would be able to do it. At that time, I would never have expected to be the one who would carry out the activity.

This is how Italian European Space Agency (ESA)¹ astronaut Luca Parmitano recalls the very first time he was asked to comment on the repair of the CO₂ cooling system of the AMS-02 cosmic detector, which is installed on the outside of the International Space Station (Fig. 1) [1]. It was February 2014, when the whole operation was still just under evaluation. And who would disagree with Colonel Parmitano when he said: “Repairing any part of a particle detector is a complex and delicate operation even on Earth, now let's imagine having to do it in Space, outside the ISS and wearing pressurized gloves.”

Fig. 1

(Photo credit NASA)

The International Space Station

Every physicist and engineer who deals with particle detectors and their infrastructure is used to dealing with unprecedented issues during the operation of this sophisticated custom-made equipment. For instance, the thermo-mechanical stability is a key parameter for the trackers—the ultralight innermost subsystems of a particle physics detector. We know that even just one-degree difference in temperature between the opposite sides of a silicon sensor can affect the micrometric precision of the particles’ path reconstruction, thus disrupting the description of the physical phenomena that caused the interaction. The work of thousands of scientists would be jeopardized by that tiny detail!

Cooling engineers at CERN would unquestionably agree with Luca Parmitano: repairing any detail of a cooling structure, which is integrated in a fragile detector structure, isn’t at all an ordinary operation, even on earth. As scientists operating some of the most extraordinary scientific tools humankind has conceived and built, we are often required to think out of the box, to find the solutions that are needed to repair or upgrade technical equipment. This is what happened to CERN engineer Claudio Bortolin in 2011, when he was a young collaborator in the ALICE experiment, one of the four giant detectors installed at the LHC,² some eighty meters underground. A stainless-steel twisted cable—the same used in bicycle brakes—was the key “tool” he used to develop a flexible drilling system that he and his team used to drill ten clogged filters of the cooling circuits of the silicon detector, which was situated four meters away from the last access point to the underground area. The elements to be drilled were at the end of a four-millimetres stainless-steel pipe with a couple of corners on the way [2]. Any alternative solution would have involved dismantling the ten-thousand ton equipment to reach that tiny filtering element—certainly not a simple operation!

Those clogged filters weren’t for Claudio just a personal success but also the reason why, on a Saturday afternoon of May 2015, he received a phone call, informing him of the cooling issue on AMS-02.

1 Back to the Beginning

On that sunny afternoon Claudio was, as usual on most Saturdays, guiding the tour of a group of Italian high-school students visiting CERN. By coincidence, they were at the Payload Operation Control Center, or POCC, the “control room” of AMS-02, one of Claudio’s favourite visiting sites on offer to CERN visitors. Claudio was fascinated by space research since his early youth, and quite naturally, this had become one of the favourite topics of his guided tours. He was particularly good at making visitors feel part of the human adventure called “Science”, by adding anecdotes and personal stories to his presentations. On that Saturday afternoon, he was explaining some details of the CO₂ cooling system of the silicon detector of AMS-02, a very similar system to the ones he was operating and maintaining in the experimental caverns at CERN, when his mobile phone rang. The display read “Bart Verlaat”, a colleague of his. The unexpected call on a Saturday worried him because they were both involved in many activities related to CO₂ cooling systems installed in the LHC caverns, and the call sounded like a red alert. Before picking up, he told the group of students: “Wow, what a coincidence, the Dutch engineer who made the first drawing of AMS-02 is just calling me!” Indeed, Bart is the most experienced expert in CO₂ detector cooling systems at CERN. “Professor Samuel Ting³ called me last night and asked me to get on the first plane to Geneva because we have a problem with the AMS CO₂ cooling pumps” – he informed Claudio. “I need your help with the fiberscope you used a few years ago to investigate the problem of the blocked filter in the ALICE experiment. We have a replica of the AMS pump at CERN, I plan to check it by looking inside. “Claudio looked at the young visitors: “Do you remember the famous sentence: Houston we have a problem? Well, this is one of those moments: Houston we have a problem on AMS.”

Was the problem at all expected? In 2015, NASA had decided to extend the International Space Station program to 2024; in 2018 the program was extended to 2030. AMS-02 was initially due to run until 2020, but the program extension was felt by the AMS-02 Collaboration as a great opportunity to run the experiment a few years more. However, after a first analysis of the pressure trends on the AMS cooling system and the investigations on the pump replica, at the end of May 2015 it became clear that, in order to operate AMS for as long as the ISS was running, it would have been necessary to replace part of the cooling system, in particular the four pumps that had shown the first signs of failure.

As Nobel laureate and AMS principal investigator Sam Ting said during an interview, “The first time something was wrong it was February 2014, when a pump stopped. We had three functioning, but I called NASA and said, if a pump stops, you can’t assume the others will not. We needed to start thinking of a replacement. There was tremendous resistance, mostly technical, because the pumps are connected with high pressure lines and we didn’t know how to start, we didn’t know what was wrong with the pump [3].”

Soon after May 2015, a team of engineers and physicists from CERN and NASA, began to study the problem and set up the repair procedure for an experiment that wasn’t designed to be repaired. Actually, it wasn’t just a matter of identifying and testing one single procedure, but rather a set of very sophisticated and unprecedented procedures, such as reaching the working site of the experiment on the ISS, or accessing the cooling system, removing the thermal insulation called MLI (Multi-Layer insulation), cut the pipes, connect a new system and the relevant power supply and sensors reading cables, and finally run the pressure test and put the system back into operation. Obviously, all this work had to be divided into more than one Extra Vehicular Activities (EVAs). Eventually, four EVAs were established, each lasting about ten hours, with the astronauts wearing a pressurized spacesuit while traveling at 28 thousand km/h around the planet at more than 400 km of altitude. Nothing similar had ever been attempted before! Even for CERN engineers, with the experience accumulated in experimental areas over several decades, this looked like a real challenge.

This repair was very difficult”, continued Sam Ting during the same interview. “It can be compared to a heart transplant. It took four EVA’s, originally it was planned they would be five or six. These were the most difficult EVA’s NASA ever took, and it took us five or six years to prepare them. First thing, we ordered an identical pump and by checking its functioning we concluded that replacing the pump was not enough, we would need to replace the whole central part of the cooling system. It’s like taking a whole heart out and replace it entirely and reconnect it to the body. NASA has been extremely supportive. It was not a question of money but ability. How to do this? When we started the system, we never thought one day we would replace it entirely. We have four pumps, so there’s a 400% redundancy. Last year, we were only at 50% efficiency, so it was a good time to replace it.

Imagining, carefully designing and finally testing all these procedures, required a precise and meticulous preparation that lasted about five years and involved many engineers, technicians and some astronauts, who practiced the whole operation over and over again, in various simulated environmental conditions, recreated on an AMS-02 mock-up installed on the ISS truss structure and at the NASA Neutral Buoyancy Lab in Houston. In mid-November 2019, a rocket launched from Virginia was the last of four launches required to bring all the materials needed for the AMS EVAs to the ISS, including a white box with the brand-new pump system, without which AMS-02 would have been left with no chance to continue its data taking activity and potentially make new discoveries. Together with the new cooling system, that rocket transported also twenty-nine special tools, custom-designed and developed by the expert team specifically for the AMS EVA programme. They included a commercial tool, a 7/8″ hexagonal key, technically called wrench, that Luca wanted to be photographed with before getting back to the airlock of the Space Station. That wrench made history. Parmitano published its photo on his Twitter account a few hours after the EVA with the following text: “My favourite picture of last Saturday's EVA, so symbolic: that wrench saved the day - and AMS!” But the story of that 7/8″ hexagonal wrench had started long before, in Aachen (Germany), in a small workshop in the basement of the apartment where Ken Bollweg, the engineer at NASA/JSC AMS-02 Project Office team, was living during the construction of the new cooling system. Talking with Ken, it becomes immediately clear that mechanical engineering is part of his DNA: “My father had a professional activity in the plumbing, electricity and heating sector and I was 11 when I started handling mechanical equipment. At 12, I built my first pump, while at the age of 14 I put together my first car engine. AMS-02, in the end, is just about plumbing, but it has to be carried out in space [4].”

2 The Key Tooling

The central part of AMS-02 consists of a permanent magnet whose role is to bend the tracks of cosmic particles crossing its magnetic field with an intensity of 1500 Gauss (more or less 3000 times higher than the Earth's magnetic field). The particles’ charge can then be inferred by observing the direction of the bended tracks, while the radius is proportional to the momentum of the particle. On AMS-02, the tracking capacity is provided by nine silicon detector planes. In total, the path of the particles from the entry point to the exit point is about three meters, the sum of the area of each silicon plane is about seven square meters, and the precision of the particle track reconstruction is about ten microns (10 millionths of a meter). Without its silicon tracker, AMS-02 couldn’t not continue to take data and do the important science it is doing. The CO₂ cooling system is an essential element of the tracking detector operation, as it guarantees the required thermal stability for the silicon sensors to work. The one-degree tolerance must be guaranteed in spite of the harsh outdoor environmental conditions of the ISS, where temperature can vary from −160 °C at night to +120 °C during the day, at each orbit, which lasts about 90 min. So, when operators observed the first signs of aging of the cooling pumps, four in total, alternating one in run and three in stand-by, they had little doubts: in order to extend the lifetime of AMS-02, and even if the experiment was not designed and built to be repaired once installed on the ISS, there was no alternative but to replace a large part of the cooling system.

The preparation of the EVAs required the development of several tools, which were needed to remove the MLI, get access to the stainless-steel pipes, cut them, install new connectors with front and back ferrules to engage the pipe. All this, obviously, while making sure the whole system was leak-tight. And with the astronauts having to carry out all these operations from inside a pressurized spacesuit and wearing pressurized gloves. To get a better idea, we should imagine ourselves trying to repair a wristwatch operating on the small cogwheels while wearing ski gloves [2].

At the RWTH⁴ University of Aachen in Germany, experts were working on developing and testing the new cooling system (Upgraded Tracker Thermal Pump System—UTTPS); in the meantime, another team in Houston was preparing the tools that the astronauts would have to use during the EVAs. Ken Bollweg had participated in the AMS adventure as senior engineer for NASA since the 90s, when AMS-01, a precursor experiment, flew onboard Space Shuttle Discovery for 10 days in June 1998 (STS-91 mission). During this test mission, AMS-01 collected enough cosmic-rays data to demonstrate the feasibility of building, transporting and running an experiment for high-energy physics in space. Ken was part of the so-called APO, the AMS Project Office, the group responsible for payload integration, safety, and oversight of AMS operations on the Space Shuttle and the International Space Station. To explain the complexity of the EVAs, Ken mentioned the long procedures that astronauts have to follow many times during their training, before they can carry them out at best in space: “In every EVA there are procedures to follow, pages and pages of instructions that increase a lot if they include the steps to solve multiple issues. A fundamental aspect was added to the EVAs of AMS-02: the experiment was not designed and built to be repaired in space. Despite this fundamental aspect, there were only two cases during four EVAs where astronauts had to apply a corrective procedure, something absolutely amazing, unprecedent.” In a complex operation such as the repair of AMS-02, the preparation was absolutely the most complex part, the tools developed, and the many hours dedicated to training, to use and improve them, were key factors: “It required an enormous effort of preparation and interaction to verify and improve the procedures; we used a new facility called ARGOS, Active Response Gravity Offload System, a system based on a cable supporting the astronaut that interacts with a mobile platform to simulate reduced gravity environment. It allowed us to closely observe the astronauts while they were exercising and we could realize what aspects we could improve, both in terms of procedure and on the tool itself.”

While some NASA engineers focused on the possible use of new connectors to facilitate the work of the astronauts, Ken decided to investigate an alternative route. His idea was to continue using the same commercial connectors already installed and performing well on the AMS-02 cooling circuits (the same connectors that are widely used at CERN). He would then integrate them with the ferrules in a single mechanical tool to make installation easier and at the same time enable the astronauts to check the pressure tightness without any additional tools. Since he was also following the construction and testing of the UTTPS, Ken decided to build a small mechanical workshop in the basement of the building where he had rented an apartment during his stay in Aachen.

In a perfect NASA style, Ken Bollweg began to develop this tool which then, thanks to the collaboration of institutes and private companies such as NASA-APO (Houston—US), Jacobs (Dallas-US), MIT (Boston-US), INFN-SERMS (Perugia—Italy), RWTH (Aachen—Germany) and (HAKU GmbH Aldsorf-Germany), became what has been called AAF—the AMS Advanced Fitting. The Fitting is a cylindrical shape mechanism, in which two pipes can be connected just by using one hand to hold and an hexagonal wrench to close, at the same time. Inside the Fitting, a pre-assembled connector, made of male, female and ferrules, was ready to be swaged. The way in which these elements are tightened is especially important for their functionality. If they are swaged too little or too much, there is the risk of a leak. The optimal rotation angle to swage them is obtained with one complete turn of 360 degrees. This is the reason why some letters and numbers were printed in the central part of the AAF body: to allow astronauts to check the rotation angle while swaging the external nut by using the famous 7/8 “hex wrench. They would take note of the letter and the number and then use the wrench to swage one complete turn, until the same letter and number were aligned again. Another crucial purpose of the AAF was the leak check, to be performed after tightening the connectors. A small piston integrated in the cylinder would be triggered in case of any pressurization of the volume due to a CO₂ leak. The system is called the Visual Leak Indicator (VLI). Luca Parmitano and Andrew Morgan, the astronauts engaged in the EVAs for this mission, were trained to deal with any event in which the piston would move and a red band would appear. Such training took place in Houston, at a mock-up of AMS-02, where the tools prepared for the EVAs programme were flown to start the testing phase of the entire operation.

3 Planning and Preparing

In space, any unexpected situation is a source of risks which must be minimized. The success of the mission is strongly linked to the training on the ground before the EVAs. To leave nothing to chance, several astronauts got involved during this phase of the mission preparation. Their invaluable experience was a key contribution to the improvement of the tools, the design of the movements to carry out, the exact sequence of the procedures and eventually the operation of the equipment. Veteran astronaut Chris Cassidy (retired on 28^th May 2021) tested the procedure twenty-nine times! The exercise took place partially inside the flotation pool at NASA's Neutral Buoyancy Laboratory—a diving tank of 62 m length, 31 m wide and more than 12 m deep. The role as lead Spacewalker (EV1) is usually assigned to astronauts who have already gained extravehicular experience, so it was decided to assign it to Luca Parmitano; for Andrew (called usually Drew) Morgan, a physician of the US Special Forces Group recruited in 2013, this was the first mission in space. Luca completed fourteen runs of training, twelve of which as EV1 and two as EV2 while Drew completed all his eight exercises as EV2. Other astronauts were trained and repeated the procedure over and over again, to refine it in detail and to find any small problem or identify any risks: Michael Hopkins did twelve runs, Jeremy Hansen eleven, Kathleen Rubins four, Anne McClain two. A further nineteen runs were carried out by another group of astronauts, for a total of ninety-five runs. Each time, the full exercise was tested, improved and optimized, the tools were tested and the backup scenarios implemented. The date of launch was approaching in the meantime. On July 20th, 2019 space mission “Beyond” was about to begin.

4 Lift-Off and the “Opening Act”

July 20th, 2019, is an important day for all of humanity. The fiftieth anniversary of the first human on the Moon is celebrated in every corner of the planet as a lasting symbol of Space conquest. The Apollo 11 mission is part of the imagination of all the children of the Earth; every child born after 1969 has at least once dreamt to become like Neil Armstrong, when he was photographed by Buzz Aldrin on the surface of the Moon, while Michael Collins was orbiting around it, inside the Command and Service Module (CSM). Seven years later, two of those children were born and thanks to their commitment, dedication and a little luck, their dreams became true: Luca Parmitano and Andrew Morgan. On that day, they were on board the Soyuz MS-13 spacecraft with Russian commander Aleksander Skvortsov, when it took off from Baykonur (Kazakhstan) at 16:28 (UTC) to dock the International Space Station about six hours later, at 22:47 (UTC). When the last payload with spares and tools for AMS-02 docked on the ISS, came the day of the first extravehicular activity (Fig. 2).

Fig. 2

(Photo credits NASA)

Andrew Morgan and Luca Parmitano check spacesuits and tools in the Quest airlock

During an interview to PassioneScienza.com,⁵ Luca, wearing a polo shirt with the AMS-02 logo, shared a detail that helps to understand the team spirit of all those involved with the AMS-02 repair operations.

We decided to give nicknames to each of the EVAs [5]. For the first one, we decided to call it “The Opening Act”, which in English can also be called “Overture”, a pun that well described the open-heart operation that we were about to begin. We literally had to create an access to the detector, by removing the protective layers, thus opening a passage and getting to the area where subsequent operations would be carried out in the following EVAs.

The difficulty of the AMS-02 operation was compared to the repair of the Hubble Space Telescope in the early 1990s. The space station, including its solar panels, is about the size of a football field, 100 m long and 80 m wide. The pressurized and inhabited part of the station is located in the centre, from where the main truss structures develop and branch off along the side of the station. Along these trusses, various payload connecting points are located, supporting structures where experiments in vacuum are carried out, such as the ExPRESS Logistics Carrier (ELC), or storage areas of components such as the External Storage Platform (ESP). AMS-02 is mounted on one of these points, approximately twenty-two meters from the pressurized zone. This was the distance that Drew had to travel repeatedly, moving by hand along the truss structure between the Airlock and the working area, and carrying the necessary equipment back and forth several times. In the meantime, Luca was hooked to the CanadArm2, manoeuvred from inside the Space Station by fellow astronauts Cristina Koch and Jessica Meir, who moved Luca directly to the AMS-02 detector [6].

The first task to be carried out once we reached the working site, was to unscrew 27 bolts, to be able to remove a shield panel, a protective metal element behind which we would have had access to the pipes of the cooling system. The bolts used for AMS-02 obviously had not been chosen with a possible repair in orbit in mind. The ones we use during the EVAs are especially made so that screws and washers cannot be dropped, which would be dangerous for the ISS and for the astronauts. The twenty-seven bolts to be removed from AMS-02 were all of different sizes and, of course, came with washers and nuts. We had to use specially designed tools, installed on what we called the Power Grip Tool or PGT, an electric unscrewing system, in order to capture these small objects which, if lost, might become potentially dangerous. There wasn’t enough room to use the PGT everywhere, so I had to remove some bolts by hand. Furthermore, before extracting the metal panel, to be able to handle it safely with pressurized gloves, we mounted a handling mechanism. Once disassembled, I passed it on to Drew who threw it down in a dispersion cone, a specific area identified by the Earth team, where the panel gradually lost altitude and eventually self-destroyed once in contact with the Earth's atmosphere.

An operation such as the one described by Luca would have been all in all trivial on Earth and, even in case of loss of a washer, one would not have worried at all. But on the orbiting station it is anything but simple. It is interesting to notice that the automatic screwdriver used by Luca was the successor of a similar tool used to repair the Hubble Space Telescope several years earlier.

Once the cavity inside which the pipes were located was opened, we began to remove the MultiLayer Insulation or MLI. These are layers of thermal insulation (necessary to protect the pipes from external temperature variations) that we had to remove to have access to the pipes, that we would then cut.

In total, the astronaut had to cut eight pipes to isolate the old system and connect the new one; six of these could be reached from the newly opened panel, while the other two were located on the lower part of the experiment, even more complicated to reach. All tubes had an outside diameter of 4 mm with an inside diameter of 3 mm. Once the MLI was removed, Luca and Drew finally had access to the Vertical Support Beam (VSB), a U-shaped carbon fiber structural beam, where the tubes were housed, protected by a carbon fiber cover.

The idea of using the VSB as an access point to the pipes came from Italian engineer Corrado Gargiulo, a CERN staff member, who during the construction of AMS-02 was responsible for the integration of the experiment [7]. If there is a person who knows where each bolt is placed and how to get there, that's him: “It is thanks to the procedures required by NASA, that we have been able to put together a reasonable procedure to connect a new cooling system to an existing experiment. Also, at CERN we usually document a lot of what is in the experiments, the various equipment, infrastructures and instrumentations, but the level of detail required by NASA is a different thing. The AMS-02 experiment is made up of thousands and thousands of elements, the effort required was enormous: from the nomenclature of the functions, the preparation of documentation regarding the position and size of the elements, together with their photos. However, this effort paid off years later! It is thanks to that information that I proposed to access the pipes from that spot, the VSB was well suited for cutting and connecting the new system. Other access points were considered during the study of the intervention, but at the end I have suggested that the VSB would be the best option based on the documentation we prepared ten years earlier.”

The proposal was well accepted by NASA, given that any other location for accessing the pipes had been discarded. Corrado points out that that decision still involved a huge amount of work before validation: “It is starting from my proposal that 29 tools were designed, developed and validated through multiple extensive trainings with the astronauts, an incredibly complex and successful job.” However, access to the VSB was not without its problems, including safety, since manipulating carbon fiber surfaces with pressurized gloves involves quite some risks. The tension was palpable both in the control room in Houston and at CERN's POCC.

Luca Parmitano recalls those moments, trying to convey the reasons behind the tension. “That cover wasn't just bolted on, we literally had to break it off, in order to take it off. It was thin enough to pull it off, but imagine grabbing a potentially sharp object with pressurized gloves! The operators at mission control were very concerned about this task, but it went well and after removing the cover, Drew used the same dispersion cone to launch it and let it degrade in the atmosphere, until it destroyed itself like the previous panel. The last task of this first EVA was to arrange the new power and data transfer cables close to the connectors in the old cooling system. With this, we concluded the first extravehicular activity and returned to the Airlock.”

5 The Nail-Biter

A week after the Opening Act, Luca and Drew were back in the Airlock to start the next stage, the second extra-vehicular activity, which took place on November 22nd, 2019.

The nickname chosen for this second EVA was “The nail-biter”, a way to represent the pressure the team was under, since it was time to cut the pipes of the old cooling system. This was a complex operation, since we first had to identify which pipes to cut: they were all the same diameter, and looked identical. Moreover, we had to cut them in the right sequence and at the right place. There was clearly a lot of tension in the team that day, because any mistake would have effectively terminated the operation and we would have had to start all over again, including the tools preparation on Earth.

AMS-02 that day was disconnected from the system that had allowed it to work until a few days earlier, an error would mean a long interruption or even the end of the data taking, most likely for good.

Inside the VSB there were ten pipes of 4 mm in diameter, but only six of them were to be cut; the remaining four were arranged in a continuous pattern, from both ends, and the risk of cutting the same pipe in two different points was therefore very high. “The pipe cutter used to perform the cut was a commercial tool re-adapted for space operations; once cut, the pipes became like real darts, and there was a serious risk of cutting the pressurized space suits, thus endangering the astronauts themselves, in this operation. Before getting to the more stressful moment, we had to prepare the data signal connectors between the new and the old system. While the new connectors were suitable for their use during extra-vehicular activities, those of the old system could damage the spacesuit if touched and for this reason I had to use a special tool to hold the old connector with a glove, while I connected the new connector with the other hand. Carrying out these tasks using pressurized gloves isn’t easy at all, it requires considerable force to be able to exert any pressure, it is not difficult to imagine how much effort is required just to hold small objects like these in the hands. Fortunately, intense training runs on the ground enabled me to acquire the required manual skills.”

After installing a mechanical interface with the PGT, that would later enable to hook the box containing the new cooling system, came the most critical moment: the cutting of the pipes. “Identifying the pipes before cutting them was a painstaking job! Since the pipes are identical, we had to be absolutely sure to use the pipe cutter on the correct ones. Once cut, I folded the ends of the tubes outwards and plugged them with special mushroom-shaped caps with the corresponding number marked on. These would allow me to avoid accidental collisions with sharp objects and then to easily identify them for their connection to the new system. The work area around AMS-02 was full of very fragile structures and sensors, so another complication was to avoid any accidental collision with this equipment which, if damaged, could have meant the end of the experiment. So, sometimes I was working in a very constrained position, movements had to be very precise and controlled. After cutting the pipes and extracting the VSB, I had to close the area with the MLI to protect the internal part from the external environment. With this operation the second extravehicular activity was completed” (Fig. 3).

Fig. 3

(Photo credit NASA)

Luca Parmitano outside the ISS with the box containing the new pumps

6 The Money Run

It was December 2 when what Luca called the moment of truth came, that of the third spacewalk.

Here we are, ‘the money run’ name was decided by the team precisely to convey the meaning of a bet, a “failure or success” operation. I was taken to the work area with the robotic arm, while holding the UTTPS, the box with the new cooling system, which had been handed over to me by Drew shortly before. The first thing Drew and I did was to hook the UTTPS on the interface we had mounted during the second EVA; only at that point we would have started the much more complex phase, that is to connect the cut pipes of the old system with new ones.

There are several tools on the market for cutting steel pipes, the most precise and accurate ones are pipe cutters designed to rotate around the pipe, so as to cut it little by little, keeping a nice round shape. In the case of AMS-02, such a tool could not be used, since there was no room available behind the pipes. So, Luca used a sort of shear, especially developed and adapted by NASA engineers. However, the effect of using a tube shear is that the tube gets pinched in the cutting area.

The first operation was to cut the pipes again using a rotational pipe cutter; if I could have rotated it around the pipe, it would have taken 22 turns to complete the operation and get what in technical terms is called a "clean cut", giving the tube its original shape with regular cutting edges. In the specific conditions of the AMS pipes, I could apply at most three quarters of a turn, so I had to repeat this manual operation many more times to get a clean cut comparable to the 22 full turns theoretically required. It was a rather long and tiring operation, flexing your fingers to use the tools with pressurized gloves is like trying to squeeze a tennis ball. Each tube required about two minutes of work; doing it on eight tubes was a rather tiring part.

When two pipes have to be connected, another important factor is their alignment; in fact one of the risks is that leaks will occur once the fittings are mounted. Some tools that help straighten the tubes are available and, exactly like on Earth, this operation was required also on the orbiting station.

Another very important phase began immediately after: we had to make sure that each pipe was straight enough to be able to insert it into the AMS Advanced Fitting (AAF). For this purpose, a special tool was prepared, essentially consisting of two C-shaped housings, that allowed them to close around the tube and straighten any folds through a closing system. At the same time, this tool also allowed us to test the insertion of the tubes and mark the right length for the point of insertion in the Fitting. It was astronaut Chris Cassidy who suggested marking that point with Kapton tape, previously prepared inside the station by Drew and me, thus creating a reliable reference at the time of junction of the pipes.

Then the time came for the connection of the new cooling system: the tubes are numbered, they all carry a label to help the astronauts insert them in the Fitting for an optimal length. All the Fittings prepared by the team on the ground, and tested among others by Ken Bollweg, were ready to play their fundamental role of swaging and leak-checking.

Thanks to the presence of two decoder rings in the Fitting, I could use the hexagonal wrench to control the tightening angle. With one hand, I held the Fitting through a knob, while with the other I applied the torque closing with the 7/8" hexagonal wrench. We already knew the last two pipes on the lower part of AMS-02 were the most problematic ones, due to the difficulty of access; we were well aware that their straightening would not be easy at all. We had to work a little, but then we managed to straighten them, plug them into the AAF system and plug them into the new system. Once the joints were completed, we had to cover the pipes with layers of MLI to ensure good thermal insulation from the outside environmental conditions. This was the last task of that EVA.

7 The Closing Act

Onboard the ISS, in the weeks after the first EVAs, various activities had taken place and other space vehicles had arrived. On January 25, 2020 [8], about two months after the third spacewalk, Luca and Drew were ready to complete the work. A very tense moment was waiting for them out there.

The name of the activity intended to reflect our main task for that day: closing AMS-02 with the MLI insulation blankets. There weren’t any particular risks and with the mission’s ground control we decided that it could be a good opportunity for Drew to act as EVA1, that is the person in charge of the operation. We were quite happy for Drew, because this change of roles would have meant not only being in charge of the EVA, but also for the first time being attached to the CanadArm2, certainly an important experience to have. As I was not going to ride the robotic arm, I positioned myself on the work area where 6 out of 8 connected tubes were reachable. We had shared the tasks, and decided that Drew would check the two tubes positioned in the lower part of AMS-02.

In the meantime, a reasonable quantity of CO₂ was taken by the AMS-02 tank to pressurize the pipes, new connections included. This pressurization would have allowed the astronauts to check the tightness of the fittings installed.

I agreed with mission control that the first pipe I would check was number 5. While I was trying to open the lever that enclosed the piston of the VLI (the Visual Leak Indicator), I immediately realized that something was wrong: that lever had to open quite easily, almost without applying any force and instead, I struggled a lot to open it. Once opened, I clearly saw the red band, which meant that the connector inside was leaking.

The first check carried out by Luca was already showing a problem, and there were seven more to check! All the other tasks performed during the previous EVAs and their long preparation depended on this very moment. Despite the great preparation, the disappointment could be perceived in Luca's voice as he communicated with mission control: “I have red showing on number five.“ In the control room in Houston and at the POCC at CERN, the tension rose, the colour and the expression on the faces of the operators and physicists of the AMS collaboration changed, the worst possible scenarios started to cross their minds (Figs. 4 and 5).

Fig. 4

(Credit CERN-PHOTO-202001-021-13)

Looking up from the POCC at CERN to the giant screen showing the live streaming of the 4th and final Extra Vehicular Activity for repair on the AMS detector.

Fig. 5

Physicists of the AMS Collaboration in the POCC at CERN (CERN PHOTO-202001-021-15)

At mission control in Houston, Ken Bollweg was wondering if he had omitted any details, if something had not been carefully considered in the preparation of the procedure. The first connector was leaking and, even if the subsequent actions were carefully prepared, the team could not stop wondering what went wrong with that connector and what would happen with the others. “You can imagine, the first connector had a big leak, what was wrong? Did we make a mistake? Is there something wrong with the visual leak indicator? Was any calculation wrong? All these questions were in my mind, and of course we were prepared to face situations like this and a series of procedures were ready to deal with this one, but during all of them I was really sweating bullets.”

Outside the ISS Luca was facing a stressful moment, it was important to put in place everything necessary to solve the problem or at least try to solve it.

For an astronaut, the execution of the work is the greatest testimony of the preparation and commitment put in place to achieve a result, therefore it was a great disappointment to be dealing with a problem at the first connector, without knowing exactly what to expect at that point from all the others. I knew exactly what I was supposed to do next, which was to use the hex wrench again and tighten the connector once more flat, 1/8 of a turn more (45°). In my mind, the idea of having to cut the pipe and apply a jumper—that is a piece of pipe with two connectors at the ends—was becoming realistic. The time needed for this additional operation also made us think that a fifth EVA would be necessary. I would have left the ISS about ten days later, so there would have been no time for me to be part of it. A fifth EVA would have had a major organizational impact, as it would have involved another team of astronauts.

Luca then let the residual gas out of the joint system and with the hexagonal wrench rotated connector number 5 by another 45 degrees. While waiting for the second pressurization, to check the piston again and verify the presence or absence of the red band, he would continue checking the remaining joints with Drew.

Mission Control: “Go for number 1 Luca”.

Luca: “Number 1 is fully open and I have no red”.

Mission Control: “Ok, number 1 is good”.

We checked all the other connectors and we realized they were all leak-tight. It was great news, the only one with a problem was number 5, the first one I checked, which was also the last one we connected during the previous EVA. After about an hour, I went back to the connector of pipe number 5 for the second check.

Luca opened the connector lever: “We do have a leak”.

The red band was still there, the connector was still leaking. “At this point we were in uncharted territory, it had never happened that, after applying one more flat, the leak persisted. From mission control, I received a recommendation that was not foreseen: since we would have to cut this connection anyway to apply the jumper, trying everything and apply a second additional flat was the thing to do. Drew and I invested a lot of time in the preparation of all EVAs, all the details were discussed many times and although in the middle of an unexpected situation, we had to deal with it. So I applied the additional flat and while waiting for the new pressurization of pipe number 5 I quickly went back to the Airlock to collect the required tools and equipment in case we had to install the jumper. In about 25 min, I was already back to AMS-02; Drew and I reversed our roles, I positioned myself again on the robotic arm about 45 min from when I had torqued the AAF a second time. Then the moment to check the VLI again came.”

Luca: “No red”.

Mission Control: “No Red?”.

Luca: “No red!!!”.

At this point a race against time began. Would we be fast enough to install all the MLI covers and blankets required to be able to restart AMS-02 as planned? From mission control they didn’t believe there was enough time: this part of the procedure was impossible to test during our training, it was not really possible to simulate it under the same conditions, we could only try to imagine it. With Drew, we actually discussed this part many times, exchanging opinions, ideas and trying to visualise what to expect and what to do. In our heads, we reviewed every step of the installation of these covers and we were able to complete the operation in time. And this is what we did.

While the giant image of Luca Parmitano holding that 7/8” wrench in his hand occupies the giant screen, an explosion of relief, (Fig. 6) gratitude and pure joy pervaded every face in the AMS POCC at CERN. A relieved and as joyful as rarely can be seen (Fig. 7) Sam Ting declared: “William Gerstenmeier, head of human spaceflight at NASA, thought this it was such a complex thing to do, because it was totally unprecedented. But NASA has to learn to do new things in space, because if they want to have a base on the Moon, or go to Mars, for instance, you need this type of things. We are very grateful to the astronauts who accomplished a fantastic feat, an almost impossible mission.”

Fig. 6

(Photo credit NASA)

Luca Parmitano outside the ISS with the wrench and tools in his hand

Fig. 7

(Photo Credit CERN-PHOTO-202001–021-40)

Samuel C.C. Ting in the POCC after the successful repair which extended the life to AMS-02 on the ISS

Thanks to that “SIMPLE 7/8’’ wrench, Parmitano was able to solve a problem that had not occurred during the training in the labs. The astronaut considers the image of him holding “the right key” as the symbol of the entire AMS spacewalk programme. “I liked that photo immediately, because it showed that complex problems are often solved with the ingenuity of the simplest tools such as a lever, first used by Archimedes and represented by that wrench, which allowed us to repair such complex equipment as AMS-02 in this case.”

Ken Bollweg describes his personal feelings during those moments while he was at Houston control center by recalling the “horrible thoughts” he had when he saw the first fitting checked and leaking. “I was really shocked and I was trying to understand what could be wrong in the procedure. Then I was a bit relieved when Luca and Drew tested the rest of them, and one by one they were fine. So, when the other seven were tested fine, I thought that it was just number 5, so we could try to fix it. When I saw the leak again after the additional tightening, I was really astonished and puzzled: we had never seen it before! But then when, after another try, it was fine, all was ok in every sense and the system is OK still now. The leak rate is so low that it is currently undetectable by the methods we have.”

Back in the POCC in Geneva, the next two days were busy although not tense at all. Zhan Zhang, the MIT lead engineer of the « Upgraded Tracker Thermal System» recalls [9]: “After the successful EVA, we knew our cooling system was really tight, so we had a lot of confidence for the following operation, which we carried out the next day: filling the system with CO₂, so that we could run it. It took 10 h to fill our circulating loop with 1.3 kg CO₂, plus another 6 h to check. On the following Monday, we could finally run the pump. Then we monitored the system for 24 h, before we could start it up for good. And that too was very successful. Finally, we were able to power on the tracker 100% and it started taking the science data. Since then, It has been running 100% smoothly. We also switched the 3 new pumps to check their performance after the launch, and they all behaved normally and smoothly. By noon on Monday, the entire detector started working in full configuration [10,11].” Giovanni Ambrosi, AMS physicist from INFN, also in the POCC, is relieved to say that: “We are now taking data with the full detector running in nominal conditions and we are really happy, because we now have ahead of us another 10 years and more of good data for science and an increase in the reach of energy where we are able to measure cosmic rays [12].”

Samuel Ting is confident the detector is entering a new era of discovery, another twenty years of new life. “We have measured many particles, electrons positrons, protons, antiprotons and all the nuclei, they all have distributions as a function of energy. None of the distributions we measured, agrees with current theoretical model. Everything was measured with extreme precision—excess of positrons, antiprotons, antimatter- nobody had expected. All the distributions with energy of carbon, oxygen, helium, lithium, beryllium, boron nuclei, all their distributions, the behaviour of their distribution with energy is totally unexpected. We are entering a region nobody has never been in before [13].”

Since the day of restart in January 2020, AMS-02 has been working 24/7, the silicon detector is back to its excellent performances, the cooling system is working wonderfully well and the experiment is collecting data without a glitch. For a few moments, the physicists and engineers who worked on this fantastic repair adventure feared the worst, as the risk that AMS-02 could never be put back into operation had begun to become more and more real. But the careful preparation of the astronauts and the ingenuity of those who developed the tools necessary to work safely in space, made the success of the mission.

Eventually, all it took was the right key at the right moment to give AMS-02 the possibility to continue making science and—who knows—maybe history!