Home office for telescope operators

October 2022

Previous | Index | Next

In the last few years, many people have adjusted to working remotely. If you have a desk job, then sometimes all you need is a computer and a stable internet connection. But how do you work from remote when your office is a telescope site on the other side of the world, 100 km away from the nearest city?

The H.E.S.S. telescopes are operated nightly by a team of two or three so-called 'shifters', whose job includes preparing the telescopes in their shelters, operating the telescopes, cameras, and auxiliary instrumentation, and monitoring the weather to ensure that the conditions are safe for operations. Traditionally, these have been young collaboration members from other parts of the world, who then spend a month at the H.E.S.S. site on shift. This allows them to get hands-on experience with the telescopes and gain a better insight into all aspects that affect measurements when obtaining the data. However, with the onset of the pandemic in 2020 and the closing of borders, it became difficult if not downright impossible to arrange for international, non-Namibian shifters to reach the site. During this time, the collaboration has leaned heavily on students at the nearby University of Namibia, who — along with our invaluable local crew members — have been instrumental in allowing the telescopes to continue to take data over the last few years. However, this also places a lot of pressure on our Namibian collaboration members. In order to ensure continuous operation of the telescopes, in 2021, the H.E.S.S. collaboration started to set up a Remote Operations room at DESY in Zeuthen, Germany.

While the observation schedule is mostly calculated beforehand, the shifters still have to react to realtime conditions (e.g., bad weather) as well as changes to the schedule, such as Target of Opportunity observations of transient astronomical sources. At the H.E.S.S. site, there is a control room that contains three main computers that the shifters use in order to perform these tasks: One computer for controlling the telescope array, and two others for communication and secondary monitoring. These computers have one to three screens each to accommodate the shifters' multiple tasks. There are also nine additional screens that show statuses; some of these monitor the health of the telescope and camera parts, some provide information on weather conditions such as cloud coverage or reports of nearby lightning, and some show the properties of the astronomical sources that are scheduled to be observed.

In setting up a Remote Operations room, all of the monitoring information must be made available there, and the setup imitated as much as possible. The connection needs to allow the remote shifter(s) to not only see what is happening on the on-site computers but also interact with them. The job of the remote shifter would be to support the on-site shifters and take over telescope operations when necessary. However, this raises some interesting questions: How do the on-site and remote shifters know who is currently operating the main computer to avoid sending conflicting commands or fighting for control of the mouse? What happens if the remote shifter is controlling the operations but the internet connection to the site drops?

fig1
Fig. 1: The on-site control room usually seats two or three shifters, who work together to monitor the telescope operations. The top row of monitors shows the statuses of the array components, while the bottom row allows for controlling the array, communication and secondary monitoring. (Photo by Sennae Kankondi)

Additionally, the control room on-site is 9m x 6m in floor area, while the remote operations room is less than half that size at 4.5m x 4.5m, and so creativity was needed to replicate the setup. One major difference is that the on-site shifters are usually two or three to the room, while the remote shifters would be one or at most two, so that the screens had to all be visible from a single seated position. The solution was to mount the screens to a wall in two rows of four instead of one row of nine, and to place the ninth screen on the side. The remote shifter's desk has another three screens that mirror the three screens of the main computer in the on-site control room, which contains the Graphical User Interface for controlling the telescope array. The remote shifter has another computer primarily for communication, including an always-on communication link with the on-site control room. To help ensure a layer of safety, there is a Remote Control Status GUI on the main shifter's computer that allows the shifters (both on-site and remote) to declare who is currently in control of operating the computer, as well as a watchdog that looks out for a loss in internet connectivity and alerts both sides.

fig2
Fig. 2: The remote control room is less than half the size of the on-site control room, so the monitors are arranged in two rows rather than one and the number of active computers is reduced from three to two. (Photo: DESY, H.E.S.S. collaboration)

Telescopes like H.E.S.S. depend greatly on the efforts of the shifters who oversee the nightly operations. By setting up a Remote Operations room, this takes some weight off of Namibian H.E.S.S. members and lets the rest of the collaboration better support them from afar. Following extensive tests, remote control observations can now be conducted. The operations of the array are less dependent on a regular stream of observers traveling to Namibia. This provides safeguards in case of emergencies and paves the way for operations with reduced intercontinental travel and associated environmental impact. And as the world adapts to new situations, so too must astronomical observatories like H.E.S.S.