Keywords

1 Introduction

The importance of data analysis in astrophysics has become indisputable as data gathering techniques have gotten larger and faster. The spotlight has thus begun to shine on data science and software development as key supporting fields of astrophysics. Given the importance of data analysis pipelines for telescopes of all kinds, we have developed a photometric pipeline, Photometry+, for the Great Basin Observatory (GBO), a 0.7-m robotic telescope located in the Great Basin National Park in Nevada [1]. The GBO is the first research grade observatory located in a U.S. National Park, shown in Fig. 1, enjoying dark skies free of light pollution. It partners with universities in order to inspire researchers and students alike to enjoy astrophysics.

The goal of a photometric pipeline is to analyze raw images of the night sky that are taken by counting photons hitting a charge coupled device (CCD) to calculate the magnitude, a measure of the flux or brightness of a target star or celestial object (this paper will refer solely to stars going forward). The measurement of fluxes represents one of the most basic deliverables from astronomical images, which in most cases is one of the key pieces of information to determine the physics driving astronomical phenomena. For example, supernovae represent exploding stars, which are discovered via the sudden appearance of a bright source in the sky. Or, monitoring changes in the flux of a star over time can (sometimes) indicate changes in its surface temperature and/or radius, or provide information on the orbital parameters of certain binary systems. However, in between those images and the result are several steps that can be tedious to do by hand. For each sky image there are an additional two to three calibration files that are used to remove noise and pixel-to-pixel variations from the raw CCD images. After the noise is removed, noise from cosmic and terrestrial background radiation needs to be subtracted as well by taking a median of photons counted from “blank” sky. Once the error is reduced by removing noise, the magnitude of a target star can be calculated by comparing it to other stars of known magnitude and taking an average of the results calculated for each comparison. The process of comparing a target object to multiple reference stars can take up to a half hour per image to do by hand. However, by automating the process Photometry+ can perform these same tasks in less than 20 s per image.

Fig. 1.
figure 1

The Great Basin Observatory, Nevada (courtesy of the Great Basin Observatory).

Full size image

To prevent Photometry+ from being a black box tool, we focus on the human-computer interaction (HCI) components of the program. The HCI goals of the proposed pipeline are to create a graphical user interface (GUI) that is easy to use, gives astronomers control over the program, increases confidence in the results of the program, and can be used to teach students the process of differential photometry. To validate the accomplishment of these goals three user studies were conducted, two of which have been used to guide the development of Photometry+, and a final user study to validate that Photometry+ can be used as a teaching tool for the complex task of differential photometry. The first two user studies tested how long it takes inexperienced and experienced astronomers, respectively, to use the GUI to complete the task, the parts of the process they found confusing, their confidence in the tool, and how much they feel they learned about differential photometry through using the tool.

Using the feedback received from the first two user studies, a public-release version of Photometry+ was developed and the final user study was completed with this version. The final user study tests the hypothesis that user-guided development of a GUI can be used to create scientific tools that are useful for (i) experienced astronomers completing routine results generation, and (ii) beginners looking to learn more about the process. User confidence in the results was measured both in terms of the control users feel they have over the program, and in their confidence in the accuracy of the results obtained. We also present examples of the application of Photometry+ for monitoring variable objects. Photometry+ provides a new look at photometric pipelines with the user and HCI principles in mind. This user-oriented design allows Photometry+ not just to be a tool for experienced astronomers, but also a teaching tool for students and others looking to learn differential photometry. While Photometry+ was designed with a specific scientific process target, this approach is generalizable enough to be used on any tool seeking to teach a difficult scientific concept through performing a task.

2 Background and Related Works

The age of big data has changed the way many fields are able to operate, including astronomy and astrophysics. Telescopes around the world produce a colossal amount of data, with some telescopes producing data in the exabyte range [26]. It is only natural that the large amount of data needing to be processed has put an emphasis on data analysis pipelines of all sorts. One kind of data analysis pipeline, the photometric pipeline, focuses on performing different kinds of photometry on telescope images to calculate the flux of stars. Most of these photometric pipelines are not generalized, but rather built with a single telescope or telescope system in mind (with only a few exceptions [17]). Some of these pipelines are open source and, although they are designed for a specific telescope they allow for other researchers to use modified versions of their pipeline. An example of this is the Legacy Survey of Space and Time (LSST) Science Pipeline [11], which is designed for the Vera C. Rubin Observatory but whose code is available and modifiable for anyone interested in it. One of the broadest photometric pipelines available is designed for the All Sky Automated Survey for SuperNovae (ASAS-SN), which is not a single telescope but a network of telescopes designed to work together to image the entire night sky [12]. The ASAS-SN photometric pipeline works through a web portal that allows users to generate a light curve of anywhere in the night sky, assuming that there is data for that space at the time when the user wants to observe. However, large telescopes and surveys are not the only systems with automatic pipelines. Smaller telescopes for different purposes, like the Watcher robotic telescope in Boyden Observatory, have photometric pipelines designed for them [7]. Some photometric pipelines are even designed with a backlog already in mind, such as the pipeline created for the Robotic Optical Transient Search Experiment (ROTSE)-IIId archival data [8]. That photometric pipeline is used almost exclusively for archival data, though that is not always the case for pipelines designed to handle archival data. The pipeline for the Wide Field Astronomy Unit (WFAU) is built to parse data fast enough to continually process new data in addition to processing archival data that has backed up [6]. Clearly the creation of photometric pipelines for telescopes around the world is widespread, and with Photometry+ there is now a new pipeline for the GBO telescope as well [24].

While the general goal of all photometric pipelines is to perform photometry on images from telescopes, many pipelines are made with additional goals in mind. For instance, some pipelines are designed to cater to specific types of stellar objects rather than a telescope. One such example, the Pippin pipeline, is an open source pipeline designed for supernova-based analysis [9]. However, not all of the pipelines with additional goals are focused on certain stellar objects. Some of these pipelines instead direct their attention toward data quality or other mechanical parts of the astrophotography process. One such example is a pipeline that uses a convolutional kernel to reduce the effect blurry images have on the final photometric calculation [10]. Like these other pipelines, Photometry+ includes more than the standard goal of performing photometry. Unlike these other pipelines however, the additional goal of Photometry+ does not focus on space or data correction, but rather on the human element of interacting with the pipeline.

Scientific software and usability have always had a complicated history. Software developers can often be entirely absent when it comes to making the computational tools that scientists use on a daily basis. Thus, good design practice can often be neglected. This problem goes back to the early days of software being used as a scientific tool with observers noting that the creation of user interfaces (UIs) for scientific tools are ill-funded, poorly understood, and less emphasized [5]. And although it is not a focus in scientific software, usability-centered design can have many benefits including reducing user errors, reducing the time it takes to learn to use a tool, and making software more generalizable. Adding user-centered design principles to scientific software doesn’t have to be difficult either, as studies have shown beginning the process doesn’t take very long and brings many benefits [16]. In recent years usability has become a focus of some astronomy developers, such as in visualization software for radio astronomy [21]. Rampersad et al. used user-guided development to create their visualization tool, holding user studies in between prototypes and molding it to be more user-friendly and easy to use. We also argue that this can be taken a step further by combining these development user studies with research studies that validate what a pipeline can do for the user. Pipelines can be more than just user-friendly; they can be a valuable teaching tool for students looking to learn complicated scientific processes. Photometry+ is a new step in the combination of HCI and astronomy, as a tool that obtains high quality results and is easy to use. It is also a teaching tool for those looking to learn about differential photometry.

3 Photometry+ Design

Photometry+ is a photometric pipeline that performs differential photometry using Python. It can be used in one of two ways, with both methods working independently of each other. The code that runs the system can be run in the Python terminal, and the backend of the program is fully functional on its own. The second method, that this paper covers in more detail, is the GUI for Photometry+. The GUI makes the program accessible even to inexperienced users and is focused on making differential photometry usable and easy to learn.

Fig. 2.
figure 2

Abridged version of the steps Photometry+ takes to perform differential photometry.

Full size image

3.1 System Design

Photometry+ performs all the stages of differential photometry with minimal user input. Figure 2 shows a flowchart representing the steps of differential photometry performed by the program. To begin, users simply need to upload a telescope image and the stellar coordinates for the target star whose magnitude will be calculated. At this stage users can also optionally add calibration files for automatic calibration of their images and change the default settings of the program. Examples of the settings that can be changed include setting an Astrometry.net API key, choosing how to calculate the radius of the target star, and choosing which VizieR [18] catalog (or SIMBAD [25]) to search for reference stars. Once the user has chosen their settings the program can be run.

Autonomous differential photometry follows the same steps as manual differential photometry. These steps are calibration of the telescope image, finding background radiation noise to subtract out, locating reference stars of known magnitude in the image, and comparing the reference stars to the target star to calculate a comparative magnitude of the target star. These individually calculated target star magnitudes are averaged together to create the final target star magnitude for that image. An error is also calculated for this magnitude through a user’s choice of standard deviation, weighted magnitude, or a jackknife method for photometric uncertainties [2]. This process is repeated for many images taken at different points in time. After calculating the magnitudes for every image in a set, Photometry+ generates a publication-quality light curve (like the light curve in Fig. 3, a graph of the magnitude of a star over time.

Fig. 3.
figure 3

Light curve made with Photometry+ utilizing GBO data (top) matched with data from AAVSO, where the Photometry+ results are green and AAVSO results are blue (bottom). The match in the bottom figure demonstrates that Photometry+ works with a comparable accuracy to other top photometric pipelines.

Full size image

To create Photometry+ and allow it to be robust required several external resources. Like many other astronomy tools, Photometry+ used the Astropy library [3, 23], photutils [4], and DAOPHOT [22]. Additionally, Photometry+ pulls information from the APIs for Astrometry.net [15], VizieR, and SIMBAD. These external dependencies are shown in the context diagram presented in Fig. 4.

Fig. 4.
figure 4

The context diagram for Photometry+ displaying the external dependencies of the system.

Full size image

3.2 User Interface Design

The user interface for Photometry+ was designed with open source PyQt5 [20]. The main components of the user interface include a page to create a new project, shown in Fig. 5, a “My Projects” page where users can view their already created projects, an “About” page where users can learn more about the program and its creators, a page where users can change their default settings, and a page with frequently asked questions and a contact form. This user interface was designed with user-guided development, following the style of user testing outlined by Steve Krug [13, 14]. Two development user studies were conducted with 3 and 4 participants respectively, and feedback from those user studies were used to improve the GUI to better accomplish the HCI goals of the program.

4 Methodology

The main experiment described in this paper involves a user study wherein participants were asked to perform tasks related to differential photometry with the fully developed Photometry+. These tasks were done with data from the GBO telescope and a SIMBAD page, shown in Fig. 6, containing location information for the star DO Dra.

Fig. 5.
figure 5

The “New Project” page of Photometry+ filled out.

Full size image

4.1 Participants

The recruitment for this user study targeted physics and astrophysics students, faculty, and researchers. Recruitment messages were sent out at the 237th meeting of the American Astronomical Society, to the Great Basin Observatory users committee, to the University of Nevada, Reno (UNR), Department of Physics, the UNR astronomy club, and other local astronomy groups. This targeted recruitment ensured that the participants using Photometry+ were a part of its final target audience in order to accurately test whether astrophysics students and researchers who may not use differential photometry often can perform the process with this tool. Participants filled out a pre-study survey that collected demographic information, including gender, education level, astronomy experience, and photometry experience.

4.2 Apparatus

Due to the unfortunate COVID-19 pandemic, all user studies for this research were conducted remotely via the video and messaging application Zoom [27]. Participants were asked to take remote control over the study administrators computer to take a brief quiz, perform some tasks on the program and then take another brief quiz. The tasks performed with Photometry+ were based entirely on the user interface of the program and no interaction with code or Python was required. Each user study took less than forty-five minutes in total.

4.3 Procedure

To maintain consistency between every study, a script was followed when performing the user studies and all emails sent to participants were the same. This ensured that each participant had the same experience to minimize confounding variables. Every study followed the following procedure:

  1. 1.

    The participant was asked to sign a consent form and recording release.

  2. 2.

    The participant filled out a pre-study survey with a unique participant ID.

  3. 3.

    The participant joined a Zoom call with the study administrator.

  4. 4.

    The participant was given remote access to the testing machine with the quizzes and Photometry+ available.

  5. 5.

    The participant was given a differential photometry quiz that briefly assessed their prior differential photometry knowledge.

  6. 6.

    The premise and purpose of the study were briefly explained.

  7. 7.

    The participant was given the task list for the study and the SIMBAD page mentioned above.

  8. 8.

    The participant performed the tasks on the list using Photometry+.

  9. 9.

    The participant was given the same quiz as they took at the beginning to assess the change in their differential photometry knowledge.

  10. 10.

    The Zoom call was ended.

  11. 11.

    The participant filled out a post-study survey.

Fig. 6.
figure 6

The example SIMBAD page that was provided to participants of the user study.

Full size image

The pre-study survey participants filled out concerned demographic information such as age, gender, education, and level of astronomy and photometry knowledge. The differential photometry quiz scored users out of 12 based on questions about photometric calibration, differential photometry steps, and other related knowledge. This quiz was given twice. The tasks the participants were asked to perform included creating a new Photometry+ project targeting DO Dra, changing default settings, and observing the results of their project. After the users finished performing tasks and retook the quiz, they were given a post-study survey that asked questions concerning the look and feel of Photometry+, their confidence in its results, and whether they learned more about differential photometry by using it.

4.4 Experimental Design

Independent variable: Prior use of Photometry+

Name: Photometry+ Exposure

Levels: Before use, after use

Dependent variable: Differential photometry knowledge

The independent variable being manipulated was whether participants had been exposed to Photometry+ before or not. Thus, participants were measured on differential photometry knowledge before and after using the program for the first time. Changing the Photometry+ Exposure variable should produce a change in the dependent variable, differential photometry knowledge, as our hypothesis was that Photometry+ can teach differential photometry.

The amount of entry in this experiment is 12 participants \(\times \)2 administered quizzes for 24 phases. Thus this is a \(12\times 2\) within-subjects design.

5 Experimental Results

For each of the twelve research user study participants, demographic information was collected through the use of a pre-study survey, with the results shown in Fig. 7. 75% of the participants were in the 18–24 age range, with a few participants in the 25–34 and 45–54 age ranges. Additionally, 75% of the participants were male, similar to the demographics in the physics field [19]. Users came from a variety of education levels ranging from undergraduate students to graduate degree holders. The predominant operating system was Windows. As expected from our targeted recruitment, all participants had at least a little experience with astronomy, though a large percentage were less experienced with photometry.

Fig. 7.
figure 7

Pie charts representing the demographics of the study participants based on the pre-study survey each participant filled out.

Full size image

Using the post-study survey, we collected data from the participants concerning how easy it was to use Photometry+. For statements like “Photometry+ is easy to use”, “The user interface of Photometry+ is well designed”, “Photometry+ is intuitive”, and “Photometry+ is easy to navigate”, users consistently rated their agreement with the statement between “Strongly Agree” or “Agree”, which were 5 and 4 respectively, on our Likert scale. The average participant scores for those statements were 4.58 for “Photometry+ is easy to use” and 4.33 for “The user interface of Photometry+ is well designed”, “Photometry+ is intuitive”, and “Photometry+ is easy to navigate”. When asked questions concerning the aesthetics of Photometry+, users rated them 4.33 on average overall. Additionally, timing the user study showed that participants took an average of 10.66 min to complete their first photometry project, despite never having used the software prior.

The primary HCI goal of Photometry+ is to teach students differential photometry in addition to being an analysis tool. To assess this we administered a short quiz on differential photometry before and after the participants used Photometry+. As shown in Fig. 8, the mean score on the quiz that participants took after using Photometry+ measured at 81%, which is 15% more than the 66% mean score from before participants used Photometry+. This difference, when analyzed with ANOVA, is statistically significant (F = 7.54, p < .05). Additionally, on the post-study survey, participants were asked to agree or disagree with the statement “Photometry+ taught me more about performing differential photometry” on a Likert scale. The average value of the responses was 4.08 on a scale where five points was the maximum.

Fig. 8.
figure 8

Detailed information on the performance of participants on a differential photometry quiz before and after exposure to Photometry+.

Full size image

6 Discussion

The data gained from the experiments shows that Photometry+ achieves its HCI goals of being easy to use and of teaching differential photometry. Through directly measuring use time, we can ascertain that, even with little experience with photometry and no experience with Photometry+, participants could still perform differential photometry in a reasonable time span, with many of the participants expressing the sentiment that they could repeat the process more quickly if they needed to use the software again. By directly measuring improvement in differential photometry knowledge with the quizzes, we conclude that there is a statistically meaningful knowledge boost associated with using Photometry+. It is apparent that both HCI goals were achieved when looking at the directly measured data.

Photometry+ not only accomplished these goals, but also convinced users that these objectives were met. In addition to being able to use the program in about ten minutes, participants directly ranked Photometry+ as being easy to use and intuitive in the post-study survey. Likewise, participants on average agreed with the statement that they learned more about differential photometry from using Photometry+, which means that users are aware of the learning potential of Photometry+. Participants both measurably increasing their learning and feeling that they learned more effectively is a great endorsement of the power of software to support teaching and performing complex scientific tasks.

7 Conclusion

In conclusion, Photometry+ is the result of the combination of successful user-guided development techniques and HCI research into making an easy-to-use software tool that can double as a teaching tool for complex scientific tasks. This addresses a clear need for scientific software designed with end-users in mind, and can decrease the difficulty involved with learning new scientific methods. The methodology detailed in this paper could be used to create software for any variety of scientific tasks in a broad variety of fields, and our results show that it works for scientists of all levels of experience. Photometry+ is an open source program that will continue to be worked on, expanded and adjusted. Its ability to create accurate, high-quality light curves and teach students differential photometry makes it an excellent candidate for adoption by other telescopes or projects.