Keywords

1 Introduction

CubeSats have the demonstrated potential to contribute to commercial, scientific, and government applications in remote sensing, communications, navigation, and research at a fraction of the size, development costs, and launch costs of the large, exquisite, multifunction satellites designed to support Cold War military requirements. Poghosyan et al. (Poghosyan and Golkar 2017) and Selva and Krejci (2012) conducted reviews of the recent history of CubeSat missions and surveyed CubeSat contributions to the scientific and experimental communities with the goal of determining the applications for which CubeSats are best suited. Missions such as Earth science, astrophysics, in situ laboratory applications, and technology demonstration have already benefitted from CubeSat contributions (Poghosyan and Golkar 2017; Selva and Krejci 2012). CubeSats offer significant advantages in terms of reduced development timelines and development costs. The small size and weight of CubeSats allow multiple satellites to be launched on the same rocket, thus greatly reducing launch costs (Selva and Krejci 2012). However, the reduced size, weight, and power margins inherent in CubeSats also have disadvantages. Smaller satellites are typically limited to single payloads or functions. The reduced functionality of CubeSats requires larger numbers of satellites to achieve the same performance. The mitigation of these disadvantages has been the subject of a significant body of research, such as Altinok et al. use of decision forests to conduct image analysis onboard a CubeSat (Altinok et al. 2016), Chang et al. and Pu et al. exploration of super-resolution through neighbor embedding (Chang et al. 2004; Pu et al. 2009), Denby’s and Lucia’s use of on-orbit edge computing to increase CubeSat efficiency (Denby and Lucia 2019), and Lüdenmann et al. employing sub-pixel image registration on a nanosatellite (Lüdenmann et al. 2019).

For traditional electro-optical (EO) imagery applications, high resolution (HR) requires large lenses and long focal lengths, which in turn require large satellites to support them (Buzzi et al. 2019). Past research has demonstrated that on-board image processing techniques can make more efficient use of limited satellite resources (Altinok et al. 2016; Blaschke et al. 2014; Denby and Lucia 2019; Lüdenmann et al. 2019). The continued miniaturization of electronics makes it increasingly possible to apply these preprocessing algorithms to CubeSat missions (Denby and Lucia 2019; Edeler et al. 2011; Lüdenmann et al. 2019). Work in pixel registration (Lüdenmann et al. 2019), feature classification (Chia et al. 2015), parallel computing (Denby and Lucia 2019), and radar interferometry (Hacker and Sedwick 1999) has laid the groundwork for the collection of EO imagery using multiple CubeSats flying in close formation.

Despite the data handling improvements, there remains one fundamental limitation of CubeSats for EO imaging applications: the small lenses and short focal lengths result in imagery with low spatial resolution. These low resolutions (LR) are sufficient for scientific applications such as weather forecasting and agricultural assessments (Poghosyan and Golkar 2017; Selva and Krejci 2012), but are insufficient for defense mission planning and intelligence operations. There are two primary methods for improving spatial image resolution: hardware solutions that focus on improved camera capabilities and analytical methods that focus on software solutions (Khattab et al. 2018). Hardware improvements are often restricted by cost, large size, or technology readiness limitations – all three of which we’ve identified as being incompatible with the CubeSat concept. Additionally, optical imaging hardware is subject to the Rayleigh criterion in which light diffraction limits the best possible resolution (Lee and Ashok 2019; Sprigg et al. 2016). Thus, a computational algorithm solution is required to improve EO spatial resolution of CubeSat images.

For reference, CubeSats are manufactured in a variety of form factors that are all based on a 1U form, which is a 10-centimeter cube (Fig. 1a) (Space Flight Now https://spaceflightnow.com/2015/10/16/nasa-to-fly-cubesats-on-three-new-commercial-launchers/). Other CubeSat form factors are based on scaling that 1U form factor, the most common variations of which are 2U, 3U, and 6U form factors. A 3U CubeSat is a 10 cm × 10 cm × 30 cm satellite, roughly the size of a loaf of bread. Figure 1b, c is 2U and 3U CubeSats, respectively. Figure 1b, c images are excerpted from a Table of Commercial CubeSat Constellations (NanoSats Database https://www.nanosats.eu/tables#constellations).

Fig. 1
figure 1

(a) NASA file photo of a 1U form factor CubeSat (Space Flight Now https://spaceflightnow.com/2015/10/16/nasa-to-fly-cubesats-on-three-new-commercial-launchers/); (b) SatRevolution (REC) CubeSat, 2U and 6U form factor, Earth imaging 50cm resolution; (c) Plant Labs (Dove) CubeSat, 3U form factor, Earth imaging 3.7 m resolution (NanoSats Database https://www.nanosats.eu/tables#constellations)

Full size image

This paper reviews the previous research on super-resolution (SR) and proposes potential applications of super-resolution to CubeSat EO imagery. Sections 2 and 3 discuss the two main categories of super-resolution: single-image super-resolution and multi-image super-resolution, respectively. Section 4 proposes potential applications of super-resolution to a CubeSat imagery system architecture, followed by concluding remarks in Sect. 5.

2 Single-Image Super-Resolution

Single-image super-resolution (SISR) is a relatively recent field of research and concerns the estimation of an HR image from a single LR image (Qureshi et al. 2012). SISR requires a training database of LR and HR pairs with specific features and segments common to both and annotated for machine learning algorithms. There are three main categories of SISR algorithms: interpolation-based algorithms reconstruct HR images using existing pixels to interpolate probable missing pixels; reconstruction-based algorithms use a priori knowledge (down-sampling, blurring, and warping) to recover the HR image; learning-based algorithms use dictionary pairs of training and testing images to estimate HR images (Yao et al. 2020).

Guo et al. (2019) developed a generalized image restoration neural network called the deep likelihood network (DL-Net). This research is focused on image restoration tasks that aren’t limited to narrow applications based on the original neural network training data set. Typically, these training data are generated by intentionally degrading a high-resolution image; these training sets tend to result in poor generalization by the network. The authors build upon single-image super-resolution (SISR) through neighbor embedding as developed by (Chang et al. 2004; Pu et al. 2009; Timofte et al. 2013) to design an image interpolation algorithm capable of generalizing from a variety of image degradation types. This process makes use of the local geometry within neighboring image segments to extrapolate from low-resolution to high-resolution images.

Fan and Yeung (2007) expand on the SISR application by studying the distribution of small image patches to determine the types of image structures (edges, corners, outlines, etc.) that are likely to occur in the image. The authors assume similar local geometries between the image patches to neighborhood relationships between the low-resolution and corresponding high-resolution training images. By retaining image patch geometric relationships and inter-patch relationships with neighbors, the authors are able to generate high-resolution images that are both accurate and smooth.

Ismail et al. (2020) explore applications of super-resolution where there is an insufficient quantity or quality of neural network training data. Propose the use of adaptive network-based fuzzy inference system (ANFIS) to interpolate effective mappings from low-resolution to high-resolution images, given sparse training data.

Al-Mansoori and Kunhu (2013) evaluate three well-known interpolation techniques: nearest neighbor, bilinear, and bicubic interpolation. In all three cases, the authors use a single low-resolution image and perform the interpolation techniques to compare the results. The initial experiment limited the magnification factors as a proof of concept. In all example-based super-resolution results, the bicubic interpolation yielded smoother edges and more detailed high-resolution images.

Since SISR methods by definition only use one LR input image, the SISR algorithms tend to be computationally faster because they don’t require motion estimation and pixel registration between input images (Bätz et al. 2015). However, SISR algorithms have a fundamental limitation in that the training set must be similar to the desired HR image in order for the reconstruction algorithms to be effective (Qureshi et al. 2012). Additionally, the possible resolution enhancement is limited compared to multi-image super-resolution techniques (Bätz et al. 2015).

SISR techniques are fast, less computationally intensive, and capable of producing sharp HR images for specific applications. However, SISR methods do not generalize well to large-scale problems and require large databases of LR/HR image pairs in order to estimate and reconstruct HR images. Additionally, as discussed in the following section, SISR methods cannot take advantage of relative motion between a series of LR images to achieve better resolution improvements. These limitations mean that SISR techniques cannot leverage all the advantages of satellite imagery and therefore are not ideal for CubeSat SR applications.

3 Multi-image Super-Resolution

Multi-image super-resolution (MISR) is a well-studied problem which typically consists of three stages: registration estimates the shifts between LR images, relative to a reference image, with sub-pixel accuracy; interpolation obtains a uniform HR image from a nonuniform composite of LR images; and restoration removes the image blur and noise. MISR can be further subdivided into frequency domain techniques and spatial domain techniques (Qureshi et al. 2012). The relative motion between LR input images produces the sub-pixel shifts necessary to achieving higher-resolution enhancement by accounting for information from adjacent image frames (Bätz et al. 2015).

Sub-pixel motion must be present in the input sequence frames in order to realize the best possible resolution enhancement using MISR techniques. However, this sub-pixel motion also requires highly accurate motion estimation in order to avoid introducing artifacts from erroneous motion vectors (Bätz et al. 2016a). Bätz and colleagues have published a series of papers (Bätz et al. 2015, 2016a, b, 2017) proposing various methods to minimize the introduction of these artifacts and to improve the overall image enhancement results. Their proposed methods include locally adaptive denoising (Bätz et al. 2017) which introduced a step between interpolation and restoration, dual weighting (Bätz et al. 2016a) which employs both a motion confidence weight and a distance weight to resolve motion estimation errors, and hybrid SISR/MISR (Bätz et al. 2015) approach that employs both SR techniques but weights SISR more heavily in the case of static targets and MISR more heavily in the case of dynamic targets. All of these techniques showed significantly improved peak signal-to-noise ratio (PSNR) compared to more traditional SR techniques.

Mandanici et al. (2019) applied an MISR algorithm to terrestrial thermal images using a novel registration technique that computes the sum of normalized distances (SND) to a given reference image. A higher SND denotes less accurate registration. These images are then excluded from the interpolation stage, based on an SND threshold value. This methodology has the added benefit of coherence analysis to identify reconstructed pixels that are less reliable which, when combined with the image frame rejection criteria, resulted in improved thermal image resolution.

Some researchers (Cohen et al. 2019; Zhang et al. 2009) are exploring the use of super-resolution in microscopy applications to obtain image resolutions beyond Rayleigh criterion diffraction-limited resolution. Cohen et al. (2019) investigate the resolution limit of image analysis algorithms. Zhang and colleagues propose a method to capture random micro-displacement offsets of multiple images without the need for a high-cost, precision mechanical device (Zhang et al. 2009). These precision offsets allow superior HR image reconstruction compared to the more expensive fixed micro-offset technique. While this research focused on microscopic image enhancement, it would be interesting to research whether their techniques may have applicability to CubeSat image resolution enhancement.

MISR techniques are able to produce superior resolution enhancement by taking advantage of the sub-pixel motion between consecutive LR images by accounting for information from adjacent pixels, given a sufficiently accurate motion estimation algorithm. However, MISR approaches have a tendency to present ill-posed problems, either due to an inadequate number of LR images or poor estimation of image capture artifacts, such as blur (Khattab et al. 2018). Despite this limitation, past research has demonstrated that regularization techniques (Irani and Peleg 1991) help to invert an ill-posed problem to a well-posed problem (Khattab et al. 2018). Overall, MISR techniques offer better potential to take advantage of CubeSat capabilities.

4 Applications to CubeSats

Researchers have developed a number of algorithms designed to improve image quality. Three common algorithms are pixel averaging, super-resolution, and mosaicking (Lüdenmann et al. 2019). In addition to lens size and focal length, CubeSat downlink data rate and onboard storage capacity are two other limiting factors in electro-optical imaging (Altinok et al. 2016). In order to apply averaging, super-resolution, or mosaicking, the CubeSats must either downlink large image files to be processed terrestrially or the CubeSats must have real-time access to a memory-intensive catalog of georectified reference images (Altinok et al. 2016; Lüdenmann et al. 2019). In either case, the requirements are impractical for use on-board a CubeSat. Lüdenmann et al. (2019) developed a method to use a combination of correlation and regression algorithms to identify the geometric transformations between consecutive images on-board the CubeSat while keeping the data downlink requirements within the size, weight, and power restrictions imposed by the CubeSat standard.

MISR techniques are most effective when sub-pixel motion is present in the input sequence frames. This attribute of MISR makes it particularly useful in CubeSat EO imagery applications since satellites in Earth orbit are in constant motion. A formation of CubeSats can capture multiple images of the same target area on Earth. The varying locations of the CubeSats within the formation combined with the orbital velocity of the CubeSats inherently provide the input image offset required for successful MISR application.

Figure 2 depicts a high-level operational view (OV-1) of one possible system architecture for CubeSat EO SR applications. This architecture assumes a predetermined CubeSat formation, optimized (Buzzi et al. 2019; Chia et al. 2015) for the number of CubeSats and orbital parameters necessary to collect the LR images. The LR images are then preprocessed for motion estimation and pixel registration (Bätz et al. 2015; Khattab et al. 2018; Lüdenmann et al. 2019) before being segmented (Blaschke et al. 2014; Denby and Lucia 2019) and input to a MISR computational algorithm for HR pixel interpolation. The processed image segments are then recombined, the HR image is restored, and a quality verification process certifies the resulting HR image. One possible method of quality verification is to compare the resulting resolution improvements, from the proposed architecture, against a theoretically perfect Rayleigh-limited image from current, state-of-the-art, CubeSat imaging hardware.

Fig. 2
figure 2

OV-1 of potential system architecture for CubeSat super-resolution

Full size image

Since traditional resolution enhancement algorithms are impractical for CubeSat applications and techniques exist to efficiently register LR images prior to downlink, MISR becomes an attractive technique to improve the spatial resolution of CubeSat images.

5 Conclusion and Future Research

This survey paper reviews recent research published regarding SR and discusses the advantages and disadvantages of the two primary SR techniques: SISR and MISR.

SISR is a relatively new research field and consists of three main categories of algorithms: interpolation-based algorithms, reconstruction-based algorithms, and learning-based algorithms. SISR algorithms tend to be computationally faster and provide good resolution enhancement for specific applications; however, they do not generalize well and cannot take advantage of information from adjacent pixels in sequential image frames. These limitations mean that SISR is not the best choice for CubeSat SR applications.

MISR is a well-studied problem that consists of geometric registration, interpolation, and restoration to derive a single HR image from multiple LR images. The relative sub-pixel motion between input image frames is the key to achieving high-quality resolution enhancement. However, the estimation of that motion, during the pixel registration process, must be highly accurate to avoid introducing error artifacts into the HR image. An additional challenge with MISR techniques is that an inadequate number of LR input images or poor estimation of image capture artifacts can contribute to making the MISR approach an ill-posed problem. Care must be taken to understand the constraints and limitations of the imaging hardware and to apply regularization techniques to define a well-posed MISR problem.

The inherent limitations of CubeSats and the nature of satellite orbits make MISR an attractive technique for improving CubeSat EO spatial resolution. Additional research is required to develop a resolution enhancement model that can enhance image resolution sufficiently enough to extend the utility of CubeSat images to defense mission planning and intelligence operations. CubeSats have already demonstrated their potential to contribute to scientific discovery; extending that potential to include that satisfaction of national defense requirements will provide intelligence value at a fraction of the current costs of large satellites.