Keywords

1 Introduction

Although monitoring cameras are omnipresent, people’s concern for details still calls for better pictures, due to limited equipment costs and constraints, weather conditions, as well as target shooting distances.

Super-resolution (SR) refers to methods to upscale, upsize, and restore pictures or video sequences. Starting from low-resolution (LR) images, SR recovers a high-resolution (HR) one that offers enriched visual results while eradicating additive noise, handling the imaging detector sizes and optical sensor constraints. It is related to image fusion, registration, and mosaicking [1,2,3], and [4]. Some SR applications are described below.

  1. (i)

    Freeze frame and region of interest (ROI) zoom for perception and analysis.

  2. (ii)

    Resolution-enhanced automatic target recognition.

  3. (iii)

    In reconnaissance, several images from the same region can help to render a better resolution image.

  4. (iv)

    To enrich the resolution and create multimodal versions of pathological areas in medical imaging (ultrasound, CT, MRI, etc.) by merging data from several limited resolution images.

  5. (v)

    Fluorescence microscopy.

  6. (vi)

    Video standard conversion, e.g., from PAL-M to HDTV signal and from 4K to 8K.

  7. (vii)

    Forensics.

SR approaches try to model the quality loss when using LR cameras and then to solve an associated ill-posed inverse problem, which does not possess a forthright solution. SR imaging typically involves regularization, optimization, and extensive computations. SR works successfully when the LR images involve somewhat different views of the same object, that is, all the object knowledge exceeds the knowledge from a single frame. Motion estimation (ME) can help to upscale an image and infer the correlation between frames or patches. If an object is steady and appears identical in all frames, no additional knowledge is available. If there is movement or fast transformations, then the target will appear distinctly in different frames. This redundancy in LR frames assists the HR frame reconstruction.

Many factors limit the resolution of imaging systems due to the diffraction limit or resolution constraints associated with the geometry of the optical elements (c.f. Fig. 1). Among the geometrical restrictions resulting from image acquisition CCD cameras, one can cite the size and the shape, and the pitch of pixels can lead to errors. Video super-resolution (VSR) seeks to reconstruct an HR video sequence from an LR one [5, 6]. There are two types of VSR algorithms:

Fig. 1
figure 1

Imaging degradation process

Full size image
  1. (i)

    Multiframe complementary information (MCI) techniques use redundancy from different frames; and

  2. (ii)

    ME methods using data on moving objects such as motion vectors (MVs) or displacement vectors (DVs). VSR performance hinges on the accuracy of the ME framework.

Sparse reconstruction is an emblematic ill-posed inverse problem where the measurement error and the sparsity terms are used as two conflicting terms to be handled simultaneously inside an objective function. This framework brings in new challenges to the optimization task. Classical optimization and analysis techniques may perform SR disappointingly because it is an expensive optimization problem. This motivates the application of computational intelligence methods [4, 7, 8].

The VSR seeks an HR video that meets certain specifications. This problem can be recast as an optimization task, which pursues the optimal result consistent with quality requirements. For a given imaging degeneration model, optimization techniques can be used to discover the global minimum (or maximum) using a proper fitness function or cost functional.

This chapter examines particle swarm optimization (PSO) applied to SR imaging and VSR schemes relying on an image degeneration model explained in Sect. 2 [7, 9]. Experimental results using evaluation metrics show that PSO methods can improve both objective and subjective results.

2 The Image Degradation Model

Due to hardware limitations, the imaging system has imperfections and various types of degradations as shown in Fig. 1. The Point Spread Function (PSF) models some kinds of optical and mechanical distortions. As the image pixel results from integration over the sensor area, the restricted sensor density causes aliasing effects, limiting the image spatial resolution. These degradations are handled entirely or partly with different SR techniques (Fig. 2). This work uses the subsequent notation:

Fig. 2
figure 2

The observation model relating HR to LR images

Full size image
  1. (i)

    Upper case bold letters X and Y symbolize lexicographically ordered vectors for HR and LR frames;

  2. (ii)

    Lower case bold letters such as x and y stand for lexicographically ordered vectors for HR and LR image patches in that order;

  3. (iii)

    Underlined upper case bold letters show the result of a vector concatenation, e.g.,\( \underline{\varvec{Y}} \) is a vector concatenation of Y k , with k = 1, …, K where K stands for the number of captured LR frames by the camera; and

  4. (iv)

    Plain upper case symbols denote matrices, and simple lower case symbols refer to scalars.

If X denotes the desired HR image (i.e., the digitally sampled image), and Y k is the kth LR camera observation, then there are K LR frame versions of X where each LR observation vector Y k is related to the HR image X by

$$ {\varvec{Y_{k}}} = D_{k} H_{k} F_{k} \varvec{X} + {\varvec{V_{k}}}, $$
(1)

where the knowledge on motion for the kth frame is encoded in F k , H k reproduces the blurring effects, D k is the operator in charge of downsampling, and V k represents the noise. If Y is the observed image, then rearranging these linear equations into a large linear system yields

$$ \underline{\varvec{Y}} = {M}\varvec{X} + \underline{\text{V}}. $$
(2)

Matrices D k , H k , F k , and M are extremely sparse, unknown, and have to be estimated from the existing LR observations, which worsen the linear system ill-conditioned nature. Thus, regularization is always advantageous and frequently essential.

3 PSO Super-resolution

SR reconstruction has been a very active research area, and many techniques have been suggested to handle this matter. Figure 3 depicts the SR rationale.

Fig. 3
figure 3

SR rationale

Full size image

Before stating some PSO variants, a genetic algorithm (GA) will be stated, because some of its reasoning appears in hybrid PSO methods.

3.1 Genetic Algorithm (GA)

A GA [10, 11] is a soft optimization strategy to find solutions to optimization problems. They are global search heuristics methods motivated by evolutionary procedures such as selection, mutation, inheritance, and crossover (aka recombination).

GAs consider a population consisting of abstract representations (or chromosomes) of candidate solutions (or individuals) evolves to improved solutions. Habitually, solutions consist of binary strings having 0s and 1s, but other forms of encodings can be used. The evolution procedure begins with a random initial population of individuals and occurs in generations. The fitness of every single individual in the population is computed for each generation. Individuals are stochastically taken from the existing population using their fitness values as the selection criterion. Then, recombination is done along with perhaps randomly mutated individuals to produce a new population for the subsequent algorithm iteration. The algorithm ends once the maximum number of generations or an acceptable population fitness level is reached. If the procedure finishes because of reaching the maximum number of generations, then a proper result may be found.

In SR imaging, GAs can deal with local resolution complications [12, 13], even if it entails huge computational time and it lacks fine-tuning possibilities.

3.2 Classical Particle Swarm Optimization (CPSO)

CPSO is a simple population-based optimization scheme, which involves minimal computational effort. It employs a search motivated by a model of social influence and learning. Individuals emulate the success of their neighbors.

Consider a present diffuse population of size S known as a swarm. Each swarm member is dubbed a particle, and it as a point belonging to the search space. A particle group has a tendency to cluster at an optimized position (maximum or minimum). Therefore, to accomplish CPSO, each particle corrects itself by comparing its previous similarity measure (SM) to its neighbor SMs to reach the best result [14].

If f: ℝn→ℝ is the cost functional to be minimized, then the real number output corresponds to the result of the optimized fitness function, for the given candidate solution [15].

Given an unknown gradient \( \nabla f \) of f, an optimum solution x * for which f(x *) ≤ f(y), for all y in the search space, is sought. Alternatively, the maximization of a function h = −f can be performed.

At iteration k, x i ∈ ℝn is the position vector of the ith particle in the search space whose velocity is v i ∈ ℝn, p besti is the best x i for particle i, g best is the global best known position among all particles of the entire swarm, and w means the weight. c 1 and c 2 are acceleration constants whose pdfs are uniformly distributed in the interval sandwiched between 0 and 1.

In the CPSO-based SR method, x i is the transform parameter vector that has to be estimated, \( f\left( {\varvec{x}_{\varvec{i}}^{\varvec{k}} } \right) \) is the functional or cost function to be optimized, the \( \varvec{p}_{{\varvec{besti}}} \) is the maximum cost functional \( f\left( {\varvec{x}_{\varvec{i}}^{\varvec{k}} } \right) \) for each particle, and \( \varvec{g}_{{\varvec{best}}} \) corresponds to the best cluster.

CPSO‐based SR Algorithm

  1. (1)

    Initialize the swarm randomly with initial values for variables \( \varvec{x}_{\varvec{i}}^{0} \), \( \varvec{v}_{\varvec{i}}^{0} \), \( \varvec{p}_{{\varvec{besti}}} \), \( \varvec{g}_{{\varvec{best}}} \)

  2. (2)

    For each particle i, repeat until the whole population has been analyzed:

  3. (3)

    Set initial values for variables \( \varvec{x}_{\varvec{i}}^{0} \),\( \varvec{v}_{\varvec{i}}^{0} \),\( \varvec{p}_{{\varvec{besti}}} \),\( \varvec{g}_{{\varvec{best}}} \)

  4. (4)

    \( \varvec{v}_{\varvec{i}}^{{\varvec{k} + 1}} = w\varvec{v}_{\varvec{i}}^{\varvec{k}} + c_{1}\, \text{rand}\,[\varvec{p}_{\varvec{besti}} - \varvec{x}_{\varvec{i}}^{\varvec{k}} ] + c_{2} \,\text{rand}\,[\varvec{g}_{\varvec{best}} - \varvec{x}_{\varvec{i}}^{\varvec{k}} ]. \).

  5. (5)

    \( w^{(k + 1)} = w^{k} + dw \).

  6. (6)

    \( dw = \frac{{(w_{ \hbox{min} } - w_{ \hbox{max} } )}}{T} \).

  7. (7)

    \( x_{\varvec{i}}^{{(\varvec{k} + 1)}} = \varvec{x}_{\varvec{i}}^{\varvec{k}} + \varvec{v}_{\varvec{i}}^{{\varvec{k}+{1}}} \).

  8. (8)

    If \( f\left( {\varvec{x}_{\varvec{i}}^{\varvec{k}} } \right) > f\left( {\varvec{p}_{{\varvec{besti}}} } \right) \) then \( \varvec{p}_{{\varvec{besti}}} = \varvec{x}_{\varvec{i}}^{\varvec{k}} \).

  9. (9)

    If \( f\left( {\varvec{x}_{\varvec{i}}^{\varvec{k}} } \right) > f\left( {\varvec{g}_{{\varvec{best}}} } \right) \) then \( {\varvec{g}}_{\varvec{best}} = {\varvec{x}}_{\varvec{i}}^{\varvec{k}} \).

  10. (10)

    If it converges, then stop.

  11. (11)

    If the maximum number of iterations is not reached, then go to step 2.

The stopping criterion can be maximum number of iterations, and/or a solution with adequate MI value. The parameters w, c 1, and c 2 are chosen by the developer and adjust the performance and effectiveness of the CPSO method.

4 SR Challenges

SR uses subpixel accuracy MC to match areas in adjacent frames to merge them and to combine details wisely, which sometimes cannot be done successfully due to lack of novelty in the LR frames [16,17,18].

Generic image priors may help regularize the solution properly but are not sufficient. Regularization becomes especially crucial when there is an insufficient number of measurements, and/or only one LR frame is observed [5]. Recently, example-based (EB) methods helped to regularize the SR reconstruction problem and to break the SR limit caused by inadequate measurements [19]. EB methods find the prior knowledge by sampling other images locally and are effective when observations are insufficient. Some issues must be addressed: (i) the choice of the optimal patch size given the target image; (ii) different databases with different statistics are necessary; and (iii) how to use the EB prior more efficiently.

Projection onto convex sets (POCSs) express the SR problem as a manifold delimited by multiple constraining convex sets whose intersection contains the sought after image [20]. The concept of a convex set brings in flexibility and adds in different types of constraints or priors, even nonlinear and nonparametric restrictions. It can handle motion blur, but it may require post-processing. The POCS can be further extended for robust, object-based SR reconstruction. The advantage of the POCS technique easily incorporates any kinds of constraints and priors difficult for stochastic approaches. POCS downsides are the substantial computation, slow convergence, and the solution nonuniqueness that depends on the initial guess, the necessity of priors on the motion parameters and system blurs, and inability to estimate parameters at the same time.

The appropriate model formulation is the basis to obtain a suitable solution. Computational intelligence [21] is a substitute to traditional optimization that offers high precision, and lower time cost in demanding applications. Experimental outcomes using the evaluation metrics defined in Sect. 5 [22] confirm improvements in both objective and subjective results. Other ways to resolve the SR problem comprise machine learning (ML) [23] and compressive sensing (CS) [21, 23]. Next, some drawbacks are listed and briefly discussed.

Absence of Motion or Change adds little innovation, and the overall image quality will resemble ordinary spatial upsizing.

Abrupt Motion Detection is problematic to track and results in low-quality frames attributable to: (i) motion blur produced by the camera, and (ii) the use of compression relying on delta-frames. Abrupt motion generates many alterations among adjacent frames, thus increasing the effects of the codec quantization on the required bit rate with a corresponding loss of details.

Heavy Compression (aka low bit-rate compression) in several circumstances can be intolerable for SR. There are two ways of implementing lossy video codecs: (i) via delta-frames, and (ii) with the help of key frames only [24]. If the video has suffered high compression by a key-frame codec, then this means that each frame was compressed independently, and many particulars are missing. This customarily leads to blocking artifacts, which are usual in heavily compressed JPEG images for the reason that the object changes a great deal during motion. Hence, to accurately replicate motion and acquire minutiae becomes unmanageable. Other frames may cause blocking artifacts along with poor picture quality. Similarly to MC, SR uses differences between frames, which results in gross and useless changes that mean no improvement for the HR frame (loss of details).

Image Registration is impacted the accomplishment of high-quality SR reconstruction where the LR frames can be seen as complementary spatial samplings of the HR image to be fused. When LR observations have heavy aliasing artifacts, the resulting HR frame will show low quality. The performance of standard image registration methods decreases proportionally to the decline in the resolution of the observations, resulting in more registration errors. LR picture registration and HR image estimation are fundamentally interdependent processes. Accurate subpixel ME improves HR image estimation. Conversely, high-quality HR image can simplify ME. Therefore, LR image registration affects the HR image reconstruction. Still, with restricted observations, the joint estimation of registration parameters and HR image may introduce overfitting. Stochastic approaches dealing with the HR image estimation and image registration simultaneously are auspicious; still, the motion models can restrict performance. Optical flow ME works when used in more intricate scenarios. Nonetheless, the scarce measurements for local MEs make SR algorithms vulnerable to errors. The 3-D SR problem brings in several extra challenges.

Computational Loads are severe due to the large number of unknowns involved in costly matrix manipulations, which obstruct real-time implementations. The number of calculations goes up dramatically for nontranslation models, which can be bettered by massive parallel computing. The FPGA technology can ease real-time SR systems. For videos with random movements, favorable results stem from parallel computing, for example GPUs and other unusual hardware deployments.

Robustness is essential since the parameters describing the image degradation cannot be found perfectly, and sensitivity to outliers may originate visually disturbing pieces. These imprecisions cannot be regarded as being Gaussian noise, which is the customary supposition when employing the l2-norm.

Performance Limits clarify SR camera development, assisting the investigation of factors such as model errors, the frame quantity, and zooming factors, but an ambitious examination of the performance bounds for all SR techniques can be difficult. Initially, SR reconstruction is considered a hard task demanding many interdependent constituents. Second, the most informative prior for the SR task is unknown. Last, good metrics are still necessary. It is appropriate to extend models with a comprehensive analysis of SR performance by adding factors such as ME, decimation factor, the number of frames, and prior knowledge. Although it is cumbersome to get a consistent performance evaluation for different SR techniques, some benchmarks and realistic datasets can simplify algorithm assessment and understanding.

5 Image Quality Assessment (IQA) in SR

Image Quality Assessment (IQA) procedures can be classified as subjective where human observers perform the image quality assessment and objective, which uses objective metrics. Since ultimately all the images have to be appraised by the human visual system (HVS), subjective evaluation is the only true metric. In reality, however, subjective evaluation is not only problematic and costly, but it cannot be transformed into real-time computer programs with output feedback. Consequently, it is more feasible to use objective IQA metrics to analyze the image quality. According to the original image accessibility (ground truth), traditional objective IQA approaches can be [25, 26, 27]:

  1. (i)

    Full-reference (FR) metrics rely on the original and distorted images;

  2. (ii)

    Reduced-reference (RR) metrics require part of the original image and the distorted image; and

  3. (iii)

    No-reference (NR) metrics need only the distorted image.

The most common IQA metrics are described below.

Mean Square Error ( MSE ): It provides the squared error between the original and the SR image. It will be closer to zero when the ground truth and registered image are equal. It increases when there is a rise in dissimilarity. The MSE can be calculated as follows:

$$ \text{MSE} = \frac{{\sum\nolimits_{i = 1}^{M} {\sum\nolimits_{j = 1}^{N} {[ \varvec{I}(i,j) - \varvec{I}^{\prime} (i,j]^{2} } } }}{MN}, $$
(3)

where I(i, j) and I ′(i, j) are the original and the SR images, M and N are correspondingly the numbers of rows and columns of each image.

Peak Signal ‐ to ‐ Noise Ratio ( PSNR ): The PSNR is expressed in decibels (dBs) as

$$ \text{PSNR} = 20\,{{ \log }_{10}} \frac{{{\text{Max}}_{I} }}{{\sqrt {\text{MSE}} }}, $$
(4)

with \( \text{Max}_{I} \) corresponding to the maximum possible pixel value of the image. Using a byte per pixel leads to \( \text{Max}_{I} = 255 \). PSNR is high when the ground truth and the SR images are similar.

Correlation Coefficient (CC): It represents how correlated (in the least squares sense) the two data sets are. It is defined as:

$$ \text{CC} = \frac{{\sum\nolimits_{i = 1}^{M} {\sum\nolimits_{j = 1}^{N} {[{\varvec{I}}(i,j) - {\mu}_{I} ][{\varvec{I}}^{\prime} (i,j) - {\mu }_{{I^{\prime} }} ]} } }}{{\sqrt {\sum\nolimits_{i = 1}^{M} {\sum\nolimits_{j = 1}^{N} {[{\varvec{I}}(i,j) - {\mu}_{I} ]^{2} } } } \sqrt {\sum\nolimits_{i = 1}^{M} {\sum\nolimits_{j = 1}^{N} {[{\varvec{I}}^{\prime} (i,j) - {\mu}_{{I^{\prime} }} ]^{2} } } } }} $$
(5)

where \( {\mu}_{I} \) and \( {\mu}_{{I^{\prime} }} \) are the mean of images of I(i, j) and I ′(i, j), respectively. The maximum absolute value of CC is 1 and signifies perfect correlation.

Structural Similarity Index (SSIM): It measures the similarities between images, and it emphasizes any image alteration as a mixture of correlation loss, the amount of luminance misrepresentation, and contrast interferences. It is given by

$$ \text{SSIM} = \frac{{\left( {2{\mu}_{{\varvec{I}}} \mu_{{{\varvec{I}}^{\prime} }} + c_{1} } \right)\left( {2\sigma_{{{\varvec{II}}^{\prime} }} + c_{2} } \right)}}{{\left( {{\mu}_{{{\varvec{I}}^{\prime} }}^{2} + {\mu}_{{\varvec{I}}}^{2} + c_{1} } \right)\left( {\sigma_{{{\varvec{I}}^{\prime} }}^{2} + \sigma_{{\varvec{I}}}^{2} + c_{2} } \right)}} , $$
(6)

where \( \sigma_{{\varvec{I}}} \) and \( \sigma_{{{\varvec{I}}^{\prime} }} \) are the variances of I(i, j) and I ′(i, j), and \( \sigma_{{{\varvec{II}}^{\prime} }} \) the covariance between I(i, j) and I ′(i, j). c 1 and c 2 are constants that prevent the denominator to be zero. SSIM ∈ [0, 1], and it approaches 1 as the images become more similar.

The MSE along with the PSNR are widely used FR metrics because they are easy to implement and have clear physical meanings. Still, they fail to be consistent with the perceived visual quality in many cases. The SSIM index takes into account the fact that the human visual system (HVS) perceives the local image structures when computing image quality. Most FR metrics detect and match the features of the original and distorted images to estimate the visual quality of the SR image. These IQA methods pose challenges for SR applications since image sizes change during the SR process.

Moreover, the original HR images are missing whenever the image SR methods are needed in real applications, which may compromise the FR metrics. So, the objective IQA metrics for SR images are desperately necessary.

6 Other PSO Variants

6.1 Some Types of Hybrid PSO (HPSO) Algorithms

PSO is an effective optimization technique that has been applied to an ample range of optimization problems. Nevertheless, its performance can be improved employing certain variants called hybrid PSOs (HPSOs) discussed in this section. The PSO changes can be done in one of the stages below-mentioned, or they can involve a combination of these strategies [28].

  1. (i)

    Initialization: Initial conditions influence the performance of PSO. If the initialization is inadequate, then the algorithm may search an undesirable area, and it will lead to the wrong optimal solution. The performance of CPSO is sensitive to the initialization of the swarms.

  2. (ii)

    Constriction Factor: The random weights controlling the CPSO parameters may cause a kind of explosion as the velocities of the particles, and positional coordinates grow toward infinity. This explosion has been traditionally restricted by an upper bound on the particle velocity, or to the step-size. Even so, the implementation of a good constriction factor can inhibit explosion and improve convergence to local optima.

  3. (iii)

    Inertia Weight ( w ): It balances the exploration–exploitation trade-off. A big value of w increases the exploration, and a small value strengthens the exploitation. Some used a constant inertia weight; others employed a linearly decreasing w and a third group used nonlinearly decreasing inertia weight.

  4. (iv)

    Mutation Operator: It expands the performance of PSO and permits escaping from the local minima. Different variants of PSO using the mutation of the global best particle and the mutation of the local best particle were suggested to prevent the PSO from stagnation in local minima.

Nature-inspired procedures, e.g., genetic algorithms (GAs) [29], simulated annealing (SA) [30], differential evolution (DE) [31], evolutionary programming (EP) [32], artificial immune systems (AISs) [33], ant colony optimization (ACO) [34], can be applied to an extensive variety of global optimization problems. These algorithms are advantageous in the optimization of intricate problems and are typically population-based metaheuristics.

Mingling PSO with GA is quite popular as it is the case with combining PSO and the DE algorithms. The PSO–ACO2 combination is better for solving discrete optimization problems than PSO–ACO [35]. Algorithms such as tabu search (TS), SA, fuzzy logic can also be employed [19, 36, 37], and [38]. As far as local search methods go the Nelder–Mead simplex, Quadratic Programming (QP) and Interior Point (IP) can be combined with PSO.

PSO methods can be associated with other algorithms as follows. (1) One procedure works as a pre-optimizer for the preliminary population of the next algorithm; (2) The entire population is divided into subpopulations, which evolve using PSO and other algorithms; and (3) The unique operators of an algorithm are inserted as local search improvement for the other algorithm.

HPSO methods have been applied to solve an assorted variety of problems encompassing image processing, remote sensing, data clustering, and engineering problems to name a few. Hence, there is still possibility of new SR applications.

GA-PSO, DE-PSO, and PSO with adaptive mutation (AMPSO) are commonly used HPSO strategies to solve unconstrained global optimization problems. GA-PSO combines the PSO and GA algorithms; DE-PSO is a hybrid version of DE with PSO and AMPSO is a mixture of PSO and EP.

HPSO algorithms are still a motivating and promising field that can offer additional comprehensions regarding the behavior and prospective benefits and disadvantages of several metaheuristics. This chapter may motivate and help to develop new hybrid models or to employ the existing models to new applications.

6.2 A Hybrid PSO (HPSO) Algorithm for SR Imaging

PSO has been effectively for SR [38]. When conventional GA and PSO find problems to determine the global optimum, HPSO approaches relying on two GA concepts: subpopulation and crossover. They are incorporated into the CPSO method to improve the accuracy of that conventional GA and CPSO since these traditional procedures cannot find the global optimum when there is a huge number of parameters to be estimated.

The particles are split into m = 1, … , M subpopulations where each one has its personal best optimum g sub-best-m . The PSO process is applied for each subpopulation. If g sub-best-m is better than g best , then g best is replaced by the g sub-best-m where m is the subpopulation number.

The g sub-best-i are organized in ascending order of fitness function values. The top two g sub-best-i are taken as parents (x i , and x j ) for a crossover with i and j as their corresponding subpopulation number. The offspring are produced for each by arithmetic crossover, using the relationships

$$ \varvec{x}_{\varvec{i}}^{\prime } = r\,\varvec{x_i} + (1 - r)\varvec{x_j,}\;{\text{and}} $$
(7)
$$ \varvec{x}_{\varvec{j}}^{\prime } = r\,\varvec{x_j} + (1 - r)\varvec{x_i,} $$
(8)

with velocities

$$ \varvec{v}_{\varvec{i}}^{\prime} =\varvec{v}_{i} V, $$
(9)
$$ \varvec{v}_{\varvec{j}}^{\prime } = \varvec{v}_{\varvec{j}} V,\,\, {\text{and}} $$
(10)
$$ V = \left( {\varvec{v}_{\varvec{i}} + \varvec{v}_{\varvec{j}} } \right)/\left| {\left| {\varvec{v}_{\varvec{i}} + \varvec{v}_{\varvec{j}} } \right|} \right|, $$
(11)

where r is uniformly distributed between 0 and 1. The offspring replaces the worst particle in the same subpopulation.

HPSO‐Based SR Algorithm

  1. (1)

    Initialize the swarm randomly with initial values for variables \( \varvec{x}_{\varvec{i}}^{0} \), \( \varvec{v}_{\varvec{i}}^{0} \), \( \varvec{p}_{{\varvec{besti}}} \), \( \varvec{g}_{{\varvec{best}}} \)

  2. (2)

    For each particle i, repeat until the whole population has been analyzed:

  3. (3)

    Set initial values for variables \( \varvec{x}_{\varvec{i}}^{0} \),\( \varvec{v}_{\varvec{i}}^{0} \),\( \varvec{p}_{{\varvec{besti}}} \),\( \varvec{g}_{{\varvec{best}}} \)

  4. (4)

    Generate M subpopulations

  5. (5)

    Perform crossover

  6. (6)

    If \( f\left( {\varvec{x}_{\varvec{i}}^{\varvec{k}} } \right) > f\left( {\varvec{p}_{{\varvec{besti}}} } \right) \) then \( \varvec{p}_{{\varvec{besti}}} = \varvec{x}_{\varvec{i}}^{\varvec{k}} \)

  7. (7)

    Compute g sub-best-m for each subpopulation m

  8. (8)

    If \( f\left( {\varvec{x}_{\varvec{i}}^{\varvec{k}} } \right) > f\left( {\varvec{g}_{{\varvec{best}}} } \right) \) then \( \varvec{g}_{{\varvec{best}}} = \varvec{x}_{\varvec{i}}^{\varvec{k}} \)

  9. (9)

    If it converges, then stop.

  10. (10)

    If the maximum number of iterations is not reached, then go to step 2.

6.3 PSO Drawbacks

CPSO works fast for optimizing complex multidimensional spaces with a broad range of applications, but it falls easily into a local optimum in high-dimensional spaces, and it converges slowly. This limitation is known as swarm stagnation (SS). There are two puzzling areas for future progress: SS and Dynamic Environments [5].

Prospective users can circumvent SS by certifying that particles continue moving. Although some methods have been suggested for this, the problem is to distinguish an optimal solution. Numerous mechanisms were proposed to restrict particle velocities and to guarantee maximal exploration, such as (i) slower particles that complete more fitness function evaluations while moving to the final solution, and (ii) to introduce mechanisms to increase the likelihood to find a global optimum. Hybrids using other nature-inspired paradigms addressed PSO’s inherent challenges to reposition stagnant particles or slowing down those using subpopulations.

Dynamic Environments imply the existence of objective function values that diverge over time and are difficult to handle when the optimum position travels thru the problem space in steps larger than the best candidate solution of the group can follow naturally. CPSO cannot track the optimum because unless the goal moves to the space of another swarm member, the swarm will continue to move to its previous best, set at the previous goal position. Re-resetting the swarm or using the old swarm with new best values helps mitigate complications. The major limitation to effective optimization when using PSO in Dynamic Environments is when the optimum position has velocity superior to the particle velocity and therefore escapes the swarm. Dynamic problems require a balance between velocity constraint and the need to track optima. Slow-moving optima can be pursued naturally since the particles keep on moving about in an area related to the velocity of the particles. Complications happen when the optimal displacements are larger than the particle dislocations. The fundamental norm is to keep swarm diversity, which increases the likelihood that the optimum will shift to an adequate area.

7 Case Studies of the Use of PSO in SR

To some extent, the determination of the appropriate model is the basis for finding the solution. Bearing in mind computational intelligence, the image or the video sequence with the highest resolution corresponds to the optimal solution obtained by PSO. This established optimization method is on the forefront for its implementation simplicity, higher computational accuracy, and lower time cost in complex applications when compared to other heuristic methods. In PSO, a particle is a potential solution vector to the problem. Nevertheless, the parameter selection affects the performance intensely, and schemes that work well for all problems are absent. In many cases, the PSO parameters have to be adjusted several times to obtain adequate precision and escape the fixed parameters limitation during the whole optimization process.

This section illustrates the use of SR in still images and video sequences using the PSO and HPSO algorithms stated beforehand [39].

7.1 SR in Still Images

Geometrical SR restoration is possible if and only if the input LR images have been undersampled, and hence, they contain aliasing. Since there is aliasing, the high-frequency content of the sought reconstructed image is embedded in the LR content of each of the observed LR images. Given an appropriate number of LR observations, and if the set of observations differ in phase (i.e., the scene images are shifted by subpixel quantities [1, 18, 40,41,42]), then the phase evidence can help to isolate the aliased high-frequency part from the true low-frequency portion, and the full-resolution image can be precisely recreated.

There exist both single-frame and multiframe SR alternatives. Multiframe SR explores the subpixel shifts between multiple LR images of the target scene. It produces a better resolution image resulting from combining data from all the LR pictures, and the created HR images are superior scene descriptions. Single-frame SR approaches attempt to enlarge the image without blurring effects. These procedures employ other parts of the LR images, or other distinct images, to predict what the HR image should appear. Nowadays, SR methods work well with both grayscale and color images.

Figures 4, 5, and 6 show results of applying PSO and HPSO to the image Lena.

Fig. 4
figure 4

a LR image; b PSO result; and c HPSO result

Full size image
Fig. 5
figure 5

PSNR curves for Lena as the number of LR images increase

Full size image
Fig. 6
figure 6

SSIM evolution for Lena as the number of LR images increase

Full size image

7.2 SR in Video

The SR main conception is to add new information to a target frame by detecting the subpixel displacements between this frame and its nearby ones. These subpixel shifts are created by the camera and/or by the objects moving in the scene. Figure 7 shows the motivation behind VSR; that is, an SR frame is generated by combining four consecutive LR frames.

Fig. 7
figure 7

In VSR processing, each HR frame is a combination of N successive LR images

Full size image

The importance of a good ME in SR naturally arises. Independently of the SR method used, the underlying ME technique must be as accurate as possible [17, 24, 40, 41, 43, 44]. In literature, little attention has been paid to the performance of ME techniques when used in SR problems. The notation for an entire image is g l , representing the lth image in the LR sequence. g k denotes the image whose resolution must be improved.

Let g l (x) be a pixel located at x = (x, y)T in the lth LR image, which is M × N. g l is divided into blocks bx of size m × n such that g l (bx p ) with p = 1, … , P and P is the total number of blocks.

Accurate ME comprises a very important part in SR problems. Pixels with bad and/or contradictory MEs due to occlusions and poor ME results degrade the SR reconstruction and consequently should not be considered.

Quantitative assessments between the enhanced and the original images can be done using the PSNR. Inaccurate MVs can be detected by applying the displaced frame difference (DFD) between the upsampled and the compensated frames given by

$$ \text{DFD}({\varvec{g}}_{{\varvec{l}}} ,{\varvec{d}}_{{{\varvec{l,k}}}}^{{{\varvec{lr}}}} ,{\varvec{g}}_{{\varvec{k}}} )({\varvec{x}}) = \left| {{\varvec{g}}_{{\varvec{k}}} ({\varvec{x}}) - {\varvec{g}}_{{\varvec{l}}} ({\varvec{x}} - {\varvec{d}}_{{{\varvec{l,k}}}}^{{{\varvec{lr}}}} )} \right| $$
(12)

The DFD allows identifying pixels in g k that are not anticipated by the MV estimates \( {\varvec{d}}_{{{\varvec{l,k}}}}^{{{\varvec{lr}}}} \) (between the LR images lth and kth) and g l . Large DFD values indicate nonpredictable/observable pixels in g k .

After estimating the motion between frames, it is possible to project the LR image on the sought after HR grid with the help of the MVs. An HR grid can be defined by locating the pixel values of the present image, while the remaining HR positions stay empty (zero MVs). A different frame having subpixel shifts with respect to the current image has new data placed onto the HR grid. An explanation of the SR algorithm steps follows:

  1. (1)

    Align the LR image with the matching positions of the HR image.

  2. (2)

    If there is subpixel motion, then new pixels from frames \( l = k \) will be placed in empty locations of the HR grid.

  3. (3)

    After filling all possible places, empty HR pixel values will be interpolated to obtain the final SR image.

  4. (4)

    Use the estimated motion fields to render the HR reconstruction.

  5. (5)

    Calculate the PSNR values between the original HR image I and the resulting SR image.

Figure 8 shows three regions R1, R2, and R3 belonging to a frame of the mobile sequence. This subsection explores the process of obtaining SR versions of these patches. Figure 9 presents 5 LR frames of the mobile sequence. Figure 10a shows the PSNR curves for the reconstructions done for R1, which is an area with texture but also some extremely flat subregions. In this case, the PSNR values improve for the PSO and HPSO algorithms. Regions R2 and R3 contain objects with motions. The most remarkable conclusion on the R2 is that when the motion does not depend from the camera motion, increasing the accuracy of the ME can introduce errors to the global SR process. The advance achieved by the use of the DFD is perceptible in Fig. 10b where the motion complexity is high (rotation and translation). Lastly, R3 has a flat region with translational motion and the PSNR results are depicted in Fig. 10c.

Fig. 8
figure 8

Region 1 (R1) has a textured area with translational motion, Region 2 (R2) shows a textured area with independent object motion, and Region 3 (R3) contains a flat area with translational motion

Full size image
Fig. 9
figure 9

Mobile LR sequence obtained using from frame 7 to frame 11

Full size image
Fig. 10
figure 10

PSNR values versus number of LR frames for the mobile sequence: a R1, b R2, and c R3

Full size image

8 Conclusions

Metaheuristic optimization algorithms like PSO have become a widespread choice for solving complex and intricate problems such as SR reconstruction, which are otherwise demanding to solve by traditional algorithms. This chapter attempts to examine the PSO algorithm and some of its hybrid versions. Hybridization means combining two (or more) methods in a sensible way such that the resultant algorithm contains the positive characteristics of both (or all) the procedures. Interesting techniques to use in hybridization include many local and global search approaches. Results from the use of SR reconstruction for still and video images are shown for the PSO and the HPSO algorithms.