Introduction

The status of insects such as butterflies provides a valuable indicator for changes in biodiversity. Butterfly populations are well-monitored both in the UK and beyond (Van Swaay et al. 2019), and abundance trends are typically produced at national level. The abundance status of butterflies can be used to assess progress towards biodiversity targets and agreements, as well as inform research and conservation activity. For example, UK butterfly abundance indices are used to form one of the biodiversity indicators employed by the UK government to assess trends in biodiversity (Department for Environment, Food and Rural Affairs 2019). Abundance trends also inform species’ Red List status (Fox et al. 2010) and UK Butterfly records were used in the first global assessment of biodiversity change (Secretariat of the Convention on Biological Diversity 2007).

Count data for butterflies in the UK are mostly gathered through the UK Butterfly Monitoring Scheme (UKBMS), which consists of a national network of transects where counts are undertaken on a weekly basis during the main period for butterfly activity (Pollard and Yates 1993). Since 1976 over 5,000 transects have contributed to the scheme, with 2,868 in 2018, from which long-term and ten year trends for 56 of the 59 species occurring regularly in the UK are reported annually (Brereton et al. 2019).

There have been several developments in the methods used for producing national abundance indices and trends from UKBMS data (Rothery and Roy 2001; Dennis et al. 2013, 2016). Fundamentally, a curve is fitted to the counts to estimate each species’ flight period in a given year. This is necessary to enable counts to be estimated in missed weeks, since approximately 30% of potential counts within the sampling period, which runs from April to September, are not walked (Dennis et al. 2016). Following the estimation of missing counts, relative abundance indices for each transect are produced and subsequently combined into an overall index of relative abundance for each species, from which trends can be deduced to assess population changes over time.

Estimating abundance trends at regional rather than national level could be beneficial for assessing and understanding changes in population at finer scale levels, for example to better inform and guide conservation effort. In this paper we consider how to produce regional trends from transect data, with a focus on butterflies in Surrey (UK) as an example. Here we broadly consider a region to be subset of a sites covering a small geographic area, where all transects are assumed to have the same flight period. In the UK for example this could be a larger region, such as South-eastern England, or a smaller Vice county, as considered in this paper. However, as discussed later, the method proposed in this paper could also be applicable to national monitoring schemes with fewer transects than the well-established UKBMS, provided the general assumption of static flight periods can be made.

Whereas at the national (UK) level data are typically available from many transects, at finer-scale regional levels there may only be data available from a few transects, in particular due to limited transects being established in the early years of a monitoring scheme, and rare or locally scarce species being only present on a few transects within the region. Methods developed for estimating flight periods and producing abundance indices for more numerous national-scale data may therefore not be directly suitable for producing finer-scale estimates of indices and trends.

Recent approaches for producing national abundance indices typically rely on the assumption that species’ flight periods are static across transects within each year, although Schmucki et al. (2015) developed an approach that accounts for variability in flight periods across climatic regions, which has been applied to an indicator for EU butterfly populations (Van Swaay et al. 2019). Furthermore, Dennis et al. (2016) developed parametric descriptions of species’ flight periods, where spatial variation in flight periods can be described by suitable covariates. In most cases variation in flight period is expected to be greater year to year than within years, but for some species flight periods vary substantially across space, for example Common Blue typically exhibits a change in voltinism across the UK (Matechou et al. 2014), and some species also exhibit spatial variation in their date of emergence, particularly those with a large range (Roy and Asher 2003). At the regional scale, flight periods may therefore not be equivalent to national flight periods, hence the relatively straightforward approach of using site indices produced from recent national analyses to estimate regional trends may not be appropriate. At finer scales biases in flight period estimates, and therefore on estimates of counts in missed weeks, may be expected to have a greater influence on the estimated trends due to the smaller number of transects contributing to the index than typically at national level.

Earlier approaches estimate species’ annual flight periods for each site (Rothery and Roy 2001), and are typically still in use for local or site-based studies. However this approach results in the exclusion of data from transects with insufficient monitoring, for example Dennis et al. (2013) estimated that on average only 60% of 10 km UK grid squares with monitored sites were used based on the site-based approach (Rothery and Roy 2001). At the regional scale the number of transects may already limited, hence the need for an approach that maximises the use of the available data.

Whereas at the national level one might expect there to be sufficient data to estimate a species’ flight period for each year, at smaller spatial scales there will be fewer counts which may be inadequate to accurately estimate species’ flight period. To overcome this problem, in this paper we present a method that incorporates a so-called “Dummy” site to support the estimation of species’ flight period. The method draws upon species’ records from other available data sources to provide an informed estimate of each species’ flight period for the region of interest. Comparisons will be made with regional indices produced without the Dummy site, as well as with indices produced using data from national analyses.

Methods

Data sources

Abundance records for butterflies in the UK are primarily collected through the UK Butterfly Monitoring Scheme (UKBMS), using a standard methodology introduced by Pollard and Yates (1993), where an in depth description of the transect method is described, as well as in brief here. The method involves walking a fixed route (transect) once a week from 1st April to 29th September, counting all butterflies that can be identified within a 5 m box from the transect walker. The date of the week that a transect is walked can vary from week to week. Transects are only walked during specified periods of the day and under suitable weather conditions, so that the detectability of the butterflies is at its most favourable. Weather conditions and transect walker availability typically mean that some weeks during the recording season are not walked. Here we use transect and site interchangeably, such that a transect represents a single site in the methods to be described.

There are 112 Vice counties in Great Britain (England, Scotland and Wales) (Dandy 1969; Copp 2003). In this paper we focus upon Vice county 17 (VC17) for Surrey (hereafter, Surrey). Although the UKBMS began in 1976, for Surrey counts are only available from 1986 onwards, where some earlier paper records have either been lost or not digitised. In 1986 Surrey had just one transect (Banstead Downs), but the number of transects has since increased to 120 which were walked in 2018.

The butterfly fauna for Surrey has changed over the years. In 1986 there were 43 resident species, excluding unauthorised introductions. Since then four species have become extinct due to habitat loss, leaving the current fauna of 39 (Collins 1995; Willmott et al. 2013). A checklist of the butterfly fauna for Surrey, including scientific names, is provided in Supplementary material 4, based on Agassiz et al. (2013). The number of UK transects with a site index used in the regional analysis, where each species was counted in Surrey in each year is presented in Supplementary material 6.

As described in the following subsection, we aim to estimate the flight period of each butterfly using all available butterfly records for the region (Surrey). The Butterflies for the New Millennium (BNM) (Asher et al. 2001) was formally launched in 1995 in the United Kingdom and comprises over 12.7 million species occurrence records covering the period 1690 to date. BNM records include date, location, species and count information. Whereas in the UKBMS records the count information has been collected under standardised conditions, this is not true for all BNM records. However, there are records for Surrey from 1986 onwards, including some historic records. The Surrey BNM dataset comprises butterfly records from all sources, including UKBMS counts, the Wider Countryside Butterfly Survey (WCBS, Roy et al. 2007; Brereton et al. 2010), records from 2007; Big Butterfly Count, records from 2011; iRecord, records from 2012 with some older records; Migrant Watch, records from 2013; and Garden Butterfly Survey, records from 2016. The number of records used in calculations is shown in Fig. 1.

Fig. 1
figure 1

Number of butterfly records for Surrey from the BNM and UKBMS, used in the flight period estimation and abundance trend calculations

Full size image

Estimating flight periods from all data sources

When monitoring data might be limited in quantity, for example when considering data for small regions, it may be difficult to accurately estimated flight period for each year from count data alone. Hence we utilise butterfly records from all available sources for each year, for the given region, to estimate the flight period of each species. This is a similar approach as described in Bishop et al. (2013). Here we describe the approach taken for estimating the flight period from all records.

Firstly, each species is classified based on their maximum voltinism. For example, the Orange-tip is considered univoltine, but very rarely individuals have been reliably recorded which could only have come from a second brood. Hence Orange-tip is considered to have a maximum voltinism of two. However, we only calculate the flight period parameters if there are at least ten records for a given brood, to ensure sufficient records to calculate a reasonable curve representing the flight period. Hence, the partial second brood of the Orange-tip is ignored because of lack of records, but this ensures the end date of the first brood is calculated more accurately. Species which hibernate in the imago stage, such as Brimstone, are treated as though their Spring emergence from hibernation is a brood, defined here as brood 0.

We identify the start of the flight period for a given brood by the first record from the start of the year, or the day after the end of the previous brood. The end of the flight period is then defined by searching for the biggest gap in the records within a window. Supplementary material 8 defines the parameters used in finding the start and end of the flight period for each brood, where we define the limits of the window based upon voltinism and overwintering life stage. Species which hibernate in the imago stage emerge in early spring, with one or more broods later in the year. Further work is required to generalise the method and to refine it for overlapping broods.

A key feature of the approach presented here is the use of a so-called Dummy site to enable better estimation of regional flight periods than from transect counts alone. Other than aiding in the flight period estimation for the UKBMS data, the Dummy site is not used in the calculation of the regional index itself, or the associated trend estimation.

Firstly a parameterised curve is calculated from all records, which in effect is a theoretical flight period for a given brood. The curve is then used in a second step to help produce a better fit to the data, in the form of a Dummy site.

To produce the Dummy site for each butterfly species, the flight period for each butterfly in each year is estimated. Figure 2 shows estimated flight curves for 2018 for four butterfly species, which are estimated as follows. Records for a given brood, b, are assumed to display the shape of a Normal distribution, \(N(\mu _b,\sigma _b^2)\), where \(\mu _b\) is based on the weighted mean of all records across all N possible days, \(d_{j,b}\), for brood b,

$$\begin{aligned} \mu _b = \frac{\sum _{j=1}^N w_{j,b} d_{j,b}}{\sum _{j=1}^N w_{j,b}}, \end{aligned}$$
(1)

where \(w_{j,b}\) is the total abundance count from all data sources for day j and brood b, and the weighted variance is

$$\begin{aligned} \sigma _b^2 = \frac{\sum _{j=1}^N w_{j,b}}{\left( \sum _{j=1}^N w_{j,b}\right) ^2-\sum _{j=1}^N w_{j,b}^2}\sum _{j=1}^Nw_{j,b}\left( d_{j,b}-\mu _b\right) ^2. \end{aligned}$$
(2)

A count value y, for Dummy site z, is calculated for each day, j, from \(j = d_{1,b}, \dots , d_{N,b}\)

$$\begin{aligned} y_{z,j,b} = c \times m \times \frac{1}{\sigma _b\sqrt{2\pi }}\text {exp}\left[ -\frac{(j-\mu _b)^2}{2\sigma _b^2}\right] , \end{aligned}$$
(3)

where m is the maximum abundance count from the UKBMS data in that year, and c is a constant, set at 100, to produce a smooth curve of integer values of y.

Using a Dummy site in the calculations of the site abundance indices has the weight of adding seven sites to the calculations, since the Dummy site has a count for each day of the week, whereas sites (transects) that are actually walked only have one count during the week. Thus the Dummy site will have a larger influence on flight period estimation when a species has only been counted on a few transects, and conversely the Dummy site will have a reduced influence where there are a large number of transects. The contribution that the Dummy site has on the calculations will also vary over time, as there were fewer transects in the early years.

Fig. 2
figure 2

Flight curves (green lines) estimated from Surrey 2018 BNM records for four species with varying overwintering strategies and voltinism. Crosses identify counts. Overwintering stages are imago for Brimstone, larvae for Meadow Brown and pupae for Green-veined White and Wood White

Full size image

Producing regional abundance indices

For a given species and year, a generalised additive model (GAM, Wood 2017) with Poisson distribution and log link function is used to estimate the annual flight curve, using counts from all available transects, as well as from the Dummy site (Eq. 3). To prevent divergence in the tails, additional count values of zero are added for 1 April and 30 September for all data, including the Dummy site, except where the flight period extends beyond those dates. If \(y_{i,t}\) represents a count at site i on day t, then

$$\begin{aligned} \mathrm{E}[y_{i,t}] = \text {exp}[\eta _i + s(t;f)], \end{aligned}$$
(4)

where \(\eta _i\) represents a site effect and s(t; f) denotes a smoothing function with f degrees of freedom, as in stage one of Dennis et al. (2013).

Annual site indices of abundance are calculated by an estimate of the area under the curve of the abundance counts. Here we calculated site indices using Simpson’s Rule applied to the predicted counts from the GAM, taking 26 sub-intervals of 7 days, starting from 1st April. Simpson’s Rule is used as it produces a smaller error when using equal intervals than other methods in calculating the area under the curve. We use estimated rather than raw counts to reduce potential bias resulting from variation in the day of the week in which counts are made.

For a given species, a regional index is then derived by fitting a Poisson generalised linear model (GLM, McCullagh and Nelder 1989) to the site indices (now excluding the Dummy site), with site and year as multi-level factors, an approach which is common practice (Pannekoek and van Strien 2005; Dennis et al. 2013). Scaled predicted year effects are then used to form an index of abundance. Indices are presented on the log10 scale, baselined such that the first year is 2, where the base year for each species was chosen as the earliest date for which there was sufficient data to calculate an index. Trends can be calculated by applying linear regression to the regional indices.

Non-parametric bootstrapping is typically used to derive confidence intervals for national indices of abundance for butterflies (Dennis et al. 2013, 2016), by resampling the raw count data, but this approach is unlikely to be appropriate for small sample sizes. Hence we use Monte Carlo simulation (Robert and Casella 2004) to estimate confidence intervals for the site and regional indices.

For each week a random count was generated 1000 times from a Normal distribution with estimated mean and standard deviation of the flight period curve estimated from the GAM. Count values were limited to the range 0 to 99,999, as counts cannot be negative, and to cap extreme values. For each iteration a site index was then calculated as described (using Simpson’s Rule). Similarly a regional abundance index was then estimated for each iteration by fitting a Poisson GLM. Regional indices and trends are presented as the mean estimate from the Monte Carlo simulations. Confidence intervals for the regional indices (as well trends) were formed by taking 95% quantiles. Trends were deemed as significant where the range of the confidence interval did not include zero.

Application

We demonstrate the above approach, which we refer to at the regional method, by application to the 43 butterfly species resident in Surrey in 1986, using data for 1986–2018. The number of transects per year which contributed to the Surrey regional indices produced from regional data are given in Supplementary material 6. Figure 2 shows the calculated flight period curves for four butterfly species with different voltinism and overwintering stage from records. The curve represents the counts for the Dummy site produced from Eq. 3, without the scaling factor c.

To eliminate cases with insufficient data, during a given year, only transects that were walked (visited) at least four times were included. Also models were only fitted where a species had at least six positive counts from all transects during a given year. Transects were also excluded if the total count was less than 6 individuals for all years.

We draw comparisons with regional indices produced using the same approach, but excluding the Dummy site from the GAM (“Effect of the Dummy site” section). Secondly we compare with regional indices and trends produced from an analysis of national data (“Comparison with regional indices produced from national analyses” and “Comparison of trends from the regional and national methods” sections), which we refer to as the national method. In the latter case regional indices were estimated from site indices produced from the national UK dataset. The national transect indices were produced following the approach of Rothery and Roy (2001) which applies a GAM to estimate flight period at the site level, i.e. to counts for each transect and year separately. Using this approach, in a given year transects with a high proportion of weeks or the peak of the flight period missed are excluded from analysis (Dennis et al. 2013), and hence fewer transects are likely to contribute to trend estimation. The number of transects per year which contributed to the Surrey regional indices produced from the national analysis are given in Supplementary material 7.

For the regional method, confidence intervals were estimated using Monte Carlo simulation for 1000 iterations (described in the previous subsection). Increasing the number of iterations to 5000 for four species showed no perceptible difference in the results obtained (Supplementary material 5). Regional abundance trends using flight periods informed by the Dummy site are presented for 10, 20 and 30 years up to 2018 (“Summary of the regional trends for Surrey” section). Regional trends were excluded from the results where they lay outside the 99.9% quantile range. The regional indices and trends produced from the national method are presented without confidence intervals, as only the calculated site indices for Surrey were available.

All analyses were undertaken using the R statistical language version 3.5.3 (The R Core Team 2019), and using the packages mgcv version 1.8–28 (Wood 2017) and snowfall version 1.84–6.1 (Knaus et al. 2009).

Results

Of the 43 butterfly species resident in 1986, it was not possible to calculate a regional index for five species (Purple Emperor, Small Pearl-bordered Fritillary, Small Skipper, Wall and White Admiral), and for a further two species a trend could not be calculated (Duke of Burgundy and Pearl-bordered Fritillary). Of these seven species, four have been lost from the county (Duke of Burgundy, Pearl-bordered Fritillary, Small Pearl-bordered Fritillary and Wall). Most transect walkers do not distinguish between Small and Essex Skipper, so there is a lack of records for these two species, although there were sufficient records to calculate a 10 year trend for Essex Skipper. Whilst there were sufficient data to calculate a regional index for Small Skipper and White Admiral, the 95% confidence limits where extremely large, hence they were excluded.

Effect of the Dummy site

The regional indices for all butterflies for Surrey, with and without the inclusion of the Dummy site in the calculations, are shown in Supplementary material 1. Of the 38 butterfly species where it was possible to calculate a regional index, the regional index calculations were visibly similar for 31 species when comparing with and without the inclusion of the Dummy site in the calculations, although with some minor differences, particularly in early years, for example for Orange-tip and Silver-washed Fritillary. More apparent differences can be seen for seven species (Adonis Blue, Brown Argus, Chalk Hill Blue, Large White, Marbled White, Peacock and Silver-studded Blue), where excluding the Dummy site from the calculations can result in large spikes in the regional index for some years, which are not supported by the raw data. For example, an unrealistic spike in the regional index for Marbled White can be seen in 2015 when the Dummy site is not included (Fig. 3). Further inspection suggested convergence issues for the regional method excluding the Dummy site in these instances.

Fig. 3
figure 3

Estimated regional indices for Marbled White in Surrey with 95% confidence intervals, estimated using the regional method with (a) and without (b) the Dummy site. 10 (red), 20 (green) and 30 (mauve) year trend lines are shown

Full size image

Neither approach offers a completely robust set of regional indices, for example the indices for Dingy Skipper exhibit a peak at the start in both cases, due to a large count in the first year at one transect, followed by low counts in subsequent years. Also some species show wide estimated confidence intervals, particularly in the early years when there were minimal transects with counts, such as for Green Hairstreak.

As shown in Fig. 4, the inclusion of the Dummy site reduces the error in differences between the raw counts and the calculated counts. The mean of the Pearson residual is an order of magnitude larger when excluding the Dummy site. With the exclusion of the Dummy site, there are more extreme estimated counts, than when the Dummy site is included in the calculations, especially for small raw counts. Excluding the Dummy site also produces some low estimated counts in cases with relatively high raw counts.

Fig. 4
figure 4

Pearson residuals of raw and estimated counts based on the regional method with and without the Dummy site. Extreme residuals excluded

Full size image

Comparison with regional indices produced from national analyses

A comparison of regional indices for Surrey produced using the regional method (with Dummy site) and national method is shown in Supplementary material 2. The approach used for the national method is described in “Application” section. The regional indices produced from the regional and national methods are baselined to the same year.

Using the national method, regional indices were produced for all 43 resident butterfly species, whereas using the regional method described in this paper, regional indices could only be calculated for 38 butterfly species. However, results from the Purple Emperor, Small Pearl-bordered Fritillary and Wall Brown are of limited value due to data deficiencies. Indices were produced for Small Skipper and White Admiral, despite their exclusion using the regional method due to wide confidence intervals, although confidence intervals could not be calculated with the available national data for comparison.

The new regional method (using the Dummy site) and the national method (using site indices from national data) produce generally similar regional indices for the 38 butterfly species. In some cases the regional method seems to produce more realistic results, for example, for Brimstone (Fig. 5), Green Hairstreak and Large White, where the regional indices from the national method show spikes in some years. Further investigation showed that such spikes are due to a very large site index in one year, for one transect in each case, which distorted the results.

Fig. 5
figure 5

Estimated regional indices for Brimstone in Surrey based on estimates from regional method with the Dummy site (a), with 95% confidence interval, and the national method (b). 10 (red), 20 (green) and 30 (mauve) year trend lines are shown

Full size image

Comparison of trends from the regional and national methods

Fig. 6
figure 6

Comparison of 10, 20 and 30 year species abundance trends for Surrey, calculated by the regional and national methods for 35 species

Full size image

10, 20 and 30 year regional trends for Surrey calculated from the regional and national methods are tabulated in  Supplementary material 3. As discussed in “ Application” section, confidence intervals could not be calculated for the regional trends produced by the national method, hence significant differences in the regional trends from the two methods could not be tested directly. However in 85% of cases the trend estimates from the national method did not fall within the 99.9% confidence intervals of the trend estimates from the regional method, suggesting considerable differences for some species and trend periods.

Overall trends from the two methods were mostly similar, at least in sign and magnitude (Fig. 6). However for some species and trend periods there were large differences, which are typically explained by visible differences in the regional indices (Supplementary material 2). Differences tend to be greater in shorter-term rather than long-term trends.

Summary of the regional trends for Surrey

Table 1 Percentage change in the species regional abundance indices for Surrey over 10 (2009–2018), 20 (1999–2018) and 30 (1989–2018) years, where * indicates 95% of simulated trends were of the same sign as the calculated trend, indicating significance, ** indicates 99% and *** indicates 99.9%
Full size table

The estimated abundance trends for Surrey, calculated using the regional method, are shown in Table 1. Thirty year trends for Surrey were calculable for 23 resident butterfly species, whereas shorter twenty and ten year trends could be estimated for 33 and 35 species, respectively. Trends could not be calculated for five butterflies. Duke of Burgundy and Pearl-bordered Fritillary were both lost during the period considered. There was insufficient data to calculate a trend for Brown Hairstreak, Small Skipper and White-letter Hairstreak.

One butterfly species has a consistent negative trend over all three time periods (Peacock), and a further two butterfly species have a negative trend over 10 and 20 years (Grayling and Silver-studded Blue), with insufficient data to calculate a 30 year trend.

Eleven butterfly species have a consistent positive trend over all three time periods (Brimstone, Chalk Hill Blue, Comma, Common Blue, Dingy Skipper, Holly Blue, Large White, Marbled White, Meadow Brown, Ringlet and Small White). A further four species have a positive trend over 10 and 20 years (Dark Green Fritillary, Purple Hairstreak, Silver-washed Fritillary and Small Copper), with insufficient data to calculate a 30 year trend.

Discussion

We have presented a new approach for producing abundance indices from transect counts, which draws upon information from other data sources to evaluate species’ annual flight periods, required to estimate missing counts in the sampling period. This has particular relevance for situations where data may be limited, for example when considering small spatial regions, as demonstrated for Surrey in the UK in this paper.

It is typical to estimate species’ flight periods from transect data, and use these to estimate missing counts, however doing so for noisy data from potentially a few transects may not reliably estimate the flight period. UKBMS data is fundamentally noisy due to various factors such as weather, time-of-day, different recorders, etc. (Pollard and Yates 1993). With the introduction of a Dummy site, which draws upon data from all available sources, a better fit to the data can be made, as shown in “Effect of the Dummy site” section. We have also shown that constraining the flight period to be the same across all sites can produce more robust and realistic species indices and trends (“Comparison with regional indices produced from national analyses” and “Comparison of trends from the regional and national methods” section).

Using the Dummy site to inform flight period estimation results in better estimates of species counts from the GAMs, therefore suggesting better estimation of the missing counts requiring imputation, and thus leading to improved regional index estimation, particularly for certain species. Estimating flight periods from a weighted abundance measure provides suitable information to form the Dummy site, albeit ignoring the lack of standardisation and differences in sampling effort resulting from using multiple data sources. A drawback of the approach presented may be the need for additional data, although data from more casual citizen-science sources and opportunistic recording are typically available in greater quantity than transect data (Fig. 1 for Surrey records).

Regional indices and trends for Surrey obtained using the national method and the new regional method were often similar, suggesting a general robustness to the indices obtained regardless of the method used, despite using different approaches to flight period estimation, as well as other minor differences, for example in the precise number of transects per year. The site indices produced by the national method were estimated using the approach of (Rothery and Roy 2001), where annual flight periods are estimated separately for each transect. The general similarity between the regional indices produced from these site indices and those produced from the regional method, where flight period is assumed to be fixed across sites, suggests that differences in flight period may not have a large influence at this regional scale.

The regional method has the potential to allow for data from more transects to contribute to regional indices given that the national method based on Rothery and Roy (2001) excludes data from transects with insufficient monitoring. Dennis et al. (2013) showed that at national scale, estimating a common flight period to maximise the use of data showed improved power for detecting trends, compared with the approach of Rothery and Roy (2001). By example, the regional indices for Silver-studded Blue produced by the regional and national methods are broadly similar, but differ for 2012 and 2013 (Supplementary material 2), resulting in a large difference between the 10 year trends (Supplementary material 3). The index from the regional method is in fact supported by data from more sites than the national method (Supplementary material 6 and 7), suggesting the trends from the regional method are likely to be more robust and realistic. Maximising the data used has the benefit of optimising spatial coverage as well as allowing more valuable data collected by volunteers to contribute to abundance indices and trend estimation, which is likely to be more important when data may be limited, such as when working at regional level. More recent approaches for national-scale modelling (Dennis et al. 2013, 2016) also make greater use of available data, but at finer-scales we have shown the benefit of using additional information to estimate species’ flight periods (for example see "Effect of the Dummy Site" section). Future work could also seek to develop recent methods applied to national-scale data but accounting for spatial variation in flight period, for example by drawing upon information from climatically similar regions (Schmucki et al. 2015; Van Swaay et al. 2019) or using suitable covariates such as growing degree days (Cayton et al. 2015). However in practice sufficient data may not be available, for example for rare species or small-scale monitoring schemes.

A regional index could be calculated for two extra butterfly species (Small Skipper and White Admiral) using the national method, but with unknown confidence limits, where they were rejected by the regional method due to the wide confidence limits. Using the national method, unusually high index values featured in early years of the regional index for three butterfly species (Brimstone, Green Hairstreak and Large White), which the index produced using the regional method did not exhibit, however these issues could be resolved by taking a later start year. In contrast, where extreme estimates were sometimes estimated in the indices in a particular year produced using the regional method without the Dummy site, these were often not limited to early years, demonstrating the advantage of including the Dummy site when using the regional method.

The regional method assumes that species’ flight periods are static across sites within a given year. This is likely to be a reasonable assumption for a small geographic region such as Surrey, where altitude varies from just above sea level to 295 metres, latitude varies between \(51.072{^{\circ }}\) N to \(51.509{^{\circ }}\) N, and longitude varies between \(0.847{^{\circ }}\) W to \(0.058{^{\circ }}\) E. Greater differences between the two main modelling approaches considered here (the regional method with the dummy site, and the national method) may be expected for a region where flight periods differ more among sites within the region, for example in a more mountainous region with greater variation in elevation, latitude and environmental conditions. In wider applications, the validity of the assumption of static flight periods within a region of interest will always require consideration. Flight periods may even vary between sites within close proximity for example due to differences in aspect and habitat condition.

The production of the Dummy site relied upon suitably identifying the start and end of each species’ brood(s). Further work is needed to refine the approach taken, which works best when there is a clear gap between species’ broods and has limitations for species with overlapping broods. This aspect of the approach taken is also not currently generalised, but specific to data for Surrey. An alternative method could be to fit a GAM to the data, and find the minima within the species flight period (Macgregor et al. 2019), or use the prediction from such a GAM to describe the shape of the flight period itself. The new method involves a two-step approach, whereas future developments could explore an integrated modelling approach to benefit from multiple data sources (Isaac et al. 2020).

The new regional method presented uses Simpson’s Rule with equal intervals between data points, whereas the site indices calculated from the national method were calculated using the trapezoidal rule, with unequal intervals. In national analyses, site indices are calculated from raw counts where available and estimated counts for missing weeks. In the new regional method, site indices have been normalized by using equal intervals between each week to reduce the variability in counts resulting from differences in the day sampled, hence estimated counts are used for all weeks rather than raw counts. With equal intervals, Simpson’s Rule generally produces the smallest error.

The general assumption in calculating a regional index (using a GLM) is that all sites behave the same. With a large number of transects this is a reasonable assumption. However, with a small number of transects, various influences could become significant, for example differences in habitat management. Species, such as the Green Hairstreak, which are typically recorded in small numbers on transects, can result in large confidence intervals, where a difference of \(\pm 1\) can have a large impact on the results. Canopy species such as Brown Hairstreak and White-letter Hairstreak cannot be reliably monitored using the standard UKBMS methodology, hence regional indices for these species are unlikely to produce meaningful results. For example, egg counts for Brown Hairstreak is a possible alternative sampling method. However, for some rare species it may not be possible to reliably monitor them.

Using the IUCN Red List Criteria A2b as used in the Red List for Great Britain (Fox et al. 2010), butterfly species declining at 30% or more over a ten year period are of conservation concern. In some cases the national and regional methods can produced considerably different trend estimates for Surrey, which could have impacts for conservation actions. For example, the 10 year abundance trends for Surrey for Silver-studded Blue were − 42.2% using the regional method and 29.6% using the national method, where the first would indicate the species is vulnerable to loss unless conservation action is taken. Although it has been noted that start year for 10 year abundance trends can be biased for insect populations which can be highly stochastic in nature (Fox et al. 2019). The 20 year abundance trends are negative by both methods (Supplementary material 3)

The base year was set as the first year and taken to be the same across methods for comparison purposes. It could be argued that using a later base year might produce more robust results for some species, for example for Green Hairstreak as discussed in “Comparison with regional indices produced from national analyses” section. However this would ideally be chosen by defining suitable criteria.

In Surrey, the abandonment of coppicing in the 1950s resulted in the loss of three woodland butterflies (Duke of Burgundy, Pearl-bordered Fritillary and Small Pearl-bordered Fritillary during the period. High Brown Fritillary was lost in 1960. Currently, heathland butterflies are suffering the most in Surrey. Grayling and Silver-studded Blue were lost from the North Downs (calcareous grassland with the scarp slope south facing) after the abandonment of marginal agriculture land after the Second World War, and the introduction of Myxomatosis into the rabbit population in the 1950s. Their decline continues on the Heathlands from the monitored sites consistently over the last 20 years. Producing robust indices for regions such as Surrey is vital for measuring ongoing population changes.

The approach used in this paper has wider applicability, both to the study of butterfly populations in other regions in the UK, as well as potentially other seasonal insects with suitable data sets. At all scales, including nationally, drawing upon other sources of information to measure species’ flight period may have particular relevance for rarer species where transect data may be insufficient for robust flight period estimation. There is also relevance beyond the UK, for example to other butterfly monitoring schemes in Europe (Van Swaay et al. 2019) and beyond. Newly established or small-scale monitoring schemes maybe be able to benefit from using information from other available data sources to improve upon flight period estimation from transect data alone.