Introduction

Effects of forest fragmentation on vegetation at edges of forest remnants have been studied in ecosystems worldwide. At created forest edges, greater temperature extremes, increased light exposure, lower humidity, and higher wind speeds than forest interiors influence vegetation through structural damage, production of deadwood and increased growth or regeneration (Chen et al. 1995; Laurance et al. 1998a; Didham and Lawton 1999; Harper et al. 2005). Despite numerous studies on vegetation at edges, site-specific results (Franklin et al. 2021) necessitate further investigation to understand impacts of fragmentation in a particular region. One characteristic that varies substantially is the distance of edge influence (DEI, the extent to which a difference in vegetation can be detected compared to interior forest), which is particularly important for conservation as it can be used for mapping and planning. Although many studies have reported low estimates of DEI of 20 m or less for plant variables (Harper et al. 2005; Franklin et al. 2021), larger estimates of up to 100 and even 300 m have been reported from the Brazilian Amazon forest (Laurance et al. 1998a).

Variable results from edge studies suggest that other factors affect edge influence such as topography, which may overshadow or interact with edge influence on vegetation (Chapman and McEwan 2013; Lippok et al. 2014; Jucker et al. 2018). Topography alters forest characteristics; small differences in relative elevation can affect forest structure, composition, and diversity through changes in hydrology and soil characteristics (Allié et al. 2015; Jucker et al. 2018). Valleys have taller canopies, more trees, greater basal area, and greater tree species and structural diversity compared to ridges and steep slopes because of greater productivity (Homeier et al. 2010; Detto et al. 2013; Fortunel et al. 2018; Jucker et al. 2018). Edge influence on vegetation (the difference between the edge and interior) was found to be more pronounced on slopes than hill tops (Guerra et al. 2013).

Structural diversity is a useful metric to compare edge studies as it has been shown to be a better predictor than species diversity when related to ecosystem functions such as productivity (Proulx and Parrott 2008). Vertical foliage distribution and horizontal structural complexity are important for bird habitat (Zellweger et al. 2013). Conserving a high level of diverse plant types and structural elements in forest remnants creates a broad range of habitats that host wildlife, plants, and other species (Farah et al. 2017; Silveira dos Santos et al. 2022). However, structural diversity has not been comprehensively evaluated or related specifically to edge influence.

High habitat diversity is particularly apparent in Atlantic Forest, a biodiversity hotspot that surpasses most of the Amazon rainforest in plant species diversity per unit area (Myers et al. 2000; Forzza et al. 2012; Joly et al. 2014; Janisova et al. 2016). However, due to urban development, cattle ranching, and plantations, only 16% of the original forest remains (Riberio et al. 2009). Understanding the impact of edge influence on structural diversity in Atlantic Forest is important for developing management, conservation, and restoration strategies. Our objectives were (i) to determine the DEI on vegetation in Atlantic Forest, (ii) to compare the effects of edge influence vs. topography on vegetation structure, (iii) to determine whether the effect of slope varies depending on distance from the edge and (iv) to assess patterns of different measures of diversity (structural, taxonomic) across forest edges. We assessed responses of vegetation structure including structural diversity and abundance of different plant groups. Despite high levels of diversity and impacts of fragmentation, we know of no studies that have quantified DEI on vegetation in the Atlantic Forest. We hypothesized that DEI would be extensive as in other tropical forests, but might be moderated by topography.

Methods

Study area

We conducted our study in the southeastern part of the Atlantic Forest surrounding the Cantariera-Mantiqueira corridor, approximately 70 km from the city of São Paulo (Fig. 1). Our study area consists of a fragmented landscape of primary and secondary forest surrounded by pastureland, agriculture, forestry (i.e., Eucalyptus plantations) and rural and urban settlements. Seasonal forest is in different succession stages due to fire history. The climate is humid subtropical with hot (25 °C to 35 °C) humid summers and milder winters (10 °C to 20 °C). Elevation varies between 700 and 1700 m above sea level.

Fig. 1
figure 1

Map of the locations of the transects (a), lateral representation of a transect (b) and partial representation of the distribution of plots along a transect (c). For (a), black triangles represent the 12 transects sampled in July-August and purple pentagons represent the 12 transects in October-December; the inset map shows the location of study area in Brazil. The source of the map is Mapbiomas (n.d.)

Full size image

Data collection

Two datasets were collected in July–August (winter) and October–December (spring) in 2015; each one sampled 12 forest edges for a total of 24 edges. The non-forest side of the edge consisted of livestock pastures with cattle ranching; most edges were fenced.

At each edge, we sampled vegetation in contiguous 2 × 2 m plots along a 250-m transect perpendicular to the edge extending 50 m into the pasture and 200 m into the forest for a total of 125 plots for each of the 24 transects. We defined the edge as the limit of continuous forest canopy. In each plot, we estimated cover of different plant groups including trees, saplings, shrubs (July dataset only), tree ferns/palms, ferns, lianas, vines, snags, graminoids, epiphytes, moss, and litter. Cover categories were < 10%, 10–20%, 20–50%, 50–80%, and > 80% (July dataset), and < 10%, 10–30%, 30–60%, 60–80%, and > 80% (October dataset); mid-points were used for analysis. For the October dataset, we sampled the number of trees by diameter classes (< 3 cm, 3–10 cm, 10–30 cm, and > 30 cm diameter at breast height) and estimated cover for each plant family. We assessed vertical structure by visualizing a column above each plot divided into 2 × 2 × 2 m cubes and estimated the abundance (July dataset) or determined the presence/absence (October dataset) of woody (lianas, branches, tree trunks) and leafy (leaves, fleshy vines) material within each cube.

Every 8 m, we recorded altitude, elevation, and coordinates using GPS, and the change in relative elevation with a clinometer. We estimated canopy cover every 8 m for the October dataset by taking a photo of the canopy over the plot at 1.4 m using an iPhone camera and visually estimating the percentage of canopy cover.

Analysis

We calculated functional diversity in each plot using the Shannon diversity index:

$$H^{\prime} = - \sum {\left[ {p_{i} *\ln \left( {p_{i} } \right)} \right]}$$

where pi was the proportion of each of plant group. Functional richness was the number of plant groups in each plot. The Shannon index was also used to calculate the diversity of plant families. For leafy and woody vertical diversity of branches and leaves, we considered each occupied 2 m cube as a pseudospecies such that pi was the proportion of leafy or woody material within each cube. Leafy and woody vertical richness was the number of cubes occupied by leafy or woody material for each plot. Canopy height was determined as the highest 2 m cube occupied by leafy or woody material.

We estimated the magnitude of edge influence (MEI) and DEI for functional, family, leafy and woody richness and diversity, individual plant groups, individual plant families (frequency > 10%), and density of trees in diameter classes using the randomization test of edge influence (RTEI) Add-In in Microsoft Excel (Harper and Macdonald 2011). The MEI measures the strength of edge influence:

$${\text{MEI}} = \left( {x_{d} - x_{i} } \right)/\left( {x_{d} \, + \,x_{i} } \right),$$

where xd is the average of the variable x at distance d from the edge and xi is the average of the variable x in interior forest (Harper et al. 2005). We considered 150 to 200 m (26 reference distances) to be interior forest.

DEI measures how far from the edge a response variable significantly differs from interior forest by testing the significance of values for each distance using randomization tests. We used RTEI with blocking using the following steps (Harper and Macdonald 2011). (1) For each transect, we randomly selected an ‘edge’ value from the data set consisting of the value at a given distance from the edge and all interior forest values. (2) Randomized differences were calculated between the average of the randomly selected ‘edge’ values for all transects and the average of all the remaining ‘interior’ values. (3) These first two steps were repeated for a total of 5000 permutations to create a distribution of randomized differences. (4) The percentile of the observed difference between the edge and the interior within the distribution of the randomized differences was compared to the p value, for which we used p = 0.05 for a two-tailed test. DEI was then estimated as the set of three or more consecutive distances (or segments of three or more distances separated by one or two distances) over which the average response was significant.

To assess the influence of distance from edge, slope, and season (wet/dry for the July/October datasets) on structural diversity and the cover of plant groups, we performed spatial generalized linear mixed models (GLMMs). GLMMs are flexibile in accounting for spatial autocorrelation as they fit overall fixed effects with linear predictors containing random effects and spatially autocorrelated within-group errors (Dormann et al. 2007). Slope, the difference in elevation across each 8 m segment, was transformed to squared slope to account for its quadratic relationship with the response variables. We used the glmmPQL function (Dormann et al. 2007) from the packages MASS (Venables and Ripley 2002) and nlme (Pinheiro et al. 2015) in R 3.2.2. (R Core Team 2015). We used a Gaussian distribution for structural diversity and a negative binomial distribution for the cover of plant groups. We only included significant interactions between distance from edge and squared slope. We applied the models for the entire transect and for the first 25 m from the edge to detect finer scale changes.

We used wavelet analysis in PASSAGE 2.0 (Rosenberg and Anderson 2011) to assess patterns of functional, family, leafy vertical and woody vertical richness across forest edges, and patterns of woody and leafy richness and diversity for the July dataset. We used the Haar wavelet template and wavelet position variance (with 10% maximum scale) to identify transitions in vegetation structure along transects (Dale and Mah 1998; Kembel and Dale 2006). We assessed the significance of abrupt transitions using randomization tests of position variance with 999 permutations and a 95% confidence interval. We considered abrupt changes as two or more distances with significant wavelet variance (excluding single distances).

Results

Although some aspects of vegetation structure were significantly affected by the forest edge, edge influence did not extend very far into the forest. Canopy cover and height increased from low values in the non-forested area to around 75% cover and 11 m in height within the forest (Fig. 2). Four different measures of richness followed the same trend (Fig. 3) and patterns of diversity were similar (not shown). The richness of plant groups and families increased abruptly at the edge from about one or two per plot, respectively, in the non-forested area to five per plot within the forest. Patterns of vertical richness (both woody and leafy material) appeared more gradual from the edge to interior forest, increasing from one layer of leafy material and virtually no woody material to about five layers of each in the forest.

Fig. 2
figure 2

Canopy cover (a) and height (b) along the edge to forest interior gradient. Negative distances are in the adjacent non-forested area and 0 m is at the forest edge. Sample size is n = 12 transects for canopy cover and 24 transects for canopy height

Full size image
Fig. 3
figure 3

Richness of plant groups (a), families (b), and the number of vertical 2 m cubes with woody (c) and leafy material (d) along the edge to forest interior gradient. Negative distances are in the adjacent non-forested area and 0 m is at the forest edge. Sample size is n = 24 transects

Full size image

The MEI of 24 out of 28 variables was negative with lower values at the edge compared to interior forest, with the exceptions of graminoids, moss, Asteraceae, and Poaceae, which had greater values at the edge (Table 1). The DEI ranged from being limited to the non-forested area (including only negative distances such that values at the edge were not significantly different from interior forest) to distances generally up to 20 m; a few exceptions included maximum DEIs of 88, 72, and 58 m for canopy height, and woody and leafy vertical richness, respectively. Edge influence on richness and diversity of plant groups extended 10 m into the forest. Although negative edge influence on vertical diversity (woody and leafy) extended 12–20 m into the forest, lower vertical richness compared to interior forest was found up to approx. 60 m from the edge. Richness and diversity of families had lower absolute values of MEI of − 0.20 to − 0.28 and DEI of 0 m compared to other measures of diversity. Edge influence was greatest for medium sized 3–10 cm diameter trees compared to other size categories; the density of the largest trees was not significantly different at the edge compared to interior forest. The MEI and DEI for individual structural components varied; notably vine cover was significantly lower at the edge but significantly higher 34–38 m from the edge compared to interior forest. Edge influence on individual families was generally limited to the non-forest area (values at the edge were not significantly different from interior forest); however, DEI extended to 8 m for Rubiaceae and the cover of Asteraceae was significantly greater 14–20 m from the edge compared to interior forest.

Table 1 Magnitude and distance of edge influence (MEI and DEI) for vegetation responses within Atlantic Forest, Brazil
Full size table

In terms of relationships with topography, functional diversity significantly increased with distance from edge across for both sets of distances (25, 200 m) but there was no significant correlation with slope (Table 2). Diversity during the wet season was about 20% less than in the dry season, regardless of distance from edge or slope (regression coefficient of − 0.19). Distance affected most structural groups, which usually increased in cover with distance from edge. Slope only had an effect on tree ferns/palms and snags; the cover of both significantly decreased with slope at the 200 m scale with regression coefficients of − 0.015 and − 0.010, respectively. Trees, vines, snags, epiphytes, and litter had significantly greater cover in the wet season. There was a significant interaction between distance and slope for ferns and epiphytes at the 25 m scale (regression coefficients = 0.02 and 0.036 for ferns and epiphytes, respectively); the influence of distance on fern and epiphyte cover was significantly greater on more abrupt slopes and there was less influence of slope on fern and epiphyte cover at greater distances from the edge.

Table 2 Estimates of regression coefficients ± standard error with p value (in brackets) of each explanatory variable [distance, squared slope, and season (dry/wet)] for 25 and 200 m from the edge for spatial generalized linear mixed models for functional diversity and cover of different plant groups (n = 600 for the 200 m analysis and n = 75 for the 25 m analysis)
Full size table

Results of the wavelet analysis showed more abrupt changes in richness within a few meters of the forest edge, but the proportion of transects with this pattern depended on the type of richness (Fig. 4). About a third of the transects had abrupt transitions in the richness of plant groups, families, and layers of woody material at or near the forest edge compared to only a fifth of transects with abrupt transitions in the number of layers of leafy material. There were more abrupt changes in the number of layers of woody and leafy material throughout the transects, but very few for functional or family richness. We observed an interesting difference between woody and leafy material richness vs. diversity (Fig. 5). Although up to 40 to 60% of transects had abrupt transitions at the edge for diversity, fewer than 20% of transects had an abrupt transition in richness at the edge.

Fig. 4
figure 4

Proportion of transects with significant abrupt changes along the edge to forest interior gradient for richness of plant groups (a), families (b), and the number of vertical 2 m cubes with woody (c) and leafy material (d). Negative distances are in the adjacent non-forested area and 0 m is at the forest edge. Sample size is n = 24 transects except for n = 12 transects for (d)

Full size image
Fig. 5
figure 5

Proportion of transects with significant abrupt changes along the edge to forest interior gradient for diversity and richness of the number of vertical 2 m cubes with woody (a, b) and leafy material (c, d). Negative distances are in the adjacent non-forested area and 0 m is at the forest edge. Sample size is n = 12 transects

Full size image

Discussion

Although edge influence on structural and taxonomic diversity in Atlantic Forest was not very extensive, forest edges had a distinct structure characterized by low canopy cover and height, low density of all sizes of trees, and greater abundance of graminoids. Lower canopy cover and height at the edge than the interior was probably due to strong winds and tree mortality common at forest edges (Oosterhoorn and Kappelle 2000) including at tropical pasture edges (Laurance et al. 1998a). Other studies on agricultural edges in tropical forests also reported low canopy tree abundance (e.g., Kapos et al. 1997; Laurance et al. 1997; Viana et al. 1997; Williams-Linera et al. 1998; Oosterhoorn and Kappelle 2000) and shorter tree height (Camargo and Kapos 1995). A synthesis by Franklin et al. (2021) found that most studies of anthropogenically created edges found lower tree abundance but higher abundance of snags and tree regeneration; edge responses for graminoids and forbs were mixed. At our agricultural edges, graminoids may have spread to the forest edge and out-competed regenerating trees leading to lower rather than higher cover and density of saplings and trees. Other studies of tropical agricultural edges found lower recruitment or understorey tree density (Turton et al. 1997; Viana et al. 1997; Benitez-Malvido 1998; Oosterhoorn and Kappelle 2000). Although the edges we studied were maintained, they had not developed a side canopy of greater vegetation growth typical of other edges (e.g., Matlack 1993). Often anthropogenic edges exhibit edge sealing, whereby dense vegetation develops at sharp edges maintained by human activity (Harper et al. 2005). Instead, our results indicate that these are degraded forest edges dominated by graminoids with lower cover of most vegetation.

Negative edge influence (lower values at the edge) for all indices of diversity contrasts with most research findings of greater plant species richness and diversity at anthropogenic edges (Franklin et al. 2021). However, other tropical edge studies have found lower richness or diversity (e.g., Olupot 2009; Mendonca et al. 2015) and a recent global review found that lower species richness is common at tropical edges (Willmer et al. 2022). In tropical forests, fewer families and plant groups are adapted to open canopied conditions found at the edge with increased light and wind; this is reflected in our results as more families and plant groups had negative rather than positive MEI. Lower woody and leafy vertical diversity can be explained by a shorter canopy at the edge that narrows the range from the ground for leafy and woody structure stratification (Marques et al. 2015; Dial et al. 2011). The shorter DEI of woody compared to leafy structures may show that edge effects are more pronounced and intense on regenerative leafy vegetation rather than slow growing woody material. Weaker MEI and longer DEI evident for vertical diversity and especially richness may be evidence of edge expansion (Harper et al. 2005). However, most other types of diversity exhibited a steeper gradient of higher values at the edge and short DEI, which is characteristic of edge sealing (Harper et al. 2005). There may be a lag before edge degradation affects family and functional diversity or these diversity measures may be more resistant to edge influence.

Edge characteristics of lower structural and taxonomic diversity may be signs of degradation of maintained forest edges in Atlantic Forest, but they did not extend very far. Overall, edge influence on vegetation limited to 20 m or less was narrow compared to other tropical forests. Franklin et al. (2021) found that estimates of DEI for anthropogenic edges were greater in tropical forests compared to boreal and temperate forests but varied considerably from 0 to 10 m in Mexico, Panama, and the Amazon (Williams-Linera et al. 1990, 1998; Sizer and Tanner 1999) to 100 to 210 m in Africa and the Amazon (Young et al. 1995; Laurance et al. 1998a, b). Our results suggest that edge influence does not extend as far into Atlantic Forest fragments as compared to other tropical forests, but with such disparate results other factors are likely involved.

Because edge influence is reported to be much more extensive in the relatively flat lowland Amazon tropical forests, we considered whether topography might alleviate edge influence in the hilly remnants of Atlantic Forest. In response to our second objective, we found that distance from edge had more of an effect on vegetation structure than slope, which only had a significant effect on the cover of tree ferns/palms and snags. We expected more of a slope effect since topographical characteristics can affect the spatial distribution of vegetation and alter canopy structure and forest dynamics (Oliveria-Filho et al. 1998; Jucker et al. 2018). Steeper slopes usually have more canopy gaps (Ediriweera et al. 2008) and greater structural complexity (Jucker et al. 2018) because of greater exposure to wind, fog, and solar radiation (Werner et al. 2012), limiting nutrients and water availability (Werner and Homeier 2015; Jucker et al. 2018), and variable light distribution (Getzin and Wiegand 2007). However, Muscarella et al. (2020) found that topographic heterogeneity only weakly affected tropical forest species and functional diversity. Snag cover was the only variable in our study affected by slope and not by edge influence; this suggests that tree mortality is due to factors other than greater wind at edges or that wind patterns may be more impacted by topography than edge creation.

The lack of interaction between slope and distance from edge for most variables means that in response to our third objective, edge influence did not vary with slope as we expected (Oliveira-Filho et al. 1998; Guerra et al. 2013). Edge influence on vegetation structure has been found to be more pronounced on slopes than hill tops in the Atlantic Forest, but topographic effects did not override edge influence (Guerra et al. 2013). Both Pereira et al. (2007) and Guerra et al. (2013) contend that topography must be considered as a potential modulating factor to understand the effects of fragmentation in the complex Atlantic Forest mountainous landscape. However, in our study, slope did not seem to impact edge influence and had less of an impact than distance from edge or season. The exception was less edge influence on steeper slopes for ferns and epiphytes, which was opposite to the interaction found by Guerra et al. (2013).

Therefore, other factors are likely responsible for less extensive edge influence in Atlantic Forest including topographic position, proximity to the coast, secondary disturbance within the forest and edge maintenance. Although we measured slope, we did not quantify topographic position such as whether edges occurred on hill tops or valley bottoms. Topographic position could have more of an impact on vegetation structure than edge influence as organic matter, nutrients, and seeds move downslope creating a gradient in vegetation that might mask the edge to interior forest gradient. In valleys, greater productivity and turnover result in taller trees, more gaps, vertical stratification and higher tree species diversity (Werner and Homeier 2015; Fortunel et al. 2018; Homeier et al. 2010; Detto et al. 2013; Jucker et al. 2018). Fragmentation may have created edge-like conditions throughout the Atlantic Forest, and impacted the forests within vegetation remnants (Ribeiro et al. 2009; Farah et al. 2017). In these heavily fragmented landscapes, the surrounding anthropogenic matrix shapes the fauna and flora responses within forest remnants, such as has been observed in birds (Barros et al. 2019), dung beetles (Martello et al. 2016), bees and wasps (Medeiros et al. 2022), ants (Martello et al. 2022), and seed predation (Mendes et al. 2016) within Atlantic Forest. Land use history can also affect processes such as tree recruitment and growth in fragments of Atlantic Forest (Torres et al. 2023). These factors likely create widespread variability in vegetation structure and composition within interior forest that overshadows variability due to edge influence.

Less extensive edge influence in these forests is also apparent from our results of abrupt changes in diversity within ~ 5 m of the edge, revealing a pattern of a steep gradient at the edge that does not extend further into the forest. Additional abrupt transitions throughout individual transects also reveal heterogeneity throughout the forest remnants. Inherent heterogeneity in structural and transitional diversity may contribute to shorter DEI that is measured within the context of the variation in interior forest. Our results suggest that abrupt changes in diversity can be detected at maintained agricultural edges, whereas there was little to no evidence of abrupt changes in vegetation due to fine scale heterogeneity in vegetation structure at natural inherent wetland edges (Brownstein et al. 2013; Harper et al. 2021) and insect outbreak edges maintained by moose browsing (Franklin and Harper 2016).

Wavelet analysis allowed us to differentiate edge influence on vegetation characteristics to reveal different edge structure compared to interior forest. Structure variables (functional richness, woody material) had more significant changes at the edge compared to plant families and leafy material. Therefore, it appears that there is a more abrupt transition in woody structure at the forest edge compared to a more gradual gradient in taxonomic diversity and leafy structure. This is compatible with the hypothesis that secondary responses (species composition, leafy material) extend further into the forest than primary responses (structure) (Harper et al. 2005). We note that our DEI results do not corroborate this conclusion (e.g., for trees), which suggests that spatial pattern analysis provides additional insight into the effects of edge influence. Another interesting result is that an abrupt transition at the edge was notable only for vertical diversity but not for richness. Changes in richness were more gradual from the edge to the interior, which matches the gradual change in canopy height. Along the same gradient, the amount of leafy and woody material must have filled in the layers at the edge, thus creating somewhat of a side canopy, albeit a shorter one than in interior forest. Therefore the wavelet results of our vertical structure sampling does provide some evidence of a side canopy that was not detectable from our other results.

Conclusions and conservation implications

Forest edges in the agricultural forest mosaic of Atlantic Forest near São Paulo are abrupt transitions from short grass-dominated pasture to tall closed canopied forests with high taxonomic and structural diversity. Edges themselves were characterized by low diversity, but DEI was short for most structural variables and diversity measures. Our results did not provide any evidence that topography (slope) may explains the lack of extensive edge influence; however, topographic position may play a role.

We found evidence of edge degradation, reduced recruitment and a shorter canopy at the edge. Although some variables such as tree and sapling cover suggested that edge sealing had not taken place, there was evidence of a side canopy of more diverse leafy vertical structure. We arrived at our understanding of edge structure only by using different analyses for the same data set: DEI and wavelet analysis to examine patterns in more detail at a fine scale. Adding multivariate analysis also allowed us to differentiate effects of topography and edge influence. Having all three approaches provided complementary information to our analysis of taxonomic and structural diversity at anthropogenic edges in Atlantic Forest.

Contrary to our hypothesis, DEI was not very extensive in Atlantic Forest but it is important to note that even a DEI of 20 m or less can be substantial in heavily fragmented landscapes (Riutta et al. 2014; Franklin et al. 2021) such as Atlantic Forest. Comparisons of results from other studies on anthropogenic edges in tropical forests make it clear that DEI is not similar and must be determined for each region. Further research is needed to determine which factors affect DEI even for the same edge types within the same biome. For Atlantic Forest, signs of edge degradation suggest that edge influence should be considered in conservation planning even though edges are narrow, and the abrupt gradient in vegetation structure may be a factor in explaining wildlife responses to edge creation.