Introduction

Unbiased and precise estimators of abundance and density are fundamental to the study of population ecology and essential for effective conservation and management decisions. A common approach to estimating the abundance and density of a species is to capture, mark, and recapture animals to apply capture–recapture (C-R) analyses (White et al. 1982). In particular, using C-R to quantify the populations of rare and/or elusive carnivores is increasingly widespread. This is due to the successful implementation of remote sampling techniques, such as hair snares or scat collection which allow the isolation of individually-identifiable DNA markers and photographic-sampling of species with uniquely identifying physical marks (Long et al. 2008).

Given the small sample sizes encountered in most carnivore studies and the nearly universal finding that detection probability is affected by heterogeneity among animals and occasional trap responses (Noyce et al. 2001; Boulanger et al. 2004a, b), carnivore biologists primarily implement closed, versus open, C-R models to estimate abundance, \( (\hat{N}) \) but see Karanth et al. (2006) and Gardner et al. (2010a). To compare populations across areas it is necessary to convert abundance to density \( (\hat{D}) \), yet traditional C-R analyses provide no direct estimate of \( (\hat{D}) \). \( (\hat{N}) \) must be divided by the sampling area (A) to estimate density \( (\hat{D} = \hat{N}/A) \). However, unless the sampling area is confined by natural barriers (Mace et al. 1994), animals have the potential of permanently immigrating into or emigrating from the sampling grid, thus violating the basic assumption of geographic closure in these closed C-R models. Additionally, at least some sampled individuals will have home ranges that extend beyond the edges of the sampling grid, thus temporally emigrating from the grid, and positively biasing \( (\hat{D}) \) due to this “edge effect” (White et al. 1982; Boulanger and McLellan 2001). Given that (1) many carnivores have large home ranges and (2) financial and logistical constraints generally prohibit sampling areas of necessary size (Bondrup-Nielsen 1983) or simultaneously tracking animals across these edges (White and Shenk 2001), the edge effect is likely to lead to biased results when sampling carnivores using grids (Greenwood et al. 1985; Mowat and Strobeck 2000; Boulanger and McLellan 2001).

If we assumed movements across the sampling area edge are random, \( (\hat{N}) \) would likely not be biased, but would correspond to the superpopulation \( (\hat{N}_{s} ) \), or those animals that occupy the sampling area and an unknown amount of the surrounding area (Kendall 1999). To accurately estimate density of what is actually a geographically open population using closed C-R models, it is necessary to estimate the effective trapping area (ETA; Wilson and Anderson 1985), or the area that pertains to the \( (\hat{N}_{s} ) \) estimate \( (\hat{D} = \hat{N}_{s} /{\text{ETA}}) \). Despite this frequent need to estimate the ETA, there is still much debate on a robust solution; most recommendations suggest variations on ad hoc boundary-strip methods (Soisalo and Cavalcanti 2006; Dillon and Kelly 2007; Maffei and Noss 2008; Balme et al. 2009).

Spatially-explicit C-R models (Efford et al. 2009a; Royle et al. 2009) incorporate the spatial component of the sampling array in the C-R framework thereby estimating density directly and accounting for the edge effect without the need of an ad hoc ETA estimate. Field studies have recently provided empirical support for the use of a maximum likelihood spatially-explicit C-R model (ML-SECR; Obbard et al. 2010) and a Bayesian spatially-explicit C-R model (B-SECR; Gardner et al. 2010b) to estimate density of geographically open populations of a large ranging carnivore, the American black bear, Ursus americanus. Additionally, a recent simulation study has also provided support for the ML-SECR model under a variety of scenarios, except when animal home range configurations are dramatically asymmetric (Ivan 2011). Despite the availability of these newer models, it is still common for studies to use traditional ad hoc density estimation techniques \( (\hat{D} = \hat{N}_{s} /{\text{ETA}}) \); Sarmento et al. 2009; Gopal et al. 2010; Kolowski and Alonso 2010; Negrões et al. 2010; Sarmento et al. 2010).

In addition to the challenge of dealing with geographically open populations, carnivores often have low detection rates, even with intense sampling efforts, which can either inhibit the application of even closed C-R analyses or simply provide imprecise estimates (White et al. 1982; Maffei et al. 2004). Thus, carnivore C-R studies, especially those using hair snares, often use bait (food reward) or lure (non-food reward) to attract animals to the detection device to more effectively (re)capture individuals (Gardner et al. 2010b; Obbard et al. 2010). In contrast, photographic-sampling studies less frequently use bait or lure (Trolle et al. 2007; Gerber et al. 2010), but rather often place cameras on trails to increase detection (Dillon and Kelly 2007). Few studies have examined the influence that these attractants may have on C-R population estimation. Using attractants can potentially increase the sample size of detected and/or repeated detections of individuals, and thereby increase detection probability for closed C-R analyses. The advantages include more efficient model selection, increased estimate precision, and the need for less sampling length or effort, thus reducing project costs (White et al. 1982). However, attractants may also introduce bias to the density estimate, irrespective of, or in combination with, the edge effect (Mowat and Strobeck 2000; Gardner et al. 2010b), by disrupting natural spatial and temporal movement patterns within the sampling area, “pulling” animals onto the sampling area, and/or deterring a proportion of the population (e.g., by sex or age) from being detected (Noyce et al. 2001).

To appropriately estimate carnivore density given the potential biases of edge effects and/or attractants, it is necessary to assess and account for violations of the closure assumption in C-R abundance and density estimation. In this paper, we (1) compare methods to account for geographic closure violation in estimating density of the Malagasy civet Fossa fossana, Müller 1776, (2) evaluate the effect of lure on permanent and temporary immigration and emigration (geographic closure), abundance and density estimation, maximum movement distances, and temporal activity patterns while photographic-sampling, as well as make recommendations for use of attractants in future studies, and (3) empirically compare four density estimators when it is necessary to use closed C-R models with a geographically open and ill-defined sampling area and make recommendations for future studies.

Methods

Study area and species

We studied Madagascar’s third largest endemic carnivore, the IUCN ‘near-threatened’ F. fossana, at the Sahamalaotra trail-system within Ranomafana National Park from 9 June–8 August, 2008 (Fig. 1; IUCN 2011). Sahamalaotra is montane rainforest, characterized by a 20-25 m tree canopy dominated by Tambourissa and Weinmannia spp. (Turk 1997). Fossa fossana is a mesocarnivore averaging 1.6 kg and 0.91 m in length. This animal is generally terrestrial, but exhibits some arboreal activity (Kerridge et al. 2003). Fossa fossana is a generalist predator; its diet includes rodents, lipotyphlans, crustaceans, snakes, frogs, lizards, and many insect taxa (Albignac 1984; Goodman et al. 2003). Fossa fossana populations are declining across Madagascar due to habitat loss and local hunting (Kerridge et al. 2003; IUCN 2011).

Fig. 1
figure 1

We placed 26 camera stations over a 6.53 km2 area along the Sahamalaotra trail system within the rainforests of Ranomafana National Park, Fianarantsoa province in southeastern Madagascar from 9 June–8 August, 2008

Full size image

Field methodology

We deployed 26 passive-infrared camera sampling stations on trails in a systematic grid with a random starting point using Deercam DC300 (DeerCam, Park Falls, USA) and Reconyx PC85 (Reconyx, Inc., Holmen, WI, USA) cameras. The photographic-sampling grid was designed based on a preliminary study (Gerber et al. 2010) and had 3.98 stations per km2 with an average distance and standard deviation of 566 ± 93 m between adjacent stations. Sampling stations consisted of two independently-operating passive-infrared cameras mounted on opposite sides of a trail to provide a photographic-capture of both flanks of each animal, thus improving individual identification in recaptures. Cameras were approximately 20 cm above the ground and set to be active for 24 h/day.

We sampled for 61 nights; during the first 36 nights we did not deploy attractants. Starting on the 37th night, 1–2 kg of chicken meat was secured within three layers of metal-wire-mesh at all sampling stations for an additional 25 nights of sampling. Chicken was inaccessible for consumption and acted as a scent-lure. We hung most of the chicken lure 2 m directly above the sampling station on a line tied between two trees. We also staked a small piece of chicken wrapped tightly in three layers of metal-wire-mesh on the ground. We checked sampling stations on average every 5 days to ensure continued operation and replaced batteries, film, and memory cards when necessary. We replaced chicken at least every other visit to ensure a maximum-volatile olfactory signal. By maintaining a strict schedule, we ensured that there was no time when lure was absent from any sampling station, thus reducing temporal variation at a station and among-station heterogeneity (Zielinski and Kucera 1995).

Animal identification and capture histories

Using F. fossana’s individually-identifiable spot pattern (Gerber et al. 2010), two researchers scored photographs independently, agreeing on the individual-identity of 96% of all capture events (n = 469) used to construct the capture histories necessary for closed C-R analyses; events for which the individual-identify could not be agreed upon were excluded from analyses. A capture event was all photographs of an individual within a 0.5 h period at a camera station (O’Brien et al. 2003). We created three datasets for comparison, (1) capture and recaptures from the complete sampling period (61 nights), (2) capture and recaptures from only the non-lure period (36 nights), and (3) capture and recaptures from only the lure period (25 nights). A sampling occasion was a 24 h period from 12:00 PM to 11:59 AM.

Assessing closure violation

We assumed demographic closure and used three methods to evaluate geographic closure. First, we used the closure hypothesis test of Otis et al. (1978), which assumes only heterogeneity in the recapture probability and is appropriate for evaluating permanent closure violations. Second, we emulated the Stanley and Burnham (1999a) closure test that assumes only time variation in recapture probability using the Pradel model (Pradel 1996) in Program MARK (v 5.1; White 2008). Third, we used the full capabilities of the Pradel model to evaluate geographic closure by estimating site fidelity (ф), immigration (f), recapture probability (p), and the composite variable of sampling area population growth rate (λ; Boulanger and McLellan 2001). Pradel model estimates of ф, f, and λ correspond to testing permanent closure violations. However, it is reasonable to assume a lower p near the grid edge will reflect temporary closure violations of animals moving across the grid edge. We included a priori biologically plausible models in this full Pradel analysis (Boulanger and McLellan 2001). Models included the effect of lure (lure) as a simple time effect between the non-lure and lure periods, males versus females (sex), and general location of animals on the camera grid (location). We classified location for each individual as either Core, individuals that were on-average detected within the interior of the sampling area, or Edge, animals that were only detected at camera stations on the edge of the sampling area.

We evaluated models using Akaike’s Information Criterion with a small sample size bias correction (AICc) and considered all models with ΔAICc <2 equally parsimonious; we model-averaged estimates among all models to incorporate uncertainty (Burnham and Anderson 2002). We calculated the relative importance of a parameter (R i ) as the sum of AICc weights (w i ) of all models containing the variable. We estimated overdispersion (ĉ) with a bootstrap goodness-of-fit test using the Cormack-Jolly-Seber model (Boulanger and McLellan 2001). Interaction models were prohibitive, thus using our global model ф(location + lure + sex) p(location + lure + sex), we estimated ĉ equal to 1.17. A ĉ correction was incorporated into model selection, so we present QAICc values. If geographic closure is met using the Pradel analysis, we expect site fidelity (ф) to be one, immigration (f) to be zero, and thus the sampling area population growth rate (λ) to be one.

Abundance and density estimation

We used four methods to estimate density using the complete, non-lure, and lure datasets for F. fossana. We defined a significant difference between methods when the 95% confidence intervals of two means overlapped no more than half the average margin of error; this is equivalent to a conservative hypothesis test at α = 0.05 (Cumming and Finch 2005).

First, we assumed random movement across the sampling grid edge (Kendall 1999) and estimated \( \hat{N}_{s} \) for all three datasets using the Huggins closed C-R model (Huggins 1991) in Program MARK. We constructed models using heterogeneity (h; Pledger 2-point mixture model; Pledger 2000), time (time), behavior (b), sex, mean-capture distance to the sampling grid edge (distedge), and mixed combinations. A lure effect was included in analyses with the complete dataset. We conducted model selection using AICc. We calculated density by dividing the model-averaged \( \hat{N}_{s} \) by the ETA and calculated variance using the delta method (Karanth and Nichols 2002).

We derived four variations of the ETA by calculating the expected half (1/2MMDM*) and expected full (MMDM*) mean maximum distance moved as the MMDM is known to increase with increasing recaptures (Tanaka 1972). We calculated MMDM* for observed animals as,

$$ E(\bar{W}_{i} ) = W^{*} (1 - e^{ - (i - 1)b} ) $$
(1)

where \( \bar{W}_{i} \) is the MMDM for animals captured (i) times, W* is the expected maximum distance moved for the given population, and b represents a model parameter (Jett and Nichols 1987). We evaluated W* using a likelihood function, hereafter referred to as MMDM*, in two ways, (1) using all animals detected at least twice (MMDM*) and (2) using the Core subset of animals (MMDM-Core*). We assumed Core animals are less likely to have truncated maximum movement distances. In contrast, Edge animals are very likely to have a maximum distance moved of zero (having not been detected at multiple stations) or a truncated distance as their home range is mostly outside the sampling area. MMDM*, 1/2MMDM*, MMDM-Core*, and 1/2MMDM-Core* values were applied as circular buffers to each sampling station, dissolving overlapping areas to calculate the ETA. We removed villages, roads, and agricultural land (non-habitat) from these buffered areas and restricted area estimation north of the Namorana river (Fig. 1) as it likely restricts regular movement (Gerber et al. 2010).

Second, we used the Huggins model to estimate \( \hat{N} \) of only the Core animals. We assumed Core animals’ home ranges are contained in the study area, thus \( \hat{N} \) pertains directly to the sampling area (A) and no ad hoc buffer value was needed. We used the same candidate models from the \( \hat{N}_{s} \) analysis to evaluate capture histories. We model-averaged to obtain Core-only \( \hat{N} \) and divided by A to calculate density; the variance was derived by dividing \( \hat{N} \) variance by the square of A (Weinberg and Abramowitz 2008).

Third, we used Program DENSITY’s ML-SECR model (v. 4.4; Efford 2009) to directly estimate density. The likelihood function was evaluated with a 2-dimensional numerical integration of 4,096 evenly distributed points using a Poisson point process within a rectangular area extending 1 km beyond the sampling area edge. We removed non-habitat, and again restricted area estimation north of the Namorana river. If both cameras at a sampling station were determined to have malfunctioned on any sampling occasions, we removed the trap from our models, only on those specific occasions. We compared the fit of three detection functions (half-normal, hazard-rate, and negative-exponential) to model detection probability variation away from an animal’s home range center. We fit a detection model by maximizing the conditional likelihood in which the parameters g 0 (detection process when a single detector is located at the center of an animal’s home range) and σ (spatial scale detection process away from the center of the home range) were modeled using a priori biologically plausible hypotheses. The same variables modeling detection probability in the \( \hat{N}_{s} \) were used, except we excluded the distedge covariate. Model selection was evaluated using AICc and we model-averaged results to derive \( \hat{D} \) and associated variance, constructing profile likelihood confidence intervals.

Fourth, we used the R package SPACECAP (v. 1.0; Royle et al. 2009) to apply the B-SECR model to estimate density. To compare with the ML-SECR estimates, we used the same 2-dimensional area as a state-space, removed non-habitat, restricted area estimation north of the Namorana river, and also incorporated sampling station malfunctions as described above. We allowed incorporation of a trap response in the model for all three datasets and ran 60,000 Markov chain Monte Carlo iterations. SPACECAP is limited to the half-normal detection function.

Effect of lure on movement and temporal activity patterns

To test the effect of lure on individual’s movements, we calculated the maximum distance moved (MaxDM) for all individuals and Core animals only, before and after lure was applied. We tested whether individuals detected during both sampling periods change their MaxDM using the Wilcoxon Signed Rank test (Zar 1998). We also tested whether MMDM of individuals captured ≥two times in each of the non-lure and lure sampling periods were different using all individuals (MMDM) and Core animals only (MMDM-Core) using the Wilcoxon Ranked Sum test (Zar 1998). Lastly, we considered ML-SECR model selection results to evaluate the effect of lure on σ and contrasted the MMDM* and MMDM-Core* for the non-lure and lure sampling periods.

We evaluated the effect of lure on the temporal activity of F. fossana by testing if activity distributions from data collected with and without lure were different using the non-parametric circular Mardia–Watson–Wheeler statistical test (MWW; Batschelet 1981). In addition, we estimated the mean temporal overlap between activity distributions using a kernel density analysis (Ridout and Linkie 2009). We defined a sample as the median time of all photographs of the same individual within a 0.5 h period, thus minimizing the issues of non-independence of consecutive photographs (O’Brien et al. 2003). We applied a kernel estimator from Ridout and Linkie (2009; see Eq. 3.3, smoothing parameter of 1.00). We tested for a shift in the proportion of activity in four time periods based on sunrise and sunset times during this study: dawn (0525–0724 hours), day (0725–1627 hours), dusk (1628–1827 hours), and night (1828–0524 hours). We derived the proportion of activity for each period from the kernel probability distribution and used a contingency table analysis with a likelihood ratio test to examine if animals spent a different amount of time in any temporal class after lure was applied at the sampling stations. We considered a difference (α = 0.05) in the activity distributions between the non-lure and lure datasets and/or a shift of activity among the four temporal classes to indicate a change in activity pattern due to lure.

Results

Animal identification and capture histories

We photographically captured 22 individual F. fossana from 61 sampling nights (Table 1). Eighteen of 22 individuals were detected in both the non-lure and lure periods; two unique individuals were detected only in the non-lure period and two unique individuals only in the lure period. We observed F. fossana attempt, but fail, to remove the staked-ground lure in only 6% of digital-camera capture events and did not observe any chicken being removed in 915 film images or 2,296 digital images. Despite significant efforts to maintain continuously working cameras, we had an average of 3.65 ± 3.05 SD malfunction days per sampling station.

Table 1 Photographic-sampling summary of the Malagasy civet (Fossa fossana), sampled with and without lure from 26 camera stations at the Sahamalaotra trail system within the rainforests of Ranomafana National Park, Madagascar from 9 June–8 August, 2008
Full size table

Assessing closure violation

We found our datasets of F. fossana to reject the assumption of geographic closure depending on the method employed, which varied by detection variation assumptions. The Otis et al. (1978) test did not reject the closure assumption during the non-lure period (Z = −1.15, P = 0.12), but did for both the lure period (Z = −2.771, P = 0.002) and the complete dataset (Z = −2.98, P = 0.001). The Stanley and Burnham (1999a) test similarly rejected the closure assumption for the complete dataset, as the model constraining site fidelity (ф) to one and immigration (f) to zero was given no support using only the Stanley and Burnham models (QAICc Weight (w i ) = 0.00; Table 2).

Table 2 Model selection summary (model likelihood >0.125) evaluating the effects of lure on geographic closure for the Malagasy civet (Fossa fossana) at a photographic sampling study grid in Ranomafana National Park, Madagascar from 9 June–8 August, 2008
Full size table

Using the full Pradel analysis, we found side fidelity (ф) and immigration (f) as constant, and recapture probability (p) varying by location and the use of lure, in the top model (Table 2). There was no evidence of permanent closure violations, as model-averaged λ ± SE for the complete, non-lure, and lure datasets were estimated at 1.00 ± 0.004, 1.00 ± 0.006, and 0.995 ± 0.008, respectively. Although there was no evidence of permanent closure violation, recapture probability ± SE was significantly higher for Core animals (Non-lure = 0.35 ± 0.04, Lure = 0.48 ± 0.04) than Edge animals (Non-lure = 0.13 ± 0.02, Lure = 0.21 ± 0.03), indicating potential temporary closure violation by Edge animals emigrating from the sampling grid, thus producing an edge effect.

Effect of lure on abundance, density, movements, and activity

Detection probability was affected by h, b, sex, and lure in most of our selected models for \( \hat{N}_{s} \), Core-only \( \hat{N} \), and \( \hat{D} \) of F. fossana (Table 3). We found that models of F. fossana \( \hat{N}_{s} \) using the complete-dataset included effects of h (R i  = 100%), distedge (R i  = 100%), b (positive trap response; R i  = 100%), sex (R i  = 100%), and lure (R i  = 93%) on the probability of detection. All models included h (R i  = 100%) in detection probabilities to estimate Core-only \( \hat{N} \). Additionally, a trap-happy b effect on the detection probability was clear in the Core-only \( \hat{N} \) complete dataset (R i  = 100%) and the non-lure (R i  = 97%), but not in the lure dataset (R i  = 49%). Males were detected more often than females when using the complete dataset for Core-only \( \hat{N} \) (R i  = 98%), but an effect of sex was less evident for the non-lure (R i  = 68%), and lure (R i  = 15%) datasets alone. In the ML-SECR model, we found the negative-exponential function fit all three datasets best and variation in g 0 and σ was best explained by h and/or sex (Table 3). Model selection for the B-SECR analysis is unavailable in SPACECAP (v. 1.0), thus the model fit is the a priori “best” model.

Table 3 Top abundance and density estimation models (model likelihoods ≥ 0.125) incorporating detection probability variation of the Malagasy civet (Fossa fossana), sampled with and without lure
Full size table

We found no effect of lure on estimates of \( \hat{N}_{s} \), \( \hat{N} \) (Table 4), and \( \hat{D} \) (Table 5) for each density estimation technique. We found higher average detection probabilities ±SE in our analyses of \( \hat{N}_{s} \) when using lure (capture probability = 0.33 ± 0.08, recapture probability = 0.45 ± 0.05) than when not using lure (capture probability = 0.07 ± 0.03, recapture probability = 0.31 ± 0.09). This increase in (re)capture probability increased the population estimate precision, as the coefficients of variation decreased when using lure, except for the SECR model estimates.

Table 4 Population abundance, the coefficient of variation (CV), and trap success of the Malagasy civet (Fossa fossana), sampled with and without lure
Full size table
Table 5 Four density estimates and associated coefficient of variation (CV) of the Malagasy civet (Fossa fossana), sampled with and without lure. Within each density estimator, we found no significant differences (defined as 95% confidence interval of two means overlap less than half the average margin of error) between the non-lure and lure datasets
Full size table

We found no effect of lure on the MaxDM of all individuals (W = 9.0, P = 0.16), nor Core individuals only (W = 0.00, P = 1.0). Similarly, we found no effect of lure on the MMDM of all individuals (Z = 1.125, P = 0.26), nor Core individuals only (Z = −0.317, P = 0.75). Within the ML-SECR model selection, we found no effect of lure on σ. The use of lure only changed MMDM* by 18 m and MMDM-Core* by 6 m (Table 5); this latter increase translates into a negligible increase of 0.2% in the ETA. The large difference between MMDM* and MMDM-Core* reflects the exclusion of animals with poorly sampled MaxDM.

We did not observe any shift in temporal activity pattern after lure was applied (W = 0.376, P = 0.83). The mean overlap of activity ± SE between the non-lure and lure datasets was 95.51 ± 0.02%. We found no significant difference in the proportion of activity during the dawn, day, dusk, and night periods for the non-lure and lure sampling periods (χ2 = 0.779, P = 0.68). Fossa fossana were predominantly active at night (85%) as compared to dusk (9%), dawn (6%), and day (<1%).

Comparison of density estimation analyses

Given that we found no effect of lure on \( \hat{D} \) (Table 5), we used the complete dataset to compare density estimation methodologies. We found \( \hat{D} \) derived as \( (\hat{D} = \hat{N}_{s} /ETA) \) varied considerably depending on the buffer value used to calculate the ETA; the 1/2MMDM* buffer produced the smallest ETA (7.99 km2) and thus the highest density estimate (Fig. 2). We found no differences in \( \hat{D} \) as estimated using (1) \( (\hat{D} = \hat{N}_{s} /MMDM) \)-Core*, (2) Core-only \( (\hat{N}/A) \), (3) ML-SECR, and (4) B-SECR (α = 0.05, Fig. 2). Our estimate precision was lowest with both SECR methods as these analyses include uncertainty and process variation in abundance and area estimation that is often underestimated with other density estimators.

Fig. 2
figure 2

Density and 95% confidence limits using four methods of estimation on the complete dataset for the Malagasy civet (Fossa fossana) with statistical significance among methods (95% confidence interval of two means overlap less than half the average margin of error) indicated with different capital letters (α = 0.05)

Full size image

Discussion

It is critical to understand whether animals are permanently or temporally immigrating to or emigrating from a sampling area (geographic closure) when using closed C-R models to estimate abundance and density, and to fully understand if the use of attractants biases these estimates. We found the Pradel model most useful for evaluating geographic closure, because it (1) is flexible in modeling recapture variation, especially to account for the common occurrence of heterogeneity, (2) uses model selection procedures to estimate the components of geographic closure, and (3) is not affected by high Type 1 errors, as are the other tests when there is a behavioral effect (White et al. 1982; Stanley and Burnham 1999b), or heterogeneity (Stanley and Burnham 1999b). Given our findings of behavior and heterogeneity variation in our datasets, we did not meet the assumptions of the Otis et al. (1978) or Stanley and Burnham (1999a) tests, thus these two tests likely rejected closure due to assumption violations concerning detection variation rather than true violation of the closure assumption. When sample sizes are inadequate to use the Pradel model, as with many large carnivore studies, Otis et al. (1978) and Stanley and Burnham (1999a) can be useful when model assumptions are met; otherwise, no test of closure is appropriate.

Despite concern that attractants might compromise geographic closure, we found no indication that F. fossana were permanently immigrating to or emigrating from our sampling area using the Pradel model analysis. This is likely due to a combination of the distance the lure could be detected and the likely territorial behavior of F. fossana. If the maximum distance F. fossana could detect the lure was small compared to its home range, only animals already overlapping sampling stations would be affected. Thus, lure could increase the detection of an animal within a small area around the sampling station. Alternatively, if the detection distance of the lure was large, animals would not be “pulled” onto the sampling area because of territoriality. Like many terrestrial carnivores, F. fossana likely defends a territory, thus preventing individuals from moving into an area where they do not normally occur. For example, the Malay Civet (Viverra tangalunga, Gray 1832), which similarly occupies secondary rainforest, is territorial (Jennings et al. 2006).

We found the use of lure did not alter abundance or density estimates of F. fossana, regardless of the density estimation method used. Similarly, we found no effect of lure on maximum movement distances or temporal activity of F. fossana. The latter is an important finding for photographic-sampling studies, which often evaluate temporal activity (Grassman et al. 2006). Our entire sampling period was conducted over 61 days, solely within the cold-dry season, such that it would be reasonable to assume that if we had found any differences between the non-lure and lure periods, the effect could likely be attributed to a lure effect, rather than temporal or seasonal variation. As such, we recommend future studies to consider whether it is reasonable to assume a lack of temporal or seasonal variation in the population or movement parameters before concluding whether there is an effect of lure or bait.

Given the challenges of detecting carnivores frequently enough to effectively apply closed C-R analyses, our results suggest lure can be used while remotely sampling some territorial animals without risking closure violations, alterations of abundance, density, or temporal activity pattern. Our findings are particularly relevant to methodologies such as hair snares that often employ attractants to detect carnivores. Higher detection rates from using lure can increase estimate precision and reduce needed sampling effort and costs. Although not employed in this study, post-hoc collapsing of sampling occasions may increase detection probabilities and thus increase abundance estimate precision as well (Dillon and Kelly 2007), but maybe not with ML-SECR density estimation (Efford et al. 2009b), and sampling efforts may still need to be quite large. We suggest our findings extend beyond our study animal and likely apply to territorial carnivores that can be attracted by any means to a detection device as F. fossana detections increased when using chicken as a scent-lure, despite receiving no food reward. Useful attractants will vary by species (Schlexer 2008) and would preferably be undetectable at large distances to reduce attracting animals from adjacent territories into the sampling grid or altering regular movement patterns. Our study demonstrates that attractants do not necessarily bias results and can be useful, likely provided that the attraction distance is small relative to the animal’s home range radius and the food reward is small. However, if lure is to be used, we suggest (1) testing its effect on the ecology of the study animal, and (2) maintaining a rigid schedule for reapplying the attractant, as to reduce temporal variation at a sampling-station or heterogeneity among sampling-stations.

Carnivore C-R studies using a grid design also face the dual challenges of the effects of sampling layout on (re)capture probabilities and the determination of the appropriate area for density analyses. We found the ML-SECR and B-SECR models estimated density significantly lower than all but one of our estimates using an ad hoc buffer value to determine the effective trapping area. In agreement with Obbard et al. (2010), we found that using a buffer of 1/2MMDM* on \( \hat{N}_{s} \) overestimated density compared to SECR model estimates, whereas our MMDM-Core* density estimate was similar to and not statistically different than either SECR density estimate. The MMDM buffer has been supported by several studies (Parmenter et al. 2003; Soisalo and Cavalcanti 2006; Trolle et al. 2007; Dillon and Kelly 2008); however, there is no theoretical framework for why this value should provide consistent and reliable density estimates. Obbard et al. (2010) argued that empirical support for \( (\hat{N}_{s} /MMDM) \) may reflect the underestimation of 1/2MMDM due to few recaptures per individual (Tanaka 1972), the truncation of movement distances due to the sampling area edge (Soisalo and Cavalcanti 2006), and the inclusion of zero distances moved (Dillon and Kelly 2007). In our study, we still found that the \( (\hat{N}_{s} /\raise.5ex\hbox{$\scriptstyle 1$}\kern-.1em/ \kern-.15em\lower.25ex\hbox{$\scriptstyle 2$} MMDM - Core^{*} ) \) density estimate was significantly higher than SECR model estimates even though 73% of our F. fossana individuals were recaptured ≥5 times, we modeled recapture rate, and we strategically ameliorated the issues of the sampling area edge by using Core animals, which had no zero distances moved. Further, in contrast to Obbard et al. (2010), we used a distance to sampling area edge covariate (distedge) to incorporate closure violation bias (temporary emigration) on variation in detection probability to more robustly estimate \( \hat{N}_{s} \) (Boulanger et al. 2004a). Though our corrected \( \hat{N}_{s} \) and 1/2MMDM-Core* buffer still produced a higher density than either SECR model, our estimate was less dramatically different (71–76%) than Obbard et al. (2010) found in some cases using the 1/2MMDM buffer (20–200%). Ultimately, the appropriate buffer value will depend on the characteristics of the sampling array layout (size, shape, and trap spacing) and the unknown home ranges of the sampled animals that may differ in size, shape, overlap, and proportion contained within the sampling area (Parmenter et al. 2003).

Given the uncertainties of using 1/2MMDM and MMDM to buffer \( \hat{N}_{s} \) in density estimation, carnivore studies often use both values, reporting two density estimates (Trolle et al. 2007). This is unsatisfying for conservation organizations attempting to identify populations and species at risk, as 1/2MMDM densities are almost twice that of using MMDM. Given the known constraints on measuring MMDM and the uncertainties in the appropriateness of any buffer value to calculate the ETA, it is best to abdicate ad hoc boundary-strip methods given the availability of newer statistical methods that avoid these issues (Efford et al. 2009a; Royle et al. 2009).

Of all four density estimators considered, the Core-only analysis \( (\hat{D} = \hat{N}/A) \) produced the most precise density estimate and was congruent with both SECR model estimates. We assumed animals with a mean capture distance >zero from the sampling area edge, which on average were captured 86% of the time at sampling stations away from the edge, were completely contained within the sampling area. Without tracking Core animals to account for the true proportion of time Core animals spend on and off the sampling area (Garshelis 1992; White and Shenk 2001), we cannot validate this assumption. Also, by assuming area is known exactly, we deflate the density variance by neglecting to account for uncertainty, leading to potentially erroneous confidence in our estimate.

Although our comparison of density estimation methods cannot evaluate estimator performance, as we do not know the true density of F. fossana, our comparisons highlight important strengths and weaknesses of estimation procedures that will be of use to practitioners attempting to reduce bias. Determining the correct area of a sampled population to ameliorate the edge effect is the limiting factor in producing robust estimates of density in the C-R framework. We agree with Obbard et al. (2010) and Gardner et al. (2010b) that SECR models are preferable to either traditional ad hoc boundary strip methods or Core-only analyses to estimate density. The SECR models incorporate the very real likelihood that the sampling layout has an effect on the detection process and area estimation (Boulanger et al. 2004b; Dillon and Kelly 2007). We encourage other carnivore C-R studies to employ SECR models, as they (1) have a sound theoretical and statistical framework, (2) free researchers from making subjective decisions on how to calculate the ETA, thus making density estimates across studies more comparable, (3) relax the geographic closure assumption and account for the edge effect, and (4) provide conservation agencies with important population information in a single answer from one underlying methodology, rather than a range of answers from multiple methodologies.