Introduction

Teleost fishes display a great diversity of behavioural mating systems, and these mating systems are shaped by local distributions and densities of potential breeding mates, essential resources, and the extent to which they can be economically defended (Emlen and Oring 1977; Petersen and Warner 2002; Kokko and Rankin 2006). For example, polygynous mating systems have been described for coral reef fishes when females are naturally site-attached for better access to food, shelter, or spawning sites and hence are defendable by males (Moyer and Nakazono 1978; Robertson and Warner 1978), or when males defend a territory that encompasses essential resources to females (Fricke 1980a; Neudecker and Lobel 1982). In fishes that care for their young, mating systems are also shaped by the relative amount of parental investment provided by each sex (Trivers 1972; Wittenberger and Tilson 1980). The mating system is strongly related to the extent and type of parental care, because care-giving may interfere with further opportunities for mating.

On the other hand, monogamy is thought to occur when opportunities for polygyny are constrained, especially by biparental egg care, intrasexual aggression, territory defence, or mate guarding (Wittenberger and Tilson 1980; Fricke 1986; Kokita 2002; Morley and Balshine 2002; Taylor et al. 2003). Additionally, some habitat specialists show inflexibility in their mating systems and are obligate monogamous species even under ecological conditions (e.g. population density, habitat availability) that would favour polygyny (Fricke 1979; Kuwamura et al. 1993; Hernaman and Munday 2007).

Adaptive modifications of mating systems and alternative mating strategies with respect to environmental, demographic, and ecological variables are not uncommon among fishes (Fricke 1980a; Donaldson 1989; Kawase and Nakazono 1996; Wong et al. 2005; Mobley and Jones 2009). Some cichlids demonstrate flexibility in both their reproductive behaviour and parental behaviour, and monogamous males show facultative mate desertion and increased fitness by means of polygamy (Barlow 1974; Townshend and Wootton 1985). One species of monacanthid, Oxymonacanthus longirostris, experiences sexual conflict over the optimal mating system, whereby the fitness of females increases when forming monogamous pair-bonds, while males benefit from polygyny (Kokita and Nakazono 1998). These fishes are all relatively small in body size, and the coral-dwelling species among them are strongly site-attached and depend on the complex structure provided by coral colonies for shelter (Munday and Jones 1998). As a consequence, site-attached habitat specialists may exhibit flexibility in their mating systems as a response to these ecological constraints.

Members of the family Balistidae (triggerfishes) possess a great diversity of reproductive strategies, and intraspecific variation of mating systems under different ecological conditions has been reported for these marine fishes (see Kawase 2003).

They are commonly associated with coral reefs, and are known to be polygynous with either maternal or biparental egg care, and with males being territorial either year-round or during the spawning season (Kawase 2003; Brandl and Bellwood 2014). The mating system of Sufflamen chrysopterum is size dependent in a high-density population with smaller males being monogamous and larger ones being polygynous (Ishihara and Kuwamura 1996), whereas low population densities result in facultative monogamy in S. chrysopterum and Pseudobalistes fuscus (Fricke 1980b; Kawase and Nakazono 1993). Understanding the adaptive significance of a mating system of a species is important as it affects mating success, which is often higher for males in polygynous species (Trivers 1972; Clutton-Brock 1989).

The ecology, reproductive behaviour, and social systems of S. chrysopterum and Rhinecanthus aculeatus appear to be very similar, and both species coexist and share similar resources (Kuwamura 1991; Ishihara and Kuwamura 1996; Kuwamura 1997). Within Balistidae, these two species have a sister group relationship and are closely related (Holcroft 2005). Both are sexually dimorphic with males being larger than the females (Seki et al. 2009; Künzli and Tachihara 2012). Territories are maintained against consexuals and are defended by border fights year-round. A haremic male territory encompasses the individual territories of up to 3 females in S. chrysopterum and up to 5 females in R. aculeatus. Females and males forage widely within their territory and spawn at various sites, and mating occurs presumably between cohabitants.

In both species, parental roles are divided unequally between the sexes and egg care is only provided by the female. Females spawn demersal eggs on the substrate within their territories around sunrise. No predation has been observed on the egg masses, and a highly aggressive egg defence of females appears to be effective enough for the survival of the embryos until hatching at dawn of the same day (Kawase 2003). Consequently, the males are completely freed from parental duties, which enables them to mate with multiple females of their harem on the same spawning day (Ishihara and Kuwamura 1996; Kuwamura 1997). On the basis of theoretical and empirical considerations, we therefore expect variation in mating patterns of R. aculeatus similar to that of S. chrysopterum, such as the occasional formation of monogamous pair territories under specific ecological conditions. Such closely related species provide potential for gaining insights via comparative studies on their mating systems and the ecological conditions under which different mating systems are expressed (Hernaman and Munday 2007).

In this study, we investigated the social organization, patterns of territorial compositions, and mating status of female and male R. aculeatus over time in their natural habitat. We attempt to identify a demographic parameter that explains the observed increase in monogamy over time such as adult sex ratio. Ecological and species-specific life history aspects are considered to understand the observed plasticity in mating type.

Based on our field observation of intraspecific behaviour and changes in territorial occupancy, we further assess whether the mode of mate acquisition in R. aculeatus is that of resource or female defence polygyny (Emlen and Oring 1977). If resource defence polygyny occurs, males should be associated with limiting resources used by females (e.g. food, shelter, breeding sites), and the males would be expected to fight for critical resources or territories regardless of female presence. If female defence polygyny prevails, males should be attached to females rather than resources. Under such circumstances, males are expected to fight only in the presence of females and should relocate in their absence.

Previous studies about balistids either aimed to determine whether resource or female defence polygyny occurred in a population (e.g. Seki et al. 2009), while other studies mainly focused on the reproductive behaviour and mating system (e.g. Ishihara and Kuwamura 1996). This paper presents a combination of both types of studies and provides new insights in the territoriality and social and mating system of a balistid and possible underlying selective forces.

Overall, our study characterizes the factor(s) responsible for polygyny in a demersal triggerfish species and the observed variation in mating systems of a local population with respect to ecological and demographic properties.

Materials and methods

Study species

Rhinecanthus aculeatus is a fairly common and widespread triggerfish in the Indo-Pacific and is usually found in shallow lagoons and flat reefs (Froese and Pauly 2010). This large balistid reaches maximum ages of 13.5 years (males) and 9.5 years (females) and maximum sizes of 209 mm (males) and 176 mm standard length (SL) (females; Künzli and Tachihara 2012). Inside large territories of males, females maintain smaller territories (ca. 10–15 m in diameter) against consexuals through aggressive defence (Kuwamura 1997). Females feed widely within their individual territory, and males patrol through all the territories of their monopolized females, thereby randomly picking food from the substratum. R. aculeatus is a diurnal fish and retreats into a shelter during the night time.

Territorial female and male R. aculeatus demonstrate high site fidelity and maintain their territories long term (e.g. over 8 years; Kuwamura 1997). These territories are actively defended during both the reproductive season (July–September, Kuwamura 1997; June–August, this study) and during the non-reproductive season, and a harem will not break-up once the spawning activity ceases.

Female and male residents are further aggressive towards unmated floaters (bachelors) when encountered. For example, in Sesoko Island, floaters aggregated in shallow areas in proximity to adult territories (Fig. 1). These floaters were observed during the whole study period and they showed no territorial or reproductive behaviour (mean abundance 17.8 ± 0.7, range 10–31 individuals; mean SL 111.2 ± 0.7 mm, range 40–200 mm). Floaters from such “assemblages” are suspected to undertake an ontogenetic habitat shift into the adult habitat to adapt a territorial lifestyle once they reach maturity (this study).

Fig. 1
figure 1

Territorial composition of female (grey area, number and letter) and male (dashed line, number only) R. aculeatus in a March 2013 and b November 2013. Note that solitary females had no corresponding monopolizing mate. Female 29a was only spawning and had no territory [denoted by X, in (a)]. The transition of the nearshore sandy area and the flat reef is indicated by a solid line

Aggression and harassment of territorial adult R. aculeatus towards other balistids and non-balistids has been reported (Kuwamura 1991). In this study, S. chrysopterum had overlapping territories with R. aculeatus at the reef edge and the former was harassed a few times by female and male R. aculeatus. Further, R. aculeatus showed aggressive behaviour towards Rhinecanthus verrucosus, which mingled with floaters in the assemblage nearshore. Male R. aculeatus with territories near the assemblage pursued after R. verrucosus (140–150 mm SL, presumably adults) when the latter approached the territory border of R. aculeatus. Other triggerfishes such as Balistoides viridescens, B. conspicillum, and Balistapus undulatus were rarely seen within the study area, and no interspecific aggression occurred between resident R. aculeatus and these transient balistids. Aggressive behaviour towards non-balistids such as Epinephelus merra, Parupeneus multifasciatus, Diodon holocanthus, and female Scarus schlegeli happened infrequently and only by female R. aculeatus (spawning days not included). A few times, other fish harassed R. aculeatus. Plagiotremus tapeinosoma, a parasitic fangblenny, and the related cleaner fish mimic, Aspidontus taeniatus, darted after passing R. aculeatus to bite off pieces of tissue. During such incidents, R. aculeatus responded by pursuing the attacker and repeatedly attempting to bite it.

A semi-lunar spawning cycle has been assumed for R. aculeatus. Females pair-spawn between 0 and 3 times within a cycle, which lasts for about 1 week around the full or new moon (Kuwamura 1997). Haremic males can mate with up to three females on a given spawning day. Females associated with the same harem tend to spawn during successive days within a cycle (our observations).

Demersal eggs are laid directly on the bottom on sand, on substrate such as calcified red algae (Jania sp., Digenea simplex), or on coral rubble. Spawning is restricted to the early morning around sunrise (ca. 0548–0634 hours), and only females care for eggs until sunset on the same day (ca. 12–14 h in total). Prior to the spawning, a territorial female starts to clean the presumable spawning site by removing sand or algae. The male closely follows the gravid female and starts to nuzzle the belly and caudal peduncle of the female. The female then moves to the spawning site and thrusts her genital pore onto the substratum, while the male positions himself slightly behind the female, thereby touching her abdomen. The male shivers for ca. 2–3 s as both fish release their gametes in a single clutch (our observation). No sperm cloud is visible. Courtship and spawning is accomplished within ca. 7–27 min (Kuwamura 1997). After spawning, the female instantly starts fanning the eggs while the male deserts to feed in the surrounding area, or to visit another female (Kuwamura 1997).

During the courtship and mating behaviour of 11 pairs in total, Kuwamura (1997) never observed other approaching males in his study, which is in accordance with our observations (N = 1). In addition, Ishihara and Kuwamura (1996) observed a total of six spawning acts of S. chrysopterum and they similarly did not notice any approaching males. Sneaking was never observed.

Egg masses of R. aculeatus are ca. 5–15 cm in diameter (Kuwamura 1997), and fertilization of such relatively small egg masses by a small sneaking male might be difficult. Further, according to our unpublished reproductive data, 50 % of the males reach maturity at a standard length of 148 mm, while the smallest mature male was 139 mm (N = 160). This strongly suggests that most floater males—presumably candidates for sneaking—are in fact still immature and that maturity in males is associated with a transition to a territorial lifestyle.

Alternative mating behaviour such as sneaking fertilization in R. aculeatus or in S. chrysopterum is therefore considered unlikely. For this reason, we assumed that the mating partner of monopolized females always corresponded to the current monopolizing male, and we did not attempt to validate mating partners of territorial females. In the case of non-monopolized (solitary) females, the identity of the mating partner could not be determined.

Study sites

Field work was carried out at Sesoko Station of the Tropical Biosphere Research Center (TBRC) on Sesoko Island (26°38′N, 127°51′E), Okinawa, southern Japan, between October 2012 and November 2013. The study site (70 × 90 m, ca. 6300 m2) was established on an emerging fringing reef just inside the reef edge (Fig. 1). All observations were conducted by skin diving at depths of 1.0–3.1 m. The nearshore intertidal area was composed of solely rocky substrate with turf algae. A primarily sandy/silty bottom interspersed with live hard corals (mainly Acropora, Faviidae, and massive Porites) and leather corals (Lobophytum) prevailed in a stretch in ca. 30–50 m from the shore line (Fig. 1). The emergent flat reef was characterized by a higher coral density with various hermatypic corals (encrusting, digitate, tabular and massive) and leather corals. Part of the study area (ca. 730 m2) was highly covered with low stands of digitate and branching corals (mainly Acropora) interspersed with tabular Acropora, and no territories of R. aculeatus or S. chrysopterum were established in this area (Fig. 1).

Territorial composition and mating status

The study site was visited during the daytime between 0900 and 1900 at least twice monthly (total N = 59, ca. 180 h). Between April and August 2013, the study site was visited more frequently to obtain data regarding spawning activity. Daily observations were conducted between July 27 and August 22, 2013. To identify all territorial adults, every fish was photographed and individuals were differentiated according to the variation of their lateral bar and caudal peduncle pattern.

The standard length (SL) of each fish was estimated to the nearest centimetre and confirmed by catching randomly selected adults (females: N = 9, males: N = 4) using a trammel net (10 × 10 mm mesh size). Captured fish were measured on a scaled plastic board and immediately released. Each fish was handled as little as possible to reduce stress. Males could primarily be distinguished from females by their larger sizes. In addition, the sex of the females was confirmed during the spawning season when they performed egg care.

The territory of an adult R. aculeatus was defined as the home range area in which the adult spent its typical daily activity (e.g. feeding, swimming, and patrolling). Each fish was initially observed for ca. 10–15 min on 2–3 occasions, and their swimming route was marked on a scaled underwater map of the study area, which was previously created based on a satellite image (Google Earth, version 7; 26°38′N, 127°51′E). Territory boundaries were assessed using data collected on range of movements as well as aggressive interactions towards neighbouring conspecifics. Range of movement was considered to be a good indicator of the territory boundaries because the range of neighbouring fish did not overlap. If a fish was recorded outside of its territory, the position was marked on the map and the fish was observed for ca. 5–10 min to document its swimming route and behaviour. Resident males (N = 14) and females (N = 12) were occasionally recorded outside of their territory for visiting a labroid cleaning station, feeding in shallow areas or for roaming through the reef. “Excursing” fish did not behave aggressively towards other territorial adults, but were often attacked by residents. Thus, the positions of excursing fish were not considered for the determination of their territory.

By March 2013, all territorial fish within the study area were identified and their territories were compiled on the map. Individual territorial areas were measured using a polygon (Google Earth Pro, version 7). According to the range of the male territory, each fish was assigned one of the following mating statuses: polygynous (group pairing with one male and multi-females) or monogamous (pairing of one male with one female). If a male or a female maintained a territory without a mate, the fish was termed as solitary.

To keep track of mate pairing and territorial occupancy, each male was numbered and each initially monopolized female was labelled by the same number as the male with a consecutive alphabetic letter (Fig. 1; Table 1). Solitary females were assigned the letter “a” as well as an individual identification number.

Table 1 Summary of observed territorial female (N = 45) and male (N = 22) R. aculeatus from March to November 2013 in the study area of Sesoko Island

During visits following March 2013, each fish was searched and localized in order to confirm or reassess its territory and mating status. If a new female joined a resident pair or harem, or if newcomer males took over the territories of females, a distinct identification number for the newcomers was used to demonstrate that the territorial composition had been altered (Fig. 1b). If the territorial compositions and pairing changed, the mating status of affected males and females changed accordingly (Table 1).

If a fish relocated from one territory into another one, data of the original territory and the newly acquired one were used for statistical analyses. If a fish stayed in its original territory but reduced or enlarged it due to an intruder or disappearance of a neighbour, a mean was calculated between the original area and the reduced or enlarged area. Each of the observed territorial compositions (i.e. harem, monogamous pair, and solitary territory) was treated as one territorial “unit” in order to compare these units between months and to quantify their average proportion during the study period.

Intraspecific interactions

All aggressive intraspecific interactions between territorial R. aculeatus were recorded whenever encountered. Attacks were categorized as (1) chasing (short rushes <2 m), (2) pursuing (long rushes >2 m), and (3) biting (audible teeth clenching). Both the aggressor and the recipient of the aggression were documented and identified if possible. The places where attacks occurred were recorded. Attacks of egg guarding females during spawning days were not considered. Aggressive bouts during escalating conflicts were counted separately during 30 min observations.

Mating success

During every visit, we checked whether females were performing parental egg care. Each spawning event of females was recorded and added to determine their overall mating success at a given mating status. If the mating status of a female changed during the reproductive period, the consecutive spawnings were accounted accordingly. Male mating success was determined as the total of spawning events of their monopolized female(s) at a given mating status.

Data analyses

All statistics were analysed first for normality and equal variances. If these assumptions were not met, data were square-root transformed. If transformation of data did not satisfied a priori assumptions, appropriate nonparametric tests were applied. Two-tailed tests were used for all data. Statistical tests are indicated throughout the text. Means are given ± standard error (SE).

Results

Female territoriality

Individual territories of female R. aculeatus were established in the sandy habitat of the subtidal zone nearshore and in the emerging flat reef until the reef edge (Fig. 1). Female territories were highly contiguous with each other and were often bordered by landmarks (e.g. reef edge, concrete block, transition of sandy/flat reef habitat, transition of high coral cover/low coral cover). Every territory contained a shelter opportunity in the form of prominent Porites heads, tabular Acropora, cracks in the seafloor or spaces between rocky boulders. During the study, we recorded a total of 45 territorial females. They ranged in length between 115 and 170 mm (female SL: 150.1 ± 2.1 mm, N = 45; Table 1). Females were either part of a harem or formed monogamous pairs. In addition, several females maintained solitary territories with no mate living in their territory (Fig. 1; Table 1). Four females were permanently solitary and were never observed to be monopolized by a male, while others (N = 5) became temporarily solitary due to the emigration of their mate from the territory. Permanent solitary (N = 3) and even temporarily solitary (N = 1) females were reproductively active and were caring for eggs, but their mating partner could not be evaluated (Table 1). There were no differences in female body size and mating status even though polygynous (SL: 151.6 ± 2.3 mm, N = 39) and solitary females (SL: 151.6 ± 2.2 mm, N = 9) were on average larger than monogamous females (SL: 145.7 ± 2.8 mm, N = 14, pooled for females with varying mating status; ANOVA: F 1,61 = 1.01, P > 0.05).

With one exception, females were always smaller than their monopolizing male (Wilcoxon signed-rank test: T = 1219, N = 49 pairs, P < 0.0001), and pairing between females and males was size-assortative (Spearman’s rank correlation: rs = 0.491, N = 49 pairs, P < 0.001; Fig. 2).

Fig. 2
figure 2

Pairing sizes of territorial R. aculeatus males and their monopolized females. The solid line is the 1:1 line. Some combinations of length and number of monopolized females were observed several times and are indicated by the size of the plotted point

The mean size of a female territory was 75.9 ± 11.0 m2 (range 13.0–350.0 m2, N = 46, including relocating female; Table 1). Female body size was positively related to territory area (Pearson correlation: r 46 = 0.435, P < 0.01, including relocating female).

In total, eighty social interactions between territorial R. aculeatus were observed during ca. 180 h of observations. Females accounted for 68.8 % of all these interactions (N = 55 bouts) and were behaviourally the more aggressive sex (Chi-square test: χ 2 = 11.25, df = 1, P < 0.001; Fig. 3). No significant seasonal trend was detected (2.2 ± 1.7 bouts/month, range 0–9 bouts/month; Kruskal–Wallis: H = 14.1, P > 0.05), and territory defence behaviour was not related to reproductive season. Chasing was the predominant behaviour displayed by females (N = 41 bouts, 74.6 %), compared to more aggressive behaviour such as long pursuits (N = 12 bouts, 21.8 %) or bites (N = 2 bouts, 3.6 %; Fig. 3). Female aggressive behaviour was biased towards males (N = 31 bouts, 56.4 %), but it was not significant (Chi-square test: χ 2 = 0.89, df = 1, P > 0.05; Fig. 3). Aggressive attacks towards males were provoked when a roaming foreigner or neighbouring territorial male attempted to pass through a female’s territory. No female was seen to attack her monopolizing male. Aggression among females occurred mostly between neighbouring females belonging to another social group (e.g. border disputes).

Fig. 3
figure 3

Intra- and intersexual aggressions (chasing: square with dots, pursuing: grey filled square, biting: black filled square) of territorial R. aculeatus in Sesoko Island

In total, ten females emigrated (10/45; 22.2 %) from their territory. Nine of them disappeared permanently from the study area, while one female relocated into another territory (Fig. 5a). Among the emigrants, seven females (including the relocating female) were monopolized by a haremic male during the time of the territory abandonment. One female emigrant never had a mate and was solitary when she disappeared, while two females disappeared within 1 week and 4 months after becoming solitary. When other females lost their mate (N = 4), they did not relocate or emigrate from their territory and were re-monopolized within less than 4 weeks by a newcomer male (Table 1).

Only three of the ten vacated female territories were re-occupied by newcomer females, while the other seven vacated territories remained unoccupied until the end of the study (Fig. 5a). Once a female abandoned her territory, no spatial changes of neighbouring female territories occurred; only one female expanded her existing territory to include the area used by the emigrated female.

Females immigrated either into vacated territories (N = 3) as already mentioned or into unoccupied space (N = 6), which accounted for totally nine individuals (9/45; 20.0 %, including relocating female). Almost all females were newcomers and immigrated from outside of the study area, including the later relocating female (Table 1). The origin of only one female (26a) was known; she was a former floater (130 mm) from the assemblage and shifted into a vacated territory in July 2013. This female, however, was not seen to care for eggs after becoming territorial (Table 1).

Immigrant females into territories thereby joined monogamous pairs (N = 4), were monopolized by newcomer or relocating males within less than three weeks (N = 3), or established a solitary territory and remained un-monopolized (N = 2; Table 1). Males that gained additional females enlarged their territory accordingly to cover the new female’s territory entirely. There were no significant differences between the body sizes of emigrant females from territories (SL: 138.0 ± 5.4 mm, N = 10) and immigrant females into territories (SL: 145.0 ± 3.8 mm, N = 9; ANOVA: F 1,18 = 1.06, P > 0.05; Fig. 5a).

The overall migration patterns of fish (see next section for males) imposed changes in the mating status in 31.1 % (N = 14) of the females (Tables 1, 2). Changes in mating status were regarded as positive when a fish became part of a larger social group (e.g. from single to monogamy/polygamy, from monogamy to polygamy) or negative when the group size was reduced (e.g. from polygamy to monogamy/single). Some fish changed their mating status during the survey but returned to their initial mating status at the end of the observation period (e.g. from monogamy to polygamy back to monogamy). Overall, females similarly changed their mating status in a positive (28.6 %, N = 4) or negative (35.7 %, N = 5) way or returned to their initial mating status (35.7 %, N = 5; Table 2).

Table 2 Number of female and male R. aculeatus with invariant and variant mating status between March and November 2013

On the other hand, females with an invariant mating status (i.e. which were not affected by migration events; 68.9 %, N = 31; Table 2) stayed in their original territorial composition and accounted for a total of six harems, three monogamous and four solitary territories.

Aggressive females with egg-caring behaviour were first recorded on 6 June 2013. Previously during May 2013, many females had swollen bellies and appeared highly gravid but were not performing egg care. The last spawning was observed on 25 August 2013. In total, 30 females (66.7 %) were involved in reproduction, while 15 females (33.3 %) were not seen to care for eggs during the observations days (Table 1).

Females always spawned within the range of their territory and at varying sites. On 6 June 2013, an unknown and apparently non-territorial female (29a) aggressively guarded eggs at a small unoccupied reef patch (Fig. 1a; Table 1). No nearby territorial or foreign male approached her. The next day, this “single” female had disappeared and was not seen again inside or outside of the study area.

In terms of reproductive success, female body size was significantly related to the number of matings (Pearson correlation: r 46 = 0.350, P < 0.01; including one non-territorial female). A comparison of the reproductive success a female realized with her monopolizing male showed that the relationship between number of matings and male body size was positive and that pairing with a larger males resulted in higher number of matings in females (Pearson correlation: r 49 = 0.352, P < 0.01; pooled for females monopolized by different males). The mating status by itself, however, did not affect female mating success (ANOVA: F 2,60 = 0.39, P > 0.05, pooled for females with varying mating statuses including one non-territorial female), and there were no differences in the number of matings between females from polygynous groups (1.2 ± 0.2 matings, range 0–5, N = 38), monogamous females (0.9 ± 0.4 matings, range 0–5, N = 15), and solitary females (1.1 ± 0.3 matings, range 0–3, N = 10; Fig. 6).

Female defence polygyny by territorial males

The males defended a territory that corresponded to the territories of all their monopolized females (Fig. 1). In total, 22 territorial males were observed during the study and they ranged in length between 150 and 210 mm (male SL: 177.5 ± 3.8 mm, N = 22; Table 1), and males were on average 27 mm larger than females. This sexual size dimorphism among territorials was significant (ANOVA: F 1,66 = 47.68, P < 0.0001).

Territorial males were monopolizing between one to four females or were temporarily solitary with no mate in their territory (1.7 ± 0.2 females per male, N = 31 pooled for males monopolizing different or no females; Figs. 1, 4; Table 1).

Fig. 4
figure 4

Number of monopolized females as a function of male body size (N = 31, including males monopolizing different females and solitary males). Some combinations of length and number of monopolized females were observed several times and are indicated by the size of the plotted point. Regression line is shown: r = 0.38, P < 0.001. The dotted line marks the threshold size for the ability of polygyny in males

Male body size and the number of monopolized females were significantly related (Pearson correlation: r 31 = 0.614, P < 0.0001), and mean number of monopolized females significantly differed between males (ANOVA: F 7,23 = 4.47, P < 0.05; Fig. 4). Males with body sizes between 150 and 160 mm were exclusively monogamous (or solitary), while males >190 mm were exclusively haremic (Fig. 4). Polygynous males were significantly larger in body size than monogamous males (ANOVA with Bonferroni post hoc analysis: F 2,28 = 7.31, P < 0.01; pooled for males with varying mating status).

The mean size of a male territory was 180.2 ± 33.5 m2 (range 16.9–676.0 m2, N = 26, including relocating males; Table 1), and similar to females, larger males had larger territories (Pearson correlation: r 26 = 0.771, P < 0.0001).

Males accounted for 31.2 % (N = 25 bouts) of all observed aggressive behaviour (Fig. 3). The differences in proportions of displayed behaviour between the sexes were not significant (Chi-square test: χ 2 = 0.95, df = 2, P > 0.05), and chasing was predominant in both females and males (Fig. 3). Males among each other were the least aggressive group (N = 9 bouts, 36.0 %; Fig. 3). Aggressive attacks occurred mainly between neighbour males during border disputes and less towards roaming males. Males also attacked females (N = 16 bouts, 64.0 %; Fig. 3). In these cases, the female recipients were roaming, neighbours from a different social group, or the territorial male harassed his own female.

In total, eight males emigrated from their territory. Four of these males disappeared from the study area, while four males relocated into other territories. This resulted in an emigration rate of 36.4 % (8/22; Fig. 5b). Two polygynous males disappeared after they lost all females of their harem, while two males disappeared even though their monopolized females were still present (Table 1). The situation was more complex among the four relocating males (Table 1). Two males experienced a complete break-up of their harem. Male 1 relocated instantly and monopolized a nearby newcomer female, while male 6 stayed solitarily in his territory for 5 weeks. Then in mid-July 2013, he was frequently observed to intrude the territory of haremic male 18 by swimming and feeding within his territory. As a consequence, resident male 18 initiated prolonged attacks (N = 54 chases; 30 min observation) on the intruder male, while male 6 counteracted with occasional chases (N = 20 bouts; 30 min observation). At the beginning of August 2013, the conflict ended because male 18 abandoned his territory and three females. Male 6 instantly overtook one female and mated with her thereafter multiple times, while the other two females became solitary. Displaced male 18 was observed to mingle with floaters in the assemblage or spotted to roam within the reef. In November 2013, he finally relocated into a vacated harem and established occupancy of two female territories and expanded the territory to integrate a nearby solitary female.

Fig. 5
figure 5

Body size of emigrating and immigrating a female and b male R. aculeatus. Lines indicate vacated territories, which were re-occupied by newcomer or relocating fish—note that the territories of two haremic males (190, 200 mm) were each re-occupied by two immigrant males. The numbers within the circles indicate the number of individuals of the same size

The fourth male to relocate was male 23—a former floater from the assemblage. He invaded first the territory of pair 19 in June 2013 and resided in close proximity to the resident female 19a (see also immigrant males). The situation escalated, and the intruding male 23 was severely attacked by pair 19 (female: N = 23 bouts, male: N = 20 bouts; 30 min observation). Male 23 occasionally counteracted by chases or bites (N = 6 bouts; 30 min observation). As a result, the resident female and the intruder male had fleshy bite wounds in the head region and on the lateral side. The conflict appeared to be settled after ca. 2 weeks and male 23 annexed a part of the resident territory but failed to monopolize the female. No further aggression occurred thereafter. Then, at the end of August 2013, male 23 relocated into the territory of a nearby female, who became available due to the displacement of male 18. After male 23 abandoned his solitary territory, the vacated space was re-reclaimed by the original owner.

On the other hand, a total of 11 males immigrated into territories to overtake females (11/22; 20.0 %, including relocating males; Fig. 5b). Seven newcomer males either immigrated from outside of the study area and were unknown (N = 2), or intruded the adult area directly from the floater assemblage between April and July 2013 (N = 5, Table 1). The other four immigrant males (including male 23) were the relocating males as previously described.

Immigrant males took over solitary or recently vacated females, or females that had themselves immigrated into vacated territories, while male 23 failed first to monopolize a mate and became solitary.

Immigrant males that monopolized vacant females were significantly smaller than the originally monopolizing males (Wilcoxon signed-rank, T = 36.0, N = 9, P < 0.05; Fig. 5b). Further, all seven newcomer males were first monogamous (or solitary).

A comparison between males and females showed that both sexes emigrated similarly often from their territory (males: 8/22, females: 10/45; Chi-square test: χ 2 = 1.50, df = 1, P > 0.05), but the immigration rate of males was significantly higher than that of the females (males: 11/22, females: 9/45; Chi-square test: χ 2 = 6.35, df = 1, P < 0.05), which affected the adult sex ratio (see next section).

A total of 68.2 % (N = 15) of the residential males remained in their established pairings and had an invariant mating status, while 31.8 % (N = 7) of the males experienced changes in their mating status (Table 2). Males thereby changed their mating status more often in a positive way (57.1 %, N = 4) than in a negative (14.3 %, N = 1) or returned to their initial mating status (28.6 %, N = 2; Table 2).

In terms of reproduction, male body size was significantly related to the number of matings (Pearson correlation: r 22 = 0.804, P < 0.0001) and larger males achieved higher reproductive success. Moreover, the mating status had a significant effect on the number of matings (ANOVA: F 1,27 = 7.77, P < 0.01, pooled for males monopolizing different females), and polygynous males (3.2 ± 0.7 matings, range 0–8, N = 15) mated more often compared to monogamous males (0.9 ± 0.4 matings, range 0–5, N = 14; Fig. 6). The solitary category (N = 2) was omitted in these analyses due to low sample sizes and zero matings.

Fig. 6
figure 6

Number (mean ± SE) of matings as a function of sex and mating type in female (grey) and male (black) R. aculeatus. *Significant differences at α = 0.05. N was pooled for individuals with varying mating statuses

Territorial compositions and adult sex ratio (ASR)

In March 2013, the territorial situation comprised 10 harems, 5 monogamous pair territories, and 4 solitary territories (N = 19 territory units; Figs. 1a, 7) accounting for a total of 37 females and 15 males. At the end of the survey in November 2013, the territory situation had noticeably changed and was then composed of 9 harems, 9 monogamous pair, and 3 solitary territories (N = 21 territory units; Figs. 1b, 7), which included 36 females and 18 males.

Fig. 7
figure 7

Seasonal changes of territorial composition in the study area composed of harems (dark grey), monogamous pair (light grey), and solitary territories (black) of R. aculeatus. The monthly adult sex ratio is indicated (ASR: number of males/the number of males and females)

Haremic groups (range 9–13) were always more abundant as a territory unit during each month than monogamous pair (range 2–9) or solitary territories (range 3–7), except between September and November 2013, when monogamous pair territories and harems were equal in number (N = 9 territory units each; Fig. 7). On average, monogamous pair territories accounted for 30.2 % of all the territorial compositions (6.4 ± 0.8 territory units) between March and November 2013, which was significantly less abundant than the overall contribution of harems (48.9 %; 10.1 ± 0.5 territory units), but not significantly more abundant than the proportion of solitary territories (20.9 %; 4.4 ± 0.4 territory units; Dunn’s multiple comparison, P < 0.05; Fig. 7).

Because haremic groups were on average the dominant territory unit, females significantly outnumbered males in every month and the ASR was strongly female-biased (binomial test, each month: P < 0.05; Fig. 7). Mean monthly number of territorial females was 37.3 ± 0.5 (range 36–40 females) and that of males 17.1 ± 0.4 (range 15–18). As a result of differences in the emigration and immigration rate between the sexes, the ASR was skewed between May and July 2013 and became less female-biased after July 2013 (Fig. 7). This overall increase in ASR positively correlated with the monthly number of monogamous pair territories (Spearman’s rank correlation: rs = 0.765, N = 9, P < 0.05; Fig. 7).

Discussion

Competition for space and females

Females appeared to be naturally site-attached, and their movements were not limited by male aggression. They actively defended a territory by excluding competitors through aggressive behaviour and did not tolerate other conspecifics asides from a monopolizing male within the range of their territory. We did not specifically evaluate non-aggressive behaviour, but females and males often erected their spine together with the pelvic complex in order to appear bigger or showed temporary dichromatism towards conspecifics. Body colour change included a complete fading of the normally blackish shading between the blue iridescent lines running downwards from the front through the eyes, and the parts of the skin—these areas which are usually uncoloured—became light emerald green. Rapid colour change in fish has been associated with communication and sexual display (Nilsson Sköld et al. 2013) and may play an important role in R. aculeatus in avoidance of rushing behaviour, which is energetically more costly.

The competition for space among females resulted in highly contiguous female territories, especially in the flat reef, where one female expanded her territory after a neighbouring female had disappeared. In contrast, several vacated female territories located in the sandy habitat remained un-occupied by newcomer females and neighbouring females did not expand their territory into the free space once it became available, suggesting that females do not randomly establish territories but rather select and compete for territories in preferred reef areas. A comparison of female body size with territory area further confirmed that larger females occupied larger territories. This indicates that larger females may have priority over smaller females and that competition may force smaller females to occupy less preferred territories (e.g. the homogenous sandy habitat).

The spatial distribution of territories among females is often explained by the competition for limited food and/or limited shelter resources (Baird and Liley 1989; Hourigan 1989; Matsumoto and Kohda 2004). In particular, shelters are a vital and indispensable resource for demersal triggerfishes, as they serve as refuges and sleeping places.

All females reproduced within the range of their territory and suitable spawning sites were not limited as females changed their spawning site within the territory. This species does not migrate between feeding and off-shore spawning sites as has been observed in other coral reef fishes that spawn pelagic eggs (Johannes 1978; Yabuta 1997). Thus, the function of female territoriality in R. aculeatus is multi-purpose and ensures access to several resources such as food, breeding-related locations, and shelter, and the distribution of these resources may strongly affect the distribution of territories.

Several females were permanently solitary and successfully reproduced despite the lack of a monopolizing mate. These solitary females did not relocate into a male territory, similar to most females when their mate disappeared. This is indicating that females defend territories largely irrespective of males.

However, two females disappeared from their territory after losing their mate. One of them was highly gravid when her monopolizing male abandoned the harem due to a conflict with an intruder. After becoming single, the female showed aggressive behaviour for three consecutive days and did not care for eggs. On the fourth day (1 day before the new moon), she spawned and exhibited parental egg care even though no newcomer males were observed patrolling within her territory. Thus, female R. aculeatus appear to be capable of retaining their eggs and delaying spawning if a male is not immediately available for fertilization. The ability to retain eggs may be especially relevant for solitary females (who may need to wait for available males) or for monopolized females (who must wait when the male is courting and mating other females within the harem).

The above-mentioned female nevertheless disappeared 2 days after she cared for eggs—still appearing gravid.

Contrary to several males, only one female was observed to relocate. This female abandoned her territory in the sandy habitat and her monopolizing male (170 mm) during the beginning of the spawning season in June 2013, and established a new territory nearby a larger male (190 mm) in the flat reef. The resident male immediately took over the additional female and reproduction was initiated shortly after. No reproduction was observed with the originally monopolizing male. Qualitative aspects of food resources, spawning sites, or male quality (e.g. fertility) may have influenced this relocation of a female into another territory.

In terms of reproduction, female body size was significantly related to the number of matings. Larger females often have higher fecundity (Gross and Sargent 1985) and may be in need of a larger and/or qualitatively better territory to sustain the energy need for egg production and parental egg care. The observed assortative pairing further suggests that the largest males were preferred by the largest females, likely to ensure the fertilization of larger egg clutches.

Site attachment and defence of territories by females in relation to home sites such as shelters, spawning sites, and/or food has also been observed in other polygynous balistids (Fricke 1980b; Thresher 1984; Seki et al. 2009), related tetraodontids (Kobayashi 1986; Gladstone 1987; Sikkel 1990), and in other fishes with a haremic mating system (Baird and Liley 1989; Yabuta and Kawashima 1997; Carvalho et al. 2003; Kadota et al. 2011). The relative importance of these factors in the selection of a territory by a female R. aculeatus should be examined further.

Female R. aculeatus were not aggregating but showed high site fidelity and occurred at a density that allowed the males to defend them economically. A male territory was subdivided by the territories of all the females he could monopolize. Importantly, male size was positively correlated to the number of monopolized females—suggesting strong male–male competition. Consequently, larger males had the larger territories. The maximum number of females, which can still be economically monopolized by a male R. aculeatus apparently, lies between 4 and 5 females (Kuwamura 1991, 1997; this study). Our data suggest that the harem size remains largely unaffected by time constraints on the spawning act because females can retain eggs. Therefore, it appears that monopolization of additional mates increases temporal and energetic costs of territory defence. Defence of more than 5 females might offset the reproduction reward of spawning with more females.

If resident males lost their mates—either because their females disappeared or because they were displaced by an intruder—the males relocated to assure access to females. Contrary to the females, males never attempted to establish a territory without a mate (i.e. solitary).

Male–male competition for territories was thus related to mate acquisition (“female defence polygyny”) and not to territory defence (Emlen and Oring 1977), and males abandoned their territories once their females disappeared or relocated. Males occasionally harassed their monopolized females, which indicated the role of intersexual aggressions in maintaining (or establishing) mating relationships in polygynous fish (Moyer 1984; Baird and Liley 1989). However, males did not assert strong behavioural pressures on females to remain within the male territory as it has been reported for ostraciids (Moyer 1984), and female R. aculeatus were limited in their movement during boarder disputes with other females rather than by their monopolizing male.

Despite infrequent aggression among territorial males, intrasexual competition became apparent when intruding males challenged resident males to claim a female. Under such circumstances, prolonged and intense aggressive interactions were carried out by the residents towards the intruder, which inflicted flesh wounds in two fish. A high level of aggression carries high energetic costs and risk of injuries. Triggerfishes are equipped with powerful jaws and sharp teeth (Matsuura 1979), which function to crush hard-shelled prey (Hiatt and Strasburg 1960; McClanahan and Shafir 1990).

It is assumed that if the potential and future benefit (e.g. access to females) of escalations is high, it pays a smaller-sized intruder to escalate. In one of the two observed conflicts, the intruder succeeded in claiming a female, while in the other conflict, the intruder annexed only a territory because the larger residential female took part in the territory defence, and thereby prevented a monopolization by the smaller intruder male. From this behaviour, it can be suggested that the monopolization of females is not entirely related to male–male competition but that females are able to reject less favoured males by active resistance, e.g. through severe bite attacks.

Mating system plasticity

The degree to which critical resources or mating opportunities of spatially aggregated females can economically be monopolized is thought to be one of the most important determinants of animal breeding systems (Emlen and Oring 1977). Sex differences in parental care, investment in gametes, and biases in the sex ratio can affect the relative number of sexually active males to receptive females and may result in increased variance in male fitness (Clutton-Brock 2007). In particular, when one sex is released from parental duties and when no additional costs are involved, the time investment for the defence of critical resources (i.e. mating partners) becomes economically defendable.

Rhinecanthus aculeatus has previously been recognized as a polygynous triggerfish with males mating with multiple females (Kuwamura 1991, 1997), fitting the criteria for such a specialized mating system as females taking exclusive care of the brood, which in turns allows the males to interact with other females for the purpose of additional mating.

In this study, however, we found plasticity in the mating system in a local population of R. aculeatus in its natural habitat and facultative monogamy was expressed. On average, monogamy accounted as high as 30 % of the territorial compositions and both sexes were further seen to maintain territories without an evident mating partner (on average 21 %). Notably, solitary females were reproductively as successful as monogamous females or females from harems. It is possible that these females mated randomly and with minimal choice of males in the population (i.e. promiscuity). For instance, promiscuous mating with a random partner must have occurred with the unknown female that appeared in an unoccupied reef patch solely for the purpose of spawning, without first establishing a territory. If solitary females, however, similarly select for larger males like monopolized females, then mate selection is operating and polygynandry is expected (Wootton and Smith 2014). More empirical evidence is needed to evaluate the identity of mating partners accepted by solitary females.

The overall stability of the territorial compositions in the study area was ambivalent even though the majority of females and males were from harems and stayed in their original pairings throughout the study. Changes in territory occupation and associated mating status occurred mainly between May and September 2013 and directly affected the ASR. The ASR is expected to influence sex roles and mating systems because the rarer sex in a population has more potential partners to mate with and benefits via sex differences in mortality, maturation rates, and movement patterns (Székely et al. 2014). The observed shift in ASR towards a less female-biased ratio corresponded with an increase in the number of monogamous pair territories. After relatively large males emigrated from their territory, they were replaced by smaller newcomer males with body sizes at or below the threshold size for the ability of polygyny (170 mm). These newcomer males sequestered one female, while one failed to do so and had to remain without a mate in his territory.

Fitted observed size at age data suggest an earliest age of ca. 5 years for polygamy (170 mm), while monogamy (150 mm) is expected to occur earlier at ca. 3 years (Künzli and Tachihara 2012). Because growth in males slows down at sizes exceeding 170 mm (e.g. 8 years at 180 mm), a relatively long time is required for males to reach appropriate sizes for the monopolization of large females. Thus, during a shortage of large and competitive males which can restrict mating opportunities in other males, some large females may reject smaller males and remain solitary, and small males may favour early maturation by sequestering one smaller female instead of focusing on physical growth to sequester multiple (and larger) females at a later time—which was observed in this study. Reproduction was, however, only observed in the largest newcomer male (170 mm), but might have occurred in the other newcomer males during non-observation days.

Intraspecific variations in mating systems among coral reef fishes (Donaldson 1989; Petersen 1990; Kokita and Nakazono 1998; Wong et al. 2005) are not uncommon and have been related to environmental and demographic conditions. Plasticity in the mating system of balistids has been found so far in two triggerfishes (Fricke 1980b; Kawase and Nakazono 1993; Ishihara and Kuwamura 1996). In a relatively high-density population of S. chrysopterum (0.59 fish/100 m2), smaller males formed pair territories and larger males were bigamous, which was related to male–male competition (Ishihara and Kuwamura 1996).

This was similar for R. aculeatus, in which the monopolization of multiple females was size dependent in a high-density population (0.86 fish/100 m2) and related to intrasexual competition for the access to females. In a previous study of Kuwamura (1997), only polygyny was observed in R. aculeatus at lower population density of 0.69 fish/100 m2 and an ASR of 0.31. In the present study, the ASR ranged between 0.29 and 0.33, and facultative monogamy was expressed throughout the whole study period, but was highest under a less female-biased ASR, namely 0.33. Therefore, plasticity in the mating system did not result from overall low population densities or higher (more male-biased) values of ASR per se. A comparison with Kuwamura’s study (1997), however, showed that females size ranged between 145 and 175 mm [total lengths (TL) was converted to standard length (SL): SL = 0.87 × TL − 0.93, Künzli and Tachihara 2012] and that females were overall larger compared to this study (115–170 mm). At the same time, the smallest male was 175 mm, while it was 150 mm in this study. It is therefore likely that relatively smaller males were prevented from access to females either because they were almost the same size as females (and rejected by females) or because the large competitive males successfully restricted smaller males from access to females, and monogamy was prevented. In addition, male size in Kuwamura’s study (1997) was over the threshold size for the ability of polygyny we found in this study (170 mm), and males were more likely to be haremic than monogamous. In an earlier study of Kuwamura (1991), sizes of R. aculeatus were indicated to range between 130 and 190 mm (TL converted to SL). According to the territorial arrangements, facultative monogamy was expressed even at lowest densities (0.49 fish/100 m2; study area modified for better comparison) and a highly female-biased ASR (0.29); but again the fish were relatively small (e.g. 130 mm, presumable females). This further supports our assumptions that access to females is size dependent in moderate to high densities of R. aculeatus. Facultative monogamy resulting from overall low population densities is expected when the density is much lower than that observed in this and previous studies.

Optimal life history patterns are determined by the fitness costs and benefits of different reproductive strategies, which underlie the trade-offs between components of fitness (Clutton-Brock 1984). For instance, if a haremic male negatively affects female fitness (e.g. growth, reproductive success), then resident females may attempt to prevent polygyny through intrasexual aggression (Kokita and Nakazono 2001). This was not the case for R. aculeatus as female mating success remained unaffected by their actual mating status, and instead female aggression was biased towards the males. Pairing with larger males, however, resulted in more number of matings in females, even though several females who paired with relatively large males did not spawn. The highest number of matings was achieved when females paired with males between 180 and 190 mm, while pairing with males <170 mm resulted in no reproduction. Females are therefore expected to be choosy.

On the other hand, males intended to sequester more than just one female because haremic males achieved much higher numbers of matings. Thus, polygyny was advantageous to the males as they benefitted from extra-matings with additional females. Larger males were competitively better in defending more females, which might be partly the result of size-dependent differences in energy reserves. However, some large males were still monogamous, suggesting high male–male competition over the access to females, and which could have forced some large males temporarily into monogamy. Sex-specific differences in mating success at a given mating status were further consistent with how females and males changed their mating status over time: males sought to become haremic and avoided to be monogamous, while no such trend was obvious in females.

In sexually dimorphic species, it is often difficult to determine whether sexual selection acts directly on female mate choice (intersexual selection) for male traits to increase female mating success or whether sexual selection acts through male–male competition (intrasexual selection). Since male parental care is absent in R. aculeatus, a large male body size is not expected to be associated with signalling particular parental qualities such as predator defence or brood care. Male parental effort is reduced to egg fertilization, and females may select indirectly for high fertile males. However, in a species with uniparental egg care, mating competition should be most intense in the lower investing sex (Trivers 1972).

In conclusion, our data suggest a high environmental potential for polygyny (Emlen and Oring 1977) in R. aculeatus despite the absence of female aggregation. Males defended site-attached females (female defence polygyny), rather than resources that females are attracted to. Females had non-overlapping home ranges and competed for better territories irrespective of the presence of males.

Male–male competition and the ability of females to reject less favoured males resulted in size-dependent access to multiple females, and plasticity in the mating system was expressed in high-density populations. Because haremic groups were the more stable territory unit and because polygyny positively affected the variance in male mating success—and concordantly did not negatively affect female mating success—polygyny can be considered as the evolutionary stable mating system in this species.

The results from this study further suggest that the degree of monogamy was related to demographic conditions. A varying adult sex ratio—mainly caused by differences in movement and recruitment pattern—resulted in the sequestration of one female, thereby favouring facultative monogamy or even promiscuity in R. aculeatus.