Introduction

Among carnivorans, mustelids represent one of the most ecomorphologically diverse clades, with taxa ranging from dietary generalists (e.g., martens) to piscivores (e.g., river otters), to carrion (e.g., wolverines) and mollusk specialists (sea otters) (Law et al. 2018a). Previous research has demonstrated that mustelids differ in terms of their craniomandibular musculature (e.g., Riley 1985), their cranial form (e.g., Dumont et al. 2016; Law et al. 2018a), and their bite force (e.g., Law et al. 2018a, b; Hartstone-Rose et al. 2019) depending on diet. Mustelids also differ in terms of the form of their dentition depending on their diet based on quantitative and meristic traits (Riley 1985; Popowics 2003). Riley (1985) argued that modifications to the cheek teeth represent the most striking difference among the jaw morphologies of modern mustelids (i.e., Martes, Enhydra, Lontra), with a shift towards grinding in taxa such as Enhydra. Popowics (2003) examined mustelid postcanine dimensions and found that Gulo gulo favors high levels of carnassial shearing on the lower first molar (m1), whereas E. lutris is characterized by an emphasis on crushing instead. The metaconid in these taxa is absent or reduced respectively, which contributes to the shearing adaptation of their carnassial. Martes and Lontra canadensis have lower molars that emphasize both carnassial and transverse shearing (transverse shearing surface provided by the presence of the metaconid), while also being adapted for crushing (Popowics 2003).

The relationship between dental form and function has been studied in great detail among extant primates and other closely related taxa (e.g., Butler 1963; Kay 1975; Kay and Hylander 1978; Boyer 2008; Bunn et al. 2011; Winchester et al. 2014; Selig et al. 2019). However, comparatively little research has aimed at examining the form-function relationship of the dentition and diet among extant mustelids, particularly in light of increased reliance on three-dimensional methods for studying the dentition. Recently, Waldman et al. (2023) examined the dental topography in a sample of living carnivorans and suggest that dental topography is indeed reflective of diet in this group, though they did not examine patterns of dental form among mustelids specifically. Moreover, Waldman et al. (2023) only sampled one individual for each species included, they examined all postcanine teeth as a single mesh or unit rather than looking at individual teeth as is more traditional in dental topographic analyses (DTA, see below), and they did not consider interspecific patterns of mustelid dental topography in their analysis. Here, DTA methods are employed, as well as methods for measuring enamel thickness and pulp volume to assess the form-function relationship of the carnassial m1 in a sample of four extant mustelid species: the wolverine (G. gulo), the beech marten (M. foina), the river otter (L. canadensis), and the sea otter (E. lutris).

Previous studies have examined the dental adaptations and morphology of mustelids (e.g., Riley 1985; Wolsan et al. 1985; Wolsan 1988; Popowics 2003; Constantino et al. 2011; Hopkins et al. 2022; Waldman et al. 2023); however, research is lacking concerning the suite of dental adaptations these animals exhibit in light of their differing diets. Moreover, most studies of extant mammalian dental form concern primates, but the degree that those patterns observed among primates converge with those of mustelids is currently unclear. The relationship between dental form and diet in mustelids is hypothesized here to be convergent with those of primates, where hard-object feeding, adaptations for resisting dental wear, and proportions of soft or tough foods consumed are manifest in the form of the molars.

The taxa included in the present study differ in terms of their dietary behaviors according to the available literature (Table 1). Martes foina is traditionally viewed as generalist omnivore, consuming a diet of fruit, insects, birds, and carrion. Though compared to other species of Martes, M. foina is described as being perhaps more highly reliant on fruit (Bakaloudis et al. 2012). Popowics (2003) describes the diet of Martes as being comprised largely of tough, soft tissue, with the addition of soft materials in their consumption of fruit. Gulo gulo is much more carnivorous, with the majority of the diet being comprised of animal tissue. They also are known to crack and consume large bones, with bones making up the majority of the material recovered in their scat (van Dijk et al. 2007). Popowics (2003) describes the diet of Gulo as being made up of tough, soft tissue, as well as hard, brittle bone. Lontra canadensis is another highly carnivorous species, with the majority of the diet made up of fish, though the makeup of the river otter diet varies seasonally and geographically, with some populations consuming larger quantities of crustaceans and/or mollusks (Roberts et al. 2008). Popowics (2003) describes the diet of Lontra as being made predominantly of soft tissue (i.e., fish). Finally, E. lutris is largely a hard-object feeder as this animal consumes large quantities of bivalves and sea urchins that are cracked open with their teeth. Although sea otters have been observed cracking invertebrate prey using rocks or other hard object (Fujii et al. 2017), this behavior varies significantly between populations and between individuals, with individuals using tools to crack food items between 0 and 98% of prey captures (Fujii et al. 2017). This behavior therefore likely has little effect on the dental morphology observed in E. lutris as a species. Popowics (2003) describes the diet of Enhydra as being largely made up of hard, brittle exoskeletons.

Table 1 Diet by taxon with references for each taxon. Food mechanical properties are adapted from Popowics (2003)
Full size table

It has been suggested that Martes is the most plesiomorphic extant mustelid, most closely resembling the omnivorous ancestral form, being characterized by a carnassial molar that is adapted for both shearing and crushing (Riley 1985). In examining their masticatory musculature and molar form, Riley (1985) suggested that a hypothetical lineage or morphocline from a land-dwelling omnivore to an extremely specialized aquatic sea otter can be represented from Martes to Lontra to Enhydra. I also considered the morphologies of these taxa to be stereotypical of the morphological stages connecting the ancestral mustelid condition with the derived condition of Enhydra on the one hand and Gulo on the other. While previous study has examined the transition from a generalized omnivore m1 to a dietary specialist (i.e., molluscivore) in this group of animals (e.g., Riley 1985), there has been no examination of three-dimensional tooth form or considerations of enamel thickness or other tissue proportions. Given that these animals split relatively recently (Fig. 1) and they differ in dietary ecology, they represent good model taxa for examining how phenotypic change may manifest given different ecologies among closely related taxa.

Fig. 1
figure 1

Relationships and divergence dates of the included taxa based on the phylogeny of Law et al. (2018b). Representative molars of Lontra (UAM:87–138-112), Enhydra (AMNH:MAMMALS:M-24186[reversed]), Martes (USNM:MAMMALS:173305[reversed]), and Gulo (AMNH:MAMMAL:M-165766) are not to scale. Silhouettes are from phylopic.org: Lontra and Enhydra courtesy of Margot Michaud (https://www.phylopic.org/images/dd5b2173-0ed5-47e7-90e5-f501dd231ae1/lontra-felina, https://www.phylopic.org/images/d3a78afb-1b9e-45e0-b6f4-144d79f399f0/enhydra-lutris), Martes courtesy of Ferran Sayol (https://www.phylopic.org/images/7f7431c6-8f78-498b-92e2-ebf8882a8923/martes-foina), and Gulo courtesy of Steven Traver (https://www.phylopic.org/images/1f9bbb79-f060-47c1-9954-ea78812d3b91/gulo-gulo-gulo)

Full size image

Variation in molar enamel thickness is commonly invoked as being related to differences in diet as thick enamel is thought to help teeth withstand the stress of hard and tough-object feeding (Dumont 1995; Lambert et al. 2004; Smith et al. 2005; Lucas et al. 2008; Braga et al. 2010; Constantino et al. 2011; Daegling et al. 2011). To date, variation in molar enamel thickness has not been examined among mustelids, with the exception of Enhydra (Constantino et al. 2011). Consistent with findings in extant primates, it is predicted that the hard-object feeding and bone cracking Enhydra and Gulo will be characterized by relatively thicker enamel compared to Lontra and Martes, whose molar enamel is theoretically subjected to less stress. Although Enhydra is characterized as having relatively thin enamel compared to modern humans (Constantino et al. 2011), it is still predicted that they will have thick enamel compared to other mustelids given their hard-object diet.

Finally, relative pulp volume has also been studied among extant primates (Selig et al. 2021a). Dental pulp helps in the maintenance and upkeep of dentine during life, and under the process of dental wear, helps in the deposit of tertiary dentine (Hamner III et al. 1964; Blumberg et al. 1971; Hillson 1996). Previous work has demonstrated that primates that consume a more abrasive diet that are consequently subject to greater dental wear, such as the hard-object feeding tufted capuchin (Sapajus apella) and the white-faced saki (Pithecia pithecia), have higher relative pulp volumes compared to closely related, more soft-object feeding taxa. It is therefore predicted that Enhyra and Gulo will be characterized by a higher relative pulp volume compared to the other taxa in the current analysis. This research will not only shed light on the dental adaptations of ecologically distinct animals, but will provide a framework for reconstructing diet in extinct mustelids as well.

Materials and methods

The sample

The specimens included in the sample are summarized in Table S1. All specimens were downloaded as micro-CT scan TIF stacks from MorphoSource (Boyer et al. 2016). Scans were reconstructed and analyzed in Avizo 8.1.1 (Visualization Sciences Group 2009). The sample includes four specimens of M. foina, four specimens of G. gulo, five specimens of L. canadensis, and three specimens of E. lutris. Only specimens with unworn to lightly worn m1s were included to avoid the confounding effects of wear, thus limiting the number of specimens that could be included.

Dental topographic analysis

Three dental topographic metrics were measured (Fig. 2). Surface curvature was measured using Dirichlet normal energy (DNE) (Bunn et al. 2011; Winchester et al. 2014; Pampush et al. 2016b; Winchester 2016). This metric is measured as the degree to which a surface deviates from being planar, capturing the level of curvature or sharpness of a surface. Taxa characterized by molars with high DNE have tall, sharp teeth, which is typically associated with consuming mechanically complex or tough foods such as leaves or insects/animal tissues (e.g., Bunn et al. 2011; Ledogar et al. 2013; Winchester et al. 2014; Pampush et al. 2016a, 2018; Selig et al. 2019, 2021b; López-Aguirre et al. 2022). Second, occlusal functional composition (sometimes described as surface complexity) was measured using three-dimensional orientation patch count rotated (3D-OPCR), which estimates the number of occlusal facets based on the number of discrete patches shared by adjacent polygons on a surface (Evans et al. 2007; Winchester 2016). Molars with more cusps, crests, and ridges have higher 3D-OPCR values and tend to be more characteristic of herbivores and insectivores, whereas low 3D-OPCR values tend to correlate with the simpler molars observed in frugivores, faunivores, and carnivores (Evans et al. 2007; Winchester et al. 2014; Selig et al. 2019; López-Aguirre et al. 2022). Omnivores tend to be intermediate between the two extremes. The third DTA metric was the relief index (RFI), which is a relative measure of crown height (Ungar and M’Kirera 2003; M’Kirera and Ungar 2003; Boyer 2008) calculated as the ratio of the crown area divided by the planimetric footprint of the occlusal plane of the tooth. This metric therefore reflects how tall a tooth is relative to its footprint, meaning teeth with tall crests and ridges, or tall teeth overall, have higher RFI values. Higher RFI values have been shown to reflect reliance on insects or on other animal tissues (Boyer 2008; Selig et al. 2019; López-Aguirre et al. 2022).

Fig. 2
figure 2

Reconstruction of the skull of Lontra canadensis (UCLA:MAMMALS:18958) with the carnassial molar highlighted in red. Meshes representing each topographic variable are shown blown up along side the skull

Full size image

Scans were segmented using Avizo 8.1.1 (Visualization Sciences Group 2009) to remove adjacent teeth. Segmented surfaces were cropped along the cervix to isolate the crown of each molar, were simplified to 10,000 faces, and smoothed 100 iterations with the lambda set at 0.6 (e.g., Boyer 2008; Bunn et al. 2011; Prufrock et al. 2016). Cropped meshes were imported into MorphoTester 1.1.1 (Winchester 2016) to measure DNE, 3D-OPCR, and RFI. The default settings for the measurement of DNE were used, with implicit fairing disabled and outlier removal percentile set at 99.9%, while the minimum patch size for 3D-OPCR was set to five polygons following Winchester (2016). Relief index was calculated as the simple ratio of the surface area of the crown to the surface area of the tooth projected on a parallel plane of the occlusal surface (Winchester 2016).

Average enamel thickness

Average enamel thickness was measured following the protocol outlined by Smith et al. (2005) and was calculated as the area of the enamel divided by the length of the enamel-dentine junction (EDJ) (Fig. 3). To take these measurements, each tooth was virtually sectioned with Avizo using the “slice” module. A slice was placed through the paraconid and protoconid perpendicular to the occlusal plane. Slices were rendered using interpolation and a user-defined texture map resolution of 4096 × 4096. An image of that slice was then opened in ImageJ 1.53 and scaled. The area of the enamel (mm2) was measured using the freehand selection tool, whereas the length of the EDJ (mm) was measured using the freehand line tool.

Fig. 3
figure 3

Representative illustration of a lower m1 of Lontra canadensis showing how average enamel thickness (AET) was calculated as the area of the enamel divided by the length of the enamel-dentine junction (EDJ) from a slice taken through the paraconid and protoconid

Full size image

Percent of tooth that is pulp (PTP)

The PTP (Fig. 4) was measured following the protocol outlined by Selig et al. (2021a) using Avizo. First, the hard tissues (enamel and dentine) were segmented using threshold-based and then manual approaches. The hard tissue was segmented as a single label field and measured using the “Surface Area Volume” module to calculate the volume of these tissues. Next, the pulp cavity was then segmented as an endocast, which was also measured using the “Surface Area Volume” module in Avizo. A consistent brightness/contrast setting was used when segmenting each scan to keep baseline thresholds the same throughout the stack, while a consistent threshold range was kept throughout each individual stack. The PTP was calculated as the volume of pulp (mm3) divided by the volume of the hard tissue, which was then multiplied by 100 to determine the percent of the total tooth volume represented by pulp.

Fig. 4
figure 4

Reconstruction of the m1 of Lontra canadensis (UAM:87–138-103) demonstrating how the percent of tooth that is pulp (PTP) was measured. Scale = 1 mm

Full size image

Statistical analysis

Data were plotted in boxplots to visualize the range of variation of each variable for each included taxon. All statistical tests were performed using PAST 4.03 (Hammer et al. 2001). The Kruskal–Wallis (non-parametric ANOVA) test was used to compare medians of each metric. In cases where the Kruskal–Wallis was significant, pairwise Dunn’s post hoc tests were used to determine which taxa differ from one another based on each metric. The Bonferroni correction was used for each Dunn’s post hoc test. Finally, a principal component analysis (PCA) was performed to visualize variation and better understand which variables explain a greater portion of the variation. The correlation matrix was used for the PCA to remove the effect of scale from non-scaled variables. Only those axes that explained more than 25% of the total variation were considered in the PCA.

Results

Results for each metric are summarized in Table 2 and in Fig. 5. Results of the Kruskal–Wallis are summarized in Table 3 and suggest that there are significant differences between sample medians based on each metric except for PTP. Results of the Dunn’s post hoc tests are summarized in Table 4. Based on the measurement of DNE, Lontra has the most highly curved teeth and has significantly higher DNE than Enhydra, which has the lowest DNE. There is more overlap and greater intraspecific variation in the measurement of 3D-OPCR, with Gulo and Lontra differing form one another significantly. Generally speaking, the lutrines have higher molar complexity than the gulonines. Based on the measurement of RFI, Enhydra has the lowest relief, whereas Martes has the highest, with this difference being significant. Gulo and Lontra are both intermediate in their RFI. The taxon with the highest AET, and therefore the thickest enamel, was Enhydra followed by Gulo. Enhydra has significantly higher AET than both Martes and Lontra. Finally, there is very little variation in PTP. Lontra generally has the highest PTP, followed by Enhydra, Martes, and then Gulo, but these differences are not significant.

Table 2 Mean, standard deviation (Stand. Dev), and the coefficient of variation (Coeff. Var) for the measurement of Dirichlet normal energy (DNE), three-dimensional orientation patch count rotated (3D-OPCR), relief index (RFI), average enamel thickness (AET), and the percent of tooth that is pulp (PTP). Sample size indicated in brackets for each taxon
Full size table
Fig. 5
figure 5

Boxplots showing the results of the measurement of Dirichlet normal energy (DNE), three-dimensional orientation patch count rotated (3D-OPCR), relief index (RFI), average enamel thickness (AET), and the percent of tooth that is pulp (PTP). The X denotes the mean, the horizontal lines denote the median, the boxes represent the upper and lower quartiles, and whiskers denote the highest and lowest values for each taxon

Full size image
Table 3 Results of the non-parametric Kruskal–Wallis test for each metric. DNE Dirichlet normal energy, 3D-OPCR three-dimensional orientation patch count rotated, RFI relief index, AET average enamel thickness, PTP percent of tooth that is pulp. Bolded values are significant
Full size table
Table 4 Bonferroni-corrected Dunn’s post hoc tests for each metric that varied significantly as per the Kruskal–Wallis (see Table 3). Bolded values are significant
Full size table

The first two axes of the PCA are plotted in Fig. 6. The eigenvalues and percent variance explained by each principal component, as well as the loadings for each variable, are summarized in Table 5. Along component 1, DNE, RFI, and AET explain most of the variation, with Lontra plotting in positive morphospace due to their high DNE and RFI, and low AET, whereas Enhydra plots in negative morphospace due to their high AET, and low DNE and RFI. Martes plots in more intermediate morphospace, near Lontra, likely due to their high RFI, whereas Gulo plots in more negative morphospace due to having relatively low DTA measures and thick enamel. Along component 2, 3D-OPCR explains the majority of the variation. This axis does not separate taxa as clearly as along component 1, but does distinguish Martes from Enhydra and Lontra given the low molar complexity of Martes. Gulo sits in intermediate morphospace along this axis.

Fig. 6
figure 6

Plot of the first two principal components depicting where each taxon sits in the morphospace. Variance explained by each PC is in brackets. The biplots indicate how Dirichlet normal energy (DNE), three-dimensional orientation patch count rotated (3D-OPCR), relief index (RFI), average enamel thickness (AET), and the percent of tooth that is pulp (PTP) are loaded along these two axes. Silhouettes are from phylopic.org: Lontra and Enhydra courtesy of Margot Michaud (https://www.phylopic.org/images/dd5b2173-0ed5-47e7-90e5-f501dd231ae1/lontra-felina, https://www.phylopic.org/images/d3a78afb-1b9e-45e0-b6f4-144d79f399f0/enhydra-lutris), Martes courtesy of Ferran Sayol (https://www.phylopic.org/images/7f7431c6-8f78-498b-92e2-ebf8882a8923/martes-foina), and Gulo courtesy of Steven Traver (https://www.phylopic.org/images/1f9bbb79-f060-47c1-9954-ea78812d3b91/gulo-gulo-gulo)

Full size image
Table 5 Eigenvalue and percent of total variance explained by each principal component. Loading coefficients for each variable included in the principal component analysis
Full size table

Discussion

These results are generally consistent with previous findings in that mustelid molar form varies as a product of dietary ecology, a pattern convergent with living primates (e.g., Butler 1963; Kay 1975; Kay and Hylander 1978; Boyer 2008; Bunn et al. 2011; Winchester et al. 2014; Selig et al. 2019). These results also support the notion that dental topography is reflective of diet among carnivorans (Waldman et al. 2023). However, these results provide new insight into more nuanced means of variation in carnassial form among mustelids.

Martes is described as being highly pleiomorphic in molar form, with a carnassial molar adapted for shearing and crushing foods (Riley 1985). The high RFI, intermediate DNE, and low 3D-OPCR exhibited by this taxon likely reflects the long, yet simple shearing crests present on the carnassial. Martes is characterized by relatively thin enamel, likely as a result of the animal consuming very few hard foods relative to Enhydra or Gulo. If the carnassial molar of Martes is viewed as the starting point in the morphocline examined herein, Gulo may be next in the evolution of more highly specialized molars. Gulo is characterized by a suite of dental features that are similar to those of Martes, with intermediate DNE and low complexity. However, Gulo is characterized by thicker enamel and lower RFI, likely reflecting the bone cracking behavior observed in wolverines. Both Martes and Gulo are characterized as consuming tough, soft tissue (Popowics 2003), which may explain their similar dental topographic values.

The lower first molar of Lontra still maintains the carnassial bauplan observed in Martes, but is characterized by a more distinctive suite of adaptations. For example, Lontra has generally high topographic values, reflecting their sharp and complex shearing crests, which likely aid in capturing and processing soft, slippery fish. Lontra is characterized by thin enamel, as with Martes. However, observational work suggests that Lontra consumes at least some prey items that could be considered “hard,” such as mollusks or crustaceans, which may make up large portions of the diet seasonally (Roberts et al. 2008). The finding of thin enamel is somewhat unexpected considering both Gulo and Enhydra have thick enamel associated with their diet, whereas Lontra does not. What’s more is the pattern of carnassial topography observed in Lontra is very much unlike that of Enhydra, which is characterized by a significant reduction in crests and cusp height. Although Lontra consumes some similar hard food items, clearly the selective pressures for teeth adapted to process soft fish tissue were greater than those for resisting dental wear or damage from the consumption of hard prey. Enhyrda is characterized by relatively thin enamel compared to humans, but has enamel microstructure that is thought to help resist wear and damage (Constantino et al. 2011). Perhaps enamel microstructure in Lontra also helps this animal resist wear and damage, while maintaining sharp shearing blades associated with piscivory. Research into dental damage, such as enamel chipping (e.g., Fannin et al. 2020; Towle and Loch 2021), may also provide further insight into the apparent evolutionary tradeoff in the carnassial molars of Lontra.

The carnassial molars of Enhydra are also characterized by a relatively unique suite of dental features compared to the plesiomorphic Martes, as well as the other included mustelids; Enhydra has low curvature (DNE) and relief (RFI), and very thick enamel. Certainly, these adaptations reflect the reliance on hard, brittle prey items such as clams. This reduction in shearing crests and cusp height is convergent on what is observed among hard-object feeding primates such as bearded sakis (Chiropotes) or mangabeys (Cercocebus and Lophocebus) (Ledogar et al. 2013; Avià et al. 2022). Enhydra, therefore, represents an extreme on the spectrum of carnassial evolution among mustelids, showing a high degree of specialization and a shift towards crushing and grinding and away from the carnassial shear observed in Martes (Riley 1985).

Previous work demonstrated that anthropoid primates that consume a hard/abrasive diet are characterized by a higher relative volume of molar pulp compared to closely related taxa with softer diets (Selig et al. 2021a). This phenomenon is thought to reflect the fact the dental pulp provides, in essence, the raw material for the deposition of tertiary dentine, which helps teeth resist wear. It was predicted here that Gulo, and certainly Enhydra, would be characterized by a relatively higher volume of pulp compared to Lontra and Martes, which consume lower volumes of hard or abrasive foods. The results of the current study do not support this hypothesis, finding no significant differences in pulp volume among the selected taxa, with Martes and then Lontra characterized by the highest mean PTP. It could be argued that both Martes and Lontra also consume a hard or abrasive diet in that Martes crushes the bones of the small prey that it consumes (birds and reptiles), and that Lontra consumes high volumes of hard and brittle crustaceans. However, given the other suite of dental features examined here, it is surprising to see no evidence of adaptations for resisting wear via high pulp volumes when carnassial topography and enamel thickness clearly reflect this dietary component in Gulo and Enhydra. Perhaps thick enamel, or even enamel microstructure (Constantino et al. 2011), provides enough resistance among mustelid carnassials such that there has been little to no adaptive pressure to increase the store of pulp. Moreover, variation in PTP among anthropoids may be apomorphic among this group. More research into variation in molar pulp volume among mammals is clearly needed to fully understand this adaptation.

Although previous research has demonstrated that mustelid dental topography varies as a product of diet (Waldman et al. 2023), the present study may provide a more directly applicable means for reconstructing diet in the mustelid fossil record. Indeed, much of the mammalian fossil record is comprised of isolated teeth, meaning comparative studies concerning single teeth may be more useful compared to those that examine the entire toothrow. Considering the degree of variation in the lower m1 observed in the present study, this tooth position in particular seems to be useful in discerning diet and will likely provide insight into the dietary adaptations of extinct mustelids.