1 Introduction

The rapid advancement of advanced manufacturing technologies has made it feasible to fabricate composite parts with complex shapes and tailored fiber orientations (Tian et al. 2022; Cheng et al. 2022). This development creates significant opportunities for designing structures with enhanced performance (Liu et al. 2021; Fernandes et al. 2021). However, fully leveraging the design freedom offered by these technologies remains a challenge.

In recent years, there has been progress in the field of topology optimization of continuous fiber-reinforced composites (Jiang et al. 2019; Safonov 2019; Chen and Ye 2021; Zhou et al. 2022; Qiu et al. 2022). Unlike that with isotropic materials, topology optimization with fiber-reinforced composite structures requires determining not only the presence of material at a given point in the design domain but also the fiber orientation at that point. Thus, parametrization of the spatially varying fiber orientations plays a critical role in the effectiveness of topology optimization of continuous fiber-reinforced composites.

The rotation-based approach is widely used to parameterize fiber orientation. At least three mathematical constructs have been applied for this purpose: Euler angles, Cartesian representation, and quaternions. In the Euler angles-based method, two angles are employed to describe the fiber orientation in 3D space (Schmidt et al. 2020; Fedulov et al. 2021). When the planar slicing strategy in conventional 3d printing is considered, a single angle per element can be sufficient, as the fiber orientation is restricted to the printing plane (Jiang et al. 2019). While the Euler angles-based parametrization is intuitive, it can encounter singularity issues during the optimization process (Nomura et al. 2015). This can be problematic, as the presence of singularities can adversely affect the optimization convergence and stability.

To mitigate the singularity issues associated with the Euler angles-based parametrization, the Cartesian representation has been used to represent fiber orientation in the simultaneous design of density and orientation of anisotropic materials for 2D problems (Nomura et al. 2015). In this approach, an isoparametric projection method is proposed to ensure that the direction vector represented by the two Cartesian components is of unit length. The Cartesian representation has been further employed in topology optimization for the continuous orientation design of functionally graded fiber-reinforced composite structures (Lee et al. 2018). In this case, the requirement of unit length for the fiber direction vector is satisfied simply by normalization, and the mapping between the fiber direction vector and the rotated stiffness matrix can be readily obtained. However, the use of Cartesian representation for 3D problems is not as straightforward (Smith and Norato 2021). Recently, the Cartesian representation has been employed to parameterize fiber orientation for 3D problems (Luo et al. 2023), but the design variables are still transformed into Euler angles, which inevitably introduces singularity issues. Nevertheless, it has been demonstrated that the Cartesian parametrization can be readily used to account for manufacturing constraints (Luo et al. 2023, 2024). This suggests that the Cartesian representation-based approach holds promise for addressing the singularity issues while providing a flexible framework for incorporating practical considerations in the topology optimization of continuous fiber-reinforced composites.

Quaternions are more suitable for representing the orientation of general orthotropic materials in 3D space (Kubalak et al. 2021). However, to represent a pure rotation, the quaternion must be of unit length. This requirement increases the solving difficulty of the topology optimization problems because element-wise quadratic constraints need to be involved. To address this issue, the concept of “natural quaternions” has been introduced in the work by Kubalak et al. (2021). In this approach, the unit length constraints are implicitly enforced, and only simple side constraints are required. While quaternions of less than unit length are allowed, elements in this state would be artificially weaker. For minimum compliance design problems, the optimization algorithm naturally drives the quaternions to unit length. However, for other types of problems, this approach may lead to the generation of non-physical designs, as the optimization is not explicitly constrained to maintain the unit length requirement.

An alternative approach to addressing the challenges associated with the rotation-based parametrization is to directly parameterize the terms of the stiffness tensor (Zowe et al. 1997). This removes the concept of rotation and the associated complexity. However, this approach requires the involvement of quadratic constraints to ensure that the resulting material property tensor is positive semi-definite, making that the topology optimization problem involve a large number of nonlinear equality constraints. To address this issue, a second-order unidirectional tensor-based representation has been proposed for the fiber orientation in the work by Nomura et al. (2015, 2019). Since the unidirectional tensor is symmetric, the six upper triangle elements are taken as the design variables. To ensure that these six values constitute a valid tensor, a multi-variable function projection (Nomura et al. 2015) is employed to satisfy the tensor invariant constraints. The corresponding stiffness matrix is then obtained according to the fourth-order unidirectional tensor defined by the directional vector. This unidirectional tensor-based representation has been applied to large-scale topology optimization of continuous fiber-reinforced lightweight composites (Zhou et al. 2022). Furthermore, this representation has been extended to the inverse design of fiber-reinforced composites with spatially varying fiber size and orientation (Nomura et al. 2019). In this case, a simple projection is employed for the diagonal terms of the orientation design variables, while the isoparametric shape functions are used for the off-diagonal terms. However, the authors have noted that the optimization results may contain some areas that violate the conditions for tensor invariants (Nomura et al. 2019). A novel material orientation filter is proposed in Jantos et al. (2020) to control the smoothness of the fiber pathways, where the filter is directly applied to the stiffness matrix, rather than the actual parametrization for the fiber orientations. In each optimization step, the proposed material orientation filter requires the recovery of the Euler angles from the rotation matrix.

In addition to the aforementioned parametrization approaches, it is also worth noting that a novel non-parametric shape and topology optimization for fiber placement design of CFRP plate and shell structures is developed in Shimoda et al. (2023). In this approach, the problem is formulated as a distributed-parameter optimization problem, where the optimal fiber placement and orientation can be obtained without the need for any explicit parametrization. This non-parametric formulation allows for a more flexible and unconstrained representation of the fiber paths, potentially leading to more optimal designs compared to the parameterized approaches.

This study focuses on the topology optimization of 3D continuous fiber-reinforced composites and revisits the Cartesian representation-based parameterization for spatially varying fiber orientations. However, unlike the work by Luo et al. (2023), the rotated stiffness matrix is directly obtained according to the Cartesian representation without the need for Euler angles. A key distinction of this work is that fiber-reinforced composite materials often exhibit transversal isotropy, which provides an opportunity to develop a very simple way to establish a direct mapping between the Cartesian representation and the rotated stiffness matrix. Building upon the proposed parametrization, the formulation for topology optimization of continuous fiber-reinforced composites is presented. Additionally, a solution method is outlined to efficiently address the optimization problem. To validate the effectiveness of the proposed method, numerical examples are conducted, showcasing its capabilities and advantages in achieving optimized designs.

The remainder of this paper is structured as follows. In Sect. 2, the Cartesian parametrization for 3D fiber orientations is presented. Section 3 outlines the problem formulation for topology optimization of continuous fiber-reinforced composites, covering aspects such as design parametrization and regularization, material interpolation schemes, and the formulation of the minimum compliance design problem. The solution method, including sensitivity analysis and decoupled design variable updating, is described in Sect. 4. Three numerical examples are provided in Sect. 5 to illustrate the performance of the proposed Cartesian parametrization and its effectiveness in solving topology optimization problems for continuous fiber-reinforced composites. Finally, Sect. 6 summarizes the conclusions drawn from this work.

2 Cartesian parametrization for 3D fiber orientations

In this section, we provide a comprehensive explanation of the Cartesian parametrization designed for 3D fiber orientations. The crucial aspect of this parametrization lies in how to establish a mapping between the orientation parameters and the rotated material stiffness matrix.

Continuous fiber-reinforced composite materials generally exhibit transversally isotropic behavior. In the fiber direction (longitudinal direction), the mechanical properties (such as modulus, strength, etc) are significantly higher than those in the transverse direction, while within the transverse plane (the plane perpendicular to the fiber direction), the mechanical properties are isotropic; i.e., they are the same in all directions.

The stress–strain relationship of fiber-reinforced composite material in the principal material coordinates is described as follows:

$$\begin{aligned} {\varvec{\sigma }} = \mathbf{{D}}{\varvec{\varepsilon }}, \end{aligned}$$
(1)

where the stress and strain tensors are expressed in the Voigt notation as \({\varvec{\sigma }} = {[{\sigma _{11}},{\sigma _{22}},{\sigma _{33}},{\tau _{23}},{\tau _{31}},{\tau _{12}}]^{\mathrm{{T}}}}\) and \({\varvec{\varepsilon }} = {[{\varepsilon _{11}},{\varepsilon _{22}},{\varepsilon _{33}},{\gamma _{23}},{\gamma _{31}},{\gamma _{12}}]^{\mathrm{{T}}}}\). The stiffness matrix is found from the inverse of the compliance matrix as follows:

$$\begin{aligned} \mathbf{{D}}\mathrm{{ = }}{\left[ {\begin{array}{*{20}{c}} {1/{E_1}}&{}{ - {v_{12}}/{E_1}}&{}{ - {v_{12}}/{E_1}}&{}0&{}0&{}0\\ { - {v_{12}}/{E_1}}&{}{1/{E_2}}&{}{ - {v_{23}}/{E_2}}&{}0&{}0&{}0\\ { - {v_{12}}/{E_1}}&{}{ - {v_{23}}/{E_2}}&{}{1/{E_2}}&{}0&{}0&{}0\\ 0&{}0&{}0&{}{1/{G_{23}}}&{}0&{}0\\ 0&{}0&{}0&{}0&{}{1/{G_{12}}}&{}0\\ 0&{}0&{}0&{}0&{}0&{}{1/{G_{12}}} \end{array}} \right] ^{ - 1}}, \end{aligned}$$
(2)

where \(E_1\) and \(E_2\) are the Young moduli of the composite material along and perpendicular to the fiber directions, respectively, \(v_{12}\) and \(v_{23}\) are the Poisson ratios, \(G_{12}\) and \(G_{23}\) are shear moduli. For transversely isotropic materials, the relationship \(G_{23}=E_2/2(1+v_{23})\) holds. Thus, only five independent coefficients, \({E_1},{E_2},{v_{12}},{v_{23}}\) and \({G_{12}}\), are sufficient to describe the mechanical properties of the composite materials in the principal material coordinates.

As shown in Fig. 1, suppose that in the global coordinates the three principal material axes 1, 2, and 3 are aligned with \({\mathbf{{e}}_1}\), \({\mathbf{{e}}_2}\), and \({\mathbf{{e}}_3}\), respectively, where

$$\begin{aligned} {\mathbf{{e}}_1} = {[{l_1},{m_1},{n_1}]^{\mathrm{{T}}}} \end{aligned},$$
(3)
$$\begin{aligned} {\mathbf{{e}}_2} = {[{l_2},{m_2},{n_2}]^{\mathrm{{T}}}} \end{aligned},$$
(4)
$$\begin{aligned} {\mathbf{{e}}_3} = {[{l_3},{m_3},{n_3}]^{\mathrm{{T}}}} \end{aligned}.$$
(5)

Then, the stress–strain relationship in the global coordinates XYZ can be described by the rotated stiffness matrix, which is given by (Kubalak et al. 2021)

$$\begin{aligned} {\mathbf{{D}}_R} = {\mathbf{{T}}_R}{} \mathbf{{DT}}_R^T \end{aligned},$$
(6)

where the transformation matrix \({\mathbf{{T}}_R}\) is provided in Eq. (7).

$$\begin{aligned} {T_R} = \left[ {\begin{array}{*{20}{c}} {l_1^2}&{}{l_2^2}&{}{l_3^2}&{}{2{l_2}{l_3}}&{}{2{l_3}{l_1}}&{}{2{l_1}{l_2}}\\ {m_1^2}&{}{m_2^2}&{}{m_3^2}&{}{2{m_2}{m_3}}&{}{2{m_3}{m_1}}&{}{2{m_1}{m_2}}\\ {n_1^2}&{}{n_2^2}&{}{n_3^2}&{}{2{n_2}{n_3}}&{}{2{n_3}{n_1}}&{}{2{n_1}{n_2}}\\ {{m_1}{n_1}}&{}{{m_2}{n_2}}&{}{{m_3}{n_3}}&{}{{m_2}{n_3} + {m_3}{n_2}}&{}{{m_1}{n_3} + {m_3}{n_1}}&{}{{m_1}{n_2} + {m_2}{n_1}}\\ {{n_1}{l_1}}&{}{{n_2}{l_2}}&{}{{n_3}{l_3}}&{}{{n_3}{l_2} + {n_2}{l_3}}&{}{{n_3}{l_1} + {n_1}{l_3}}&{}{{n_2}{l_1} + {n_1}{l_2}}\\ {{l_1}{m_1}}&{}{{l_2}{m_2}}&{}{{l_3}{m_3}}&{}{{l_2}{m_3} + {l_3}{m_2}}&{}{{l_1}{m_3} + {l_3}{m_1}}&{}{{l_1}{m_2} + {l_2}{m_1}} \end{array}} \right] \end{aligned}.$$
(7)

It can be seen that when \({\mathbf{{e}}_1}\), \({\mathbf{{e}}_2}\), and \({\mathbf{{e}}_3}\) are all known, the rotated stiffness matrix can be readily obtained. However, in the topology optimization of continuous fiber-reinforced composite structures, usually only the fiber direction \({\mathbf{{e}}_1}\) is given. In such cases, there is no obvious choice for the selection of \({\mathbf{{e}}_2}\) and \({\mathbf{{e}}_3}\) (Smith and Norato 2021). This section proposes a simple and explicit way to construct \({\mathbf{{e}}_2}\) and \({\mathbf{{e}}_3}\). For transversely isotropic fiber-reinforced composite materials, the material properties are the same in any direction perpendicular to the fiber direction. Therefore, any direction in the transverse plane can be selected as \({\mathbf{{e}}_2}\).

Fig. 1
figure 1

Fiber orientation description in global coordinate system

Let us introduce the following vector:

$$\begin{aligned} {\varvec{\tau }} = {[{m_1} - {n_1},{n_1} - {l_1},{l_1} - {m_1}]^T} \end{aligned}.$$
(8)

It can be readily verified that

$$\begin{aligned} {{\varvec{\tau }}^T}{\mathbf{{e}}_1} = {l_1}({m_1} - {n_1}) + {m_1}({n_1} - {l_1}) + {n_1}({l_1} - {m_1}) = 0 \end{aligned}.$$
(9)

Thus, the unit vector \({\mathbf{{e}}_2}\) can be constructed by normalizing the vector \({\varvec{\tau }}\) as follows:

$$\begin{aligned} {\mathbf{{e}}_2} = {\varvec{\tau }}/\left\| {\varvec{\tau }} \right\| \end{aligned}.$$
(10)

After \({\mathbf{{e}}_2}\) is obtained, the unit vector \({\mathbf{{e}}_3}\) can then be computed as the cross product of \({\mathbf{{e}}_1}\) and \({\mathbf{{e}}_2}\):

$$\begin{aligned} {\mathbf{{e}}_3} = {\mathbf{{e}}_1} \times {\mathbf{{e}}_2} \end{aligned}.$$
(11)

This approach ensures that the vectors \({\mathbf{{e}}_1}\), \({\mathbf{{e}}_2},\) and \({\mathbf{{e}}_3}\) form an orthonormal basis, which is a necessary requirement for the subsequent transformation of the material properties from the principal material coordinates to the global coordinate system.

It can be seen that by this approach, the values of \({l_i}, {m_i}, {n_i} (i = 2, 3)\) can be constructed once the fiber direction vector \({[{l_1},{m_1},{n_1}]^{\mathrm{{T}}}}\) is given. Consequently, the mapping between the rotated stiffness matrix \({\mathbf{{D}}_R}\) and the fiber orientation \(\mathbf{{e}}_1\) can be established. Thus, for any given fiber direction, the corresponding rotated stiffness matrix can be obtained. The direct mapping between the Cartesian representation of the fiber orientation and the corresponding stiffness matrix, without the need for Euler angles, is a central aspect of the proposed parametrization approach and a key contribution of this work.

3 Problem formulation

In this section, we provide a brief overview of the problem formulation for topology optimization of 3D continuous fiber-reinforced composites. To perform topology optimization, the density-based approach (Bendsøe 1989; Zhou and Rozvany 1991) is utilized in this study. For a more comprehensive understanding of the methodology, we recommend referring to the relevant literature where further details can be found in Schmidt et al. (2020).

3.1 Design parametrization and regularization

Assume that the design domain is discretized into N elements and the volume of each element is \({v_1},{v_2}, \cdots ,{v_N}\), respectively. In the topology optimization problem, each element is assigned four design variables: \({\eta _e}, {p_{\mathrm{{e}}1}}, {p_{\mathrm{{e}}2}}, {p_{\mathrm{{e}}3}}\). Here, \({\eta _e}\) indicates whether the element is filled with material or void, while \({p_{\mathrm{{e}}1}}, {p_{\mathrm{{e}}2}}, {p_{\mathrm{{e}}3}}\) represent the fiber direction of the composite material within that element. Consequently, there are a total of 4N design variables in the optimization problem.

To mitigate numerical issues such as the checkerboard phenomenon and mesh-dependency commonly arise in continuum topology optimization, a filtering technique, as proposed in Bourdin (2001) and Schmidt et al. (2020), is employed to regularize both the density and fiber orientation variable fields. Specifically, the density filter is applied to transform the original density design variable \({\eta _e}\), resulting in a modified density value as follows:

$$\begin{aligned} {{\tilde{\eta }} _e} = \frac{{\sum \limits _{j \in {\aleph _e}} {{H_{ej}}} {v_j}{\eta _j}}}{{\sum \limits _{j \in {\aleph _e}} {{H_{ej}}{v_j}} }} \end{aligned},$$
(12)

where \({\aleph _e}\) is the set of elements j for which the center-to-center distance d(ej) to element e is smaller than the filter radius \(r_{min}\) and \({H_{ej}}\) is a weight factor defined as follows:

$$\begin{aligned} {H_{ej}}=max(0,r_{min}-d(e,j)). \end{aligned}$$
(13)

Similarly, the filtered orientation variables are obtained as follows:

$$\begin{aligned} {\tilde{\varvec{p}}_e} = \frac{{\sum \limits _{j \in {\aleph _e}} {{H_{ej}}} {v_j}{{\hat{\varvec{p}}}_j}}}{{\sum \limits _{j \in {\aleph _e}} {{H_{ej}}{v_j}} }}, \end{aligned}$$
(14)

where

$$\begin{aligned} {\hat{\varvec{p}}_j}\mathrm{{ = }}\left\{ {\begin{array}{*{20}{c}} {{{\varvec{p}}_j},\mathrm{{ if}}\mathrm{{ }}{{\varvec{p}}_e} \cdot {{\varvec{p}}_j} \ge 0}\\ { - {{\varvec{p}}_j},\mathrm{{ otherwise}}} \end{array}} \right. \end{aligned}.$$
(15)

It should be noted that in practical engineering applications, the density and orientation design variables can potentially be filtered using different filter radii. This is because the filter radius for the density filter is primarily used to control the length scale of the structural features, while the filter radius for the orientation variables is used to regulate the smoothness of the fiber orientation transitions. Therefore, it is common to employ different filter radii for these two types of design variables.

The filtered orientation vector \({\tilde{\varvec{p}}_e}\) may not be of unit length after the filtering operation. To ensure that the rotated stiffness matrix is properly parameterized, \({\tilde{\varvec{p}}_e}\) is further normalized, resulting in a unit direction vector:

$$\begin{aligned} {\bar{\varvec{p}}_e} = \frac{{{{\tilde{\varvec{p}}}_e}}}{{\left\| {{{\tilde{\varvec{p}}}_e}} \right\| }}. \end{aligned}$$
(16)

The normalization of the filtered orientation vector guarantees that the physical orientation vector \({{\varvec{{\bar{p}}}}_e}\) represents a valid fiber direction within the topology optimization problem. Once the unit orientation vector \({{\varvec{{\bar{p}}}}_e}\) is obtained, the rotated stiffness matrix can be computed according to the mapping method developed in Sect. 2.

3.2 Material interpolation scheme

In the topology optimization of composite structures, the modified SIMP (Solid Isotropic Material with Penalization) model is employed to achieve clear designs (Andreassen et al. 2011). The stiffness matrix of the composite material in the eth element can be defined as follows:

$$\begin{aligned} {\mathbf{{D}}_e} = g({{\tilde{\eta }} _e})\mathbf{{D}}_e^0({\bar{p}_{\mathrm{{e}}1}},{{\bar{p}}_{\mathrm{{e}}2}},{{\bar{p}}_{\mathrm{{e}}3}}), \end{aligned}$$
(17)

where \(\mathbf{{D}}_e^0({{\bar{p}}_{\mathrm{{e}}1}},{{\bar{p}}_{\mathrm{{e}}2}},{\bar{p}_{\mathrm{{e}}3}})\) is the rotated stiffness matrix of the composite material corresponding to the fiber direction vector \({[{l_1},{m_1},{n_1}]^{\mathrm{{T}}}} = {[{{\bar{p}}_{e1}},{\bar{p}_{e2}},{{\bar{p}}_{e3}}]^{\mathrm{{T}}}}\) and it can be obtained by Eqs. (6, 7, 81011). To penalize intermediate densities and promote clear designs, a penalization function \(g({{\tilde{\eta }} _e})=\epsilon +(1-\epsilon ){{\tilde{\eta }} _e}^p\) is used, with \(p=3\) and \(\epsilon =10^{-6}\).

3.3 Formulation of minimum compliance design

The equilibrium equation of a structure under an external load can generally be written as follows:

$$\begin{aligned} \mathbf{{Ku}} = \mathbf{{f}}, \end{aligned}$$
(18)

where \(\textbf{K}\) is the global stiffness matrix, \(\mathbf{{f}}\) is the global force vector corresponding to the load applied to the structure.

The global stiffness matrix \(\textbf{K}\) can be obtained by assembling the corresponding element stiffness matrices. The element stiffness matrix \(\mathbf{{k}}_e\) can be computed as follows:

$$\begin{aligned} {\mathbf{{k}}_e} = g({{\tilde{\eta }} _e})\mathbf{{k}}_e^0, \end{aligned}$$
(19)

where

$$\begin{aligned} \mathbf{{k}}_e^0 = \int _{{\Omega _e}} {{\mathbf{{B}}^{\mathrm{{T}}}}{} \mathbf{{D}}_e^0\mathbf{{B}}d\Omega } \end{aligned}$$
(20)

with \(\textbf{B}\) being the matrix of shape function derivatives.

Finally, minimum compliance design of composite structures under an external force can be formulated as follows:

$$\begin{aligned} \begin{aligned} \mathop {\min }\limits _{{\varvec{\eta }},{\varvec{p}}}:&c = {\mathbf{{f}}^T}{} \mathbf{{u}}\\ s.t.:&\mathbf{{Ku}} = \mathbf{{f}}\\&(\sum \limits _{e = 1}^{N} {{v_e}{{\tilde{\eta }} _e}} )/V - {v_f} \le 0\\&0 \le {\eta _e} \le 1,1 \le e \le N\\&- 1 \le {p_{e1}} \le 1,1 \le e \le N\\&- 1 \le {p_{e2}} \le 1,1 \le e \le N\\&- 1 \le {p_{e3}} \le 1,1 \le e \le N\\ \end{aligned} \end{aligned}.$$
(21)

Here for simplicity, \({\varvec{\eta }}\) and \({\varvec{p}}\) denote the density and orientation design variables vectors, respectively. \(\textbf{f}\) and \(\textbf{u}\) are the load vector and the corresponding displacement vector, respectively. \(v_e\) is the volume of element e, V is the volume of the design domain, and \(v_f\) is the prescribed volume fraction.

In this approach, it is not necessary to enforce unit length constraint on the orientation variables. Rather, only bound constraints on the design variables are introduced, making it particularly suitable for solving large-scale problems.

4 Solution method

One computational challenge in solving the topology optimization problem (19) is the nested finite element analysis, especially when the number of elements used to discretize the design domain is large. However, when the Cartesian grids are employed, the template stiffness matrices (TSMs)-based method can be used to efficiently perform the finite element analysis (Chandrasekhar et al. 2023; Zhao et al. 2024). Thus, this section will focus on the sensitivity analysis method and design variables update strategy.

4.1 Sensitivity analysis method

To solve topology optimization problems, sensitivity analysis is an indispensable step to quantify the influence of each design variable on the structural performance. According to the adjoint method (Bendsøe and Sigmund 2003), when the load is design independent, the sensitivity of the structural compliance with respect to any design variable \(x_e\) can be given as follows:

$$\begin{aligned} \frac{{\partial c}}{{\partial {x_e}}} = - \mathbf{{u}}_e^T\frac{{\partial {\mathbf{{k}}_e}}}{{\partial {x_e}}}{} \mathbf{{u}}_e \end{aligned}.$$
(22)

Substituting Eq. (17) into Eq. (20) yields

$$\begin{aligned} \frac{{\partial c}}{{\partial {{\tilde{\eta }} _e}}} = - g'({\tilde{\eta }_e})\mathbf{{u}}_e^T\mathbf{{k}}_e^0\mathbf{{u}}_e \end{aligned},$$
(23)
$$\begin{aligned} \frac{{\partial c}}{{\partial {{\bar{p}} _{ei}}}} = - g({{\tilde{\eta }} _e})\mathbf{{u}}_e^T\frac{{\partial \mathbf{{k}}_e^0}}{{\partial {{\bar{p}} _{ei}}}}{\mathbf{{u}}_e},i=1,2,3. \end{aligned}$$
(24)

When the TSMs-based method is employed to compute the element stiffness matrices (Zhao et al. 2024), \(\mathbf{{k}}_e^0\) can be expressed as a linear combination of 21 TSMs,

$$\begin{aligned} \mathbf{{k}}_e^0 = \sum \limits _{n = 1}^{21} {\mathbf{{D}}_{e,n}^0\mathbf{{k}}_{t,n}^0}, \end{aligned}$$
(25)

where \(n = i(i - 1)/2 + j\) and

$$\begin{aligned} \mathbf{{D}}_{e,n}^0 = \mathbf{{D}}_{e,ij}^0, \end{aligned}$$
(26)
$$\begin{aligned} \mathbf{{k}}_{t,n}^0 = \left\{ {\begin{array}{*{20}{c}} {\mathbf{{k}}_{e,ii}^0,}&{}{i = j}\\ {\mathbf{{k}}_{e,ij}^0 + \mathbf{{k}}_{e,ji}^0,}&{}{i > j} \end{array}} \right. \end{aligned}.$$
(27)

Here \(\mathbf{{D}}_{e,ij}^0\) is the element of \(\mathbf{{D}}_{e}^0\) at the position of row i and column j, \(\mathbf{{k}}_{e,ij}^0=\int _{{\Omega _e}} {\mathbf{{B}}_e^T} {\mathbf{{D}}_{ij}}{\mathbf{{B}}_e}d\Omega (i,j,=1,\cdots ,6)\) with \(\mathbf{{D}}_{ij}\) being a \(6\times 6\) matrix with 1 at the position of row i and column j and 0 at other positions.

Then, the sensitivity of the elemental stiffness matrix with respect to the physical orientation variables can be computed as follows:

$$\begin{aligned} \frac{{\partial \mathbf{{k}}_e^0}}{{\partial {{\bar{p}}_{ei}}}} = \sum \limits _{n = 1}^{21} {\frac{{\partial \mathbf{{D}}_{e,n}^0}}{{\partial {{{\bar{p}}}_{ei}}}}{} \mathbf{{k}}_{t,n}^0}. \end{aligned}$$
(28)

In this work, the sensitivity term \(\partial \mathbf{{D}}_e^0/\partial {{\bar{p}}_{ei}}\) in Eq. (27) is computed using the forward finite difference method to avoid complicated analytical derivation.

The sensitivity of the volume constraint function with respect to the physical design variables can also be obtained explicitly, but will be omitted here for the sake of brevity.

Once the sensitivity of the objective and constraint functions with respect to the physical design variables is available, their sensitivity with respect to the original design variables can be obtained based on the chain rule of differentiation. The derivation for the density variables is trivial and will not be presented here.

For the orientation variables, according to the chain rule of differentiation, the sensitivity of the compliance with respect to the filtered orientation variables is given as follows:

$$\begin{aligned} \begin{aligned} \left[ {\begin{array}{*{20}{c}} {\frac{{\partial c}}{{\partial {{{\tilde{p}}}_{e1}}}}}\\ {\frac{{\partial c}}{{\partial {{{\tilde{p}}}_{e2}}}}}\\ {\frac{{\partial c}}{{\partial {{{\tilde{p}}}_{e3}}}}} \end{array}} \right] = \left[ {\begin{array}{*{20}{c}} {\frac{{\partial {{{\bar{p}}}_{e1}}}}{{\partial {{{\tilde{p}}}_{e1}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e2}}}}{{\partial {{{\tilde{p}}}_{e1}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e3}}}}{{\partial {{{\tilde{p}}}_{e1}}}}}\\ {\frac{{\partial {{{\bar{p}}}_{e1}}}}{{\partial {{{\tilde{p}}}_{e2}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e2}}}}{{\partial {{{\tilde{p}}}_{e2}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e3}}}}{{\partial {{{\tilde{p}}}_{e2}}}}}\\ {\frac{{\partial {{{\bar{p}}}_{e1}}}}{{\partial {{\tilde{p}}_{e3}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e2}}}}{{\partial {{\tilde{p}}_{e3}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e3}}}}{{\partial {{\tilde{p}}_{e3}}}}} \end{array}} \right] \left[ {\begin{array}{*{20}{c}} {\frac{{\partial c}}{{\partial {{{\bar{p}}}_{e1}}}}}\\ {\frac{{\partial c}}{{\partial {{{\bar{p}}}_{e2}}}}}\\ {\frac{{\partial c}}{{\partial {{{\bar{p}}}_{e3}}}}} \end{array}} \right], \end{aligned} \end{aligned}$$
(29)

where it can be readily found that

$$\begin{aligned} \begin{aligned} \left[ {\begin{array}{*{20}{c}} {\frac{{\partial {{{\bar{p}}}_{e1}}}}{{\partial {{{\tilde{p}}}_{e1}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e2}}}}{{\partial {{{\tilde{p}}}_{e1}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e3}}}}{{\partial {{{\tilde{p}}}_{e1}}}}}\\ {\frac{{\partial {{{\bar{p}}}_{e1}}}}{{\partial {{{\tilde{p}}}_{e2}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e2}}}}{{\partial {{{\tilde{p}}}_{e2}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e3}}}}{{\partial {{{\tilde{p}}}_{e2}}}}}\\ {\frac{{\partial {{{\bar{p}}}_{e1}}}}{{\partial {{\tilde{p}}_{e3}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e2}}}}{{\partial {{\tilde{p}}_{e3}}}}}&{}{\frac{{\partial {{{\bar{p}}}_{e3}}}}{{\partial {{\tilde{p}}_{e3}}}}} \end{array}} \right] = \frac{1}{\left\| {{\tilde{{\varvec{p}}}_e}} \right\| }\left( {{\varvec{I}} - {{\bar{\varvec{p}}}_e}\bar{\varvec{p}}_e^T} \right) \end{aligned}. \end{aligned}$$
(30)

Finally, the sensitivity of the compliance with respect to the original orientation design variables can be given as follows:

$$\begin{aligned} \frac{{\partial c}}{{\partial {p_{ei}}}} = \sum \limits _{j \in {N_e}} {\frac{{\partial c}}{{\partial {{\tilde{p}}_{ji}}}}\frac{{\partial {{{\tilde{p}}}_{ji}}}}{{\partial {p_{ei}}}}}. \end{aligned}$$
(31)

When the sensitivity of the objective and constraint functions with respect to all design variables is ready, the design variables can be updated.

4.2 Decoupled design variables update strategy

In minimum compliance design problems, there is typically only one constraint related to the available amount of material, and the orientation variables do not affect the volume fraction constraint. This characteristic provides an opportunity for efficient design variable updating in each optimization step. Instead of updating all design variables simultaneously using the method of moving asymptotes (MMA) proposed by Svanberg (1987), we update the density variables and orientation variables separately. This approach has been proven to be robust and more efficient, as demonstrated in the work of Zhang et al. (2021).

To update the density variables, we employ the optimality criteria method described in Andreassen et al. (2011). Since the orientation variables are subject to lower and upper bounds rather than general constraints, we adopt an explicit convex approximation for the objective function in each topology optimization step, as proposed by Zhang et al. (2021). This approximation allows us to construct an explicit convex sub-problem:

$$\begin{aligned} \begin{aligned} \mathop {\min }\limits _{{\varvec{p}}}:&f({\varvec{p}}) = {f_1(\varvec{p}_1 )} + {f_2(\varvec{p}_2 )} + {f_3(\varvec{p}_3 )} + {r_0}\\ s.t.:&{p _{e1,l}} \le {p _{e1}} \le {p _{e1,u}},1 \le e \le {N}\\&{p _{e2,l}} \le {p _{e2}} \le {p _{e2,u}},1 \le e \le {N}\\&{p _{e3,l}} \le {p _{e3}} \le {p _{e3,u}},1 \le e \le {N}\\ \end{aligned}, \end{aligned}$$
(32)

where

$$\begin{aligned} {f_i(\varvec{p}_i )} = \sum \limits _{e = 1}^{{N}} {(\frac{{{P_{ei}}}}{{{U_{ei}} - {p_{ei}}}} + \frac{{{Q_{ei}}}}{{{p_{ei}} - {L_{ei}}}})} \end{aligned}$$
(33)

and \({r_0}\) is a constant only dependent on the current values of the design variables. The specific expressions of \({P_{ei}}\), \({Q_{ei}}\), \({U_{ei}},\) and \({L_{ei}}\) can be found in Aage and Lazarov (2013). The purpose of setting lower and upper bounds \({p _{ei,l}}\) and \({p _{ei,u}}\) on the orientation variables is to prevent excessively large step sizes during the optimization process.

The sub-problem formulation allows for the separation of the objective function, enabling the creation of independent sub-problems with only bound constraints. By exploiting this separability, the optimum solution for each sub-problem can be analytically determined as follows:

$$\begin{aligned} p_{ei}^s = \left\{ \begin{aligned}&{p_{ei,l}},{} & {} \text {if } p_{ei}^* < {p_{ei,l}} \\&p_{ei}^*,{} & {} \text {if } p_{ei}^* \in [{p_{ei,l}},{p_{ei,u}}] \\&{p_{ei,u}},{} & {} \text {if } p_{ei}^* > {p_{ei,u}} \end{aligned} \right., \end{aligned}$$
(34)

where

$$\begin{aligned} p_{ei}^* = \frac{{\sqrt{{P_{ei}}} {L_{ei}} + \sqrt{{Q_{ei}}} {U_{ei}}}}{{\sqrt{{P_{ei}}} + \sqrt{{Q_{ei}}} }}. \end{aligned}$$
(35)

4.3 Optimization procedure

The procedure for the topology optimization of 3D continuous fiber-reinforced composites is illustrated in Fig. 2. In each optimization iteration, four steps: calculation of physical design variables, finite element analysis, sensitivity analysis, and design variables updating, are sequentially performed. This iterative process is repeated until the optimization process converges. Once the optimization process has converged, the optimized physical density and orientation variables are output for further post-processing and visualization.

Fig. 2
figure 2

Flowchart for topology optimization of 3D continuous fiber-reinforced composites

5 Numerical examples

This section presents three numerical examples to illustrate the effectiveness of the Cartesian parametrization for topology optimization of 3D continuous fiber-reinforced composites. The material properties of the composite material TFP used in all examples are the same as those specified in Zhou et al. (2022). In all examples, the TSMs-based MGPCG solver developed in the authors’ recent work (Zhao et al. 2024) is utilized for finite element analysis. The move limit for the density variables is set to 0.2, while the move limit \(m_{p}\) for the fiber orientation variables is initially set as 1/4 and decay exponentially at a rate of 0.95 in each optimization iteration. The maximum number of optimization iterations is set to 150.

The computations are performed on a compute node within a high-performance computing cluster, equipped with an Intel(R) Xeon(R) Gold 6240 Processor and an NVIDIA Tesla V100 GPU. The implementation is carried out using the Taichi 1.7 programming language (Hu et al. 2019). For visualizing the density distributions and fiber orientations, ParaView 5.8.1 (Ayachit 2015) is employed. A threshold value of 0.3 is set for the density distribution during the visualization process.

5.1 Case study 1: a 3D cantilever beam design problem

In this example, we consider a 3D cantilever beam design problem to showcase the effectiveness of the proposed method. The design domain and boundary conditions are depicted in Fig. 3. The dimensions of the design domain are 1 m\(\times 2\) m\(\times 1\) m. Part of the left end of the design domain is fixed in all three directions. A force of magnitude 20000 N is uniformly applied on the half-circle of diameter 0.25 m. The prescribed volume of the material is set to be \(10\%\) of the volume of the design domain.

Fig. 3
figure 3

Problem definition for Case study 1

To conduct topology optimization, the design domain is discretized into \(128\times 256\times 128\) tri-linear cubic elements. Three different design initializations are considered, where the fiber orientations in all elements are aligned with the X, Y, and Z directions, respectively. The filtering radii for the density and fiber orientation fields are set to be 1.5 times the element size.

The optimization designs for these three cases using the Cartesian parametrization and the Euler angles-based parametrization (Schmidt et al. 2020; Zhao et al. 2024) are displayed in Figs. 4 and 5, respectively. The compliance values of the optimized designs using Cartesian parametrization are 1.062 J, 1.099 J, and 1.123 J, respectively, while those for the optimized designs obtained using Euler angles-based parametrization are 1.160 J, 1.115 J, and 1.124J, respectively. This indicates that both parametrization methods yield designs with comparable compliance values, but the optimized design obtained by Cartesian parametrization is smaller than that obtained by Euler angles-based parametrization. The differences between the results obtained by the two methods can be partly attributed to the differences in the parametrization methods and the non-convex nature of the objective function when dealing with composite materials.

Fig. 4
figure 4

Optimized designs for Case study 1 obtained using Cartesian parametrization

Fig. 5
figure 5

Optimized designs for Case study 1 obtained using Euler angles-based parametrization

To validate the optimality of the obtained designs, the principal stress directions that minimize elemental strain energy density are computed for all elements. The correlation between the principal stress directions and the optimized fiber orientations is plotted for all elements. As shown in Figs. 4c, f, i, 5c, f, i, the correlation is equal to 1.0 for almost all elements, validating the optimality of the obtained designs (Safonov 2019; Fedulov et al. 2021).

The iteration histories of objective function are plotted in Fig. 6. The most significant reduction in the value of the objective function occurs within the initial 10 iterations, and after around 40 iterations, the improvement becomes marginal. Furthermore, although the optimized designs are quite different from each other, the difference in their corresponding compliances is within \(10\%\). This implies that different initializations lead to variations in the optimized design, but they still achieve comparable levels of compliance performance.

Fig. 6
figure 6

Iteration histories of objective function for Case study 1

To demonstrate the mesh-independence of the filter, this problem is solved again using Cartesian parametrization and three different resolutions: \(32\times 64\times 32\), \(64\times 128\times 64\), and \(128\times 256\times 128\). Correspondingly, the size of both the density and orientation filters is set to be 1.5, 3, and 6 times the element size for three resolutions, respectively. The fibers are initially aligned with the X direction for all elements. The optimized designs are displayed in Fig. 7. The three designs obtained from the different resolutions are quite similar, except that the thin bars are not present in the design obtained with the lowest resolution. After careful inspection, it is found that the thin bars in Fig. 7c have density values between 0.3 and 0.5, which is much lower than 1.0, causing them to disappear in Fig. 7a and b. This can be attributed to the low resolution, which often leads to designs without clear load transfer paths, especially when thin features are present. Additionally, the correlation coefficient is equal to 1.0 for almost all elements in Figs. 7c, f, i, validating the optimality of the obtained designs. However, for the case using a resolution of \(128\times 256\times 128\), the fiber orientation may not be aligned with the principal stress direction in some regions, which can be partly attributed to the multi-axial stress state at these locations and partly to the relatively large filter size.

Fig. 7
figure 7

Mesh independence of Cartesian parametrization for Case study 1

To demonstrate that the Cartesian parametrization works well for large-scale topology optimization problems, four resolutions are considered: \(32\times 64\times 32\), \(64\times 128\times 64\), \(128\times 256\times 128\), and \(256\times 512\times 256\). In the case of \(256\times 512\times 256\), the number of elements is approximately 33.6 million, and the number of design variables is around 134.2 million. For all cases, the filter radius is set to 1.5 times the element size, resulting in potentially different topologies obtained for each mesh size. The design results are presented in Fig. 8, showing that as the resolution increases, thin bars and plates start to appear in the optimized structure. The optimized compliances are plotted in Fig. 9. It is evident that the compliance corresponding to both the original computing grids and re-analyzed using the finest grid (resolution of \(256\times 512\times 256\)) becomes smaller with increasing resolution. This is different from the authors’ previous work (Zhao et al. 2024), where the compliance corresponding to the original computing grids became larger with increasing resolution. The key is that the point load violates fundamental finite element rules as well as physical reality, and mesh-independent boundary condition implementations should be employed (Sigmund 2022). The fact that the designs obtained with higher resolutions exhibit lower compliances again demonstrates the need for large-scale topology optimization. This example also demonstrates the effectiveness of the Cartesian parametrization even for large-scale topology optimization problems.

Fig. 8
figure 8

Optimized compliance obtained using Cartesian parametrization and different resolutions for Case study 1

Fig. 9
figure 9

Optimized compliance obtained using Cartesian parametrization and different resolutions for Case study 1

5.2 Case study 2: a 3D cube design problem

This problem is taken from the study Zhou et al. (2022). As shown in Fig. 10, the design domain is a 1 m\(\times 1\) m\(\times 1\) m cube. On the top face of the structure, two directional distributed loads are simultaneously applied: one is a shear load \(F_S\) and the other is a vertical load \(F_n\). Both loads have a magnitude of 10,000 N. The bottom face of the structure is fixed in all directions at one location and fixed only in the Z direction at the other three locations. The objective of this problem is to showcase the potential of the new method for lightweight design. To achieve this, a prescribed volume fraction of the material is set to \(2\%\).

To perform topology optimization, the design domain is discretized into \(160\times 160\times 160\) tri-linear cubic elements. Three different design initializations are considered, where the fiber orientations in all elements are aligned with the X, Y, and Z directions, respectively. The filtering radii for the density and fiber orientation fields are set to be 1.5 times the element size. The optimized designs for the three cases using the Cartesian parametrization and the Euler angles-based parametrization (Schmidt et al. 2020; Zhao et al. 2024) are displayed in Figs. 11 and 12, respectively. The compliance values of the optimized designs using Cartesian parametrization are 1.0618 J, 0.9705 J, and 0.8605 J, respectively, while those for the optimized designs using Euler angles-based parametrization are 0.9425 J, 1.039 J, and 1.032 J. Interestingly, the optimized design obtained by Cartesian parametrization is smaller than that obtained by Euler angles-based parametrization for the latter two cases, but it is larger than the first case. This is likely due to the non-convexity of the optimization problem. The iteration histories of the objective function for all cases are plotted in Fig. 13. The most considerable reduction in the objective function value occurs within the initial 10 iterations. After approximately 40 iterations, the improvement in the objective function value becomes negligible.

Fig. 10
figure 10

Problem definition for Case study 2

Fig. 11
figure 11

Optimized designs for Case study 2 obtained using Cartesian parametrization

Fig. 12
figure 12

Optimized designs for Case study 2 obtained using Euler angles-based parametrization

These two examples show that Cartesian parametrization usually leads to better designs than the Euler angles-based parametrization. Additionally, the transformation between the Cartesian representation and Euler angles-based representation (Schmidt et al. 2020; Luo et al. 2023; Zhao et al. 2024) is avoided in the proposed Cartesian parametrization method, which is another notable advantage.

Fig. 13
figure 13

Iteration histories of objective function for Case study 2

5.3 Case study 3: jet engine bracket design problem

The jet engine bracket design problem from the GrabCAD challenge is tested here to demonstrate the potential of the proposed method for the design of engineering structures. The design domain, non-design domain, as well as displacement and load boundary conditions are shown in Fig. 14. Different from the requirement of the GrabCAD challenge, where 6 load cases should be considered, only one load case is considered in this example. The load has a magnitude of 36288 N and it is applied upwards along the Z direction. The six cylinders in orange indicate non-design domains for density, but the fiber orientations in them can be varied. The inner surfaces of the four bottom cylinders are fixed in all directions. To achieve this, a prescribed volume fraction of the material for the design domain is set to \(10\%\).

Fig. 14
figure 14

Problem definition for Case study 3

To perform topology optimization, a grid of resolution \(448\times 320\times 192\) is employed for finite element analysis, and the displacement and load boundary conditions are applied according to the method presented in Eom et al. (2022). Three different design initializations are considered, where the fiber orientations in all elements are aligned with the X, Y, and Z directions, respectively. The filtering radii for the density and fiber orientation fields are set to be 1.5 times the element size.

The optimized designs for the three cases using the Cartesian parametrization are displayed in Fig. 15. The compliance values of the optimized designs are 13.969 J, 15.869 J, and 13.033 J, respectively. The iteration histories of the objective function for all cases are plotted in Fig. 16. It can be observed again that the most considerable reduction in the objective function value occurs within the initial 10 iterations. After approximately 40 iterations, the improvement in the objective function value becomes little.

Fig. 15
figure 15

Optimized designs for Case study 3 obtained using Cartesian parametrization

Fig. 16
figure 16

Iteration histories of objective function for Case study 3

6 Conclusion

This study revisits the Cartesian representation-based parametrization for 3D fiber orientation and establishes the the direct mapping between the Cartesian representation of the 3D fiber orientation and corresponding stiffness matrix explicitly. The proposed method employs filters to regularize fiber orientations and normalization to ensure unit length of the fiber direction vector. The proposed parametrization avoids the use of Euler angles, and thus, it is singularity free and simpler to be implemented.

The proposed Cartesian parametrization is then integrated into the problem formulation of topology optimization for 3D continuous fiber-reinforced composites. The proposed parametrization method is straightforward and does not introduce any general nonlinear constraints, which facilitates the design variable updating strategy. Three numerical examples are presented to demonstrate the effectiveness of the proposed method. It is observed that the proposed Cartesian parametrization achieves comparable or better objective function values compared to the conventional Euler angles-based parametrization. Overall, the proposed parametrization method proves to be effective in topology optimization of composite structures with spatially varying fiber paths, offering simplicity and comparable or better results to existing methods.

The Cartesian parametrization is designed based on the important fact that fiber-reinforced composite materials are transversally isotropic. Thus, it is not applicable for general 3D orthogonal materials because, for these materials, the orientation in the transverse plane is also important. For such problems, the quaternions representation or vector-axis representation may be suitable choices.