1 Introduction

Fiber-reinforced materials have been on the rise due to their advantageous characteristics, such as a high strength-to-mass ratio. Consequently, there has been a surge in the development of new additive manufacturing technologies specifically designed for such materials [1,2,3]. This aspect has captured the attention of the scientific community, prompting efforts to identify optimal orientations for fibers. Consequently, numerous studies have surfaced, presenting innovative approaches for optimizing the orientation of fibers in the literature.

In optimizing fiber orientation, some works leverage heuristic algorithms independent of gradient calculations and claim that the solutions are "global minima." Examples include applications in minimizing the mass of laminate composites and optimizing fiber orientation and thickness, as demonstrated by Kim et al. [4] and António [5]. Nevertheless, Sigmund [6] raises concerns about the efficiency of non-gradient methodologies, particularly in situations involving numerous design variables.

Soares et al. [7] and Luo and Hae [8] used a gradient-based approach for optimizing fiber orientations, but direct consideration of angles as design variables led to challenges. Stegmann and Lund [9] proposed a material interpolation model, Bruyneel [10] introduced an interpolation model using finite element shape functions, and Gao et al. [11] generalized it. Kiyono et al. [12] used normal distribution functions to address the issue of increasing design variables. Salas et al. [13] avoided candidate angles, approximating sine and cosine functions with Taylor series. In a hybrid approach, Salas et al. [14] combined Taylor series approximation with the candidate angles method.

Some works address the optimization of continuous fiber orientation for additive manufacturing. Kato and Ramm [15] present an optimization strategy aiming to improve structural ductility through the simultaneous determination of fiber layouts and global geometry, employing a combination of multiphase material optimization and material shape optimization methods. Boddeti et al. [16] introduce a workflow for designing laminated continuous fiber-reinforced composites with variable stiffness, leveraging multiscale topology optimization and voxel-based multimaterial jetting. Fedulov et al. [17] present a topology optimization framework for continuous composite fiber 3D printing, minimizing compliance with density and orientation design parameters. Zhang et al. [18] present a topology optimization framework for continuous fiber-reinforced composites in 3D printing, incorporating strength constraints and utilizing topology, fiber volume fraction, and fiber orientation as design variables.

However, to the best of the authors’ knowledge, only three recent works address the critical issue of stress constraints in structural topology optimization applied to materials reinforced with fibers. One of these studies is the previously mentioned work by Zhang et al. [18], in which the authors utilize the p-norm, a global approach, to restrict the maximum strength. In a previous study [19], fibers were optimized in each element of the mesh considering the Tsai–Hill yield criteria. Another work by Kundu and Zhang [20] explores topology results where each member has a specific fiber angle, and the Tsai–Wu yield criteria are considered. The Tsai–Wu yield criteria are considered more comprehensive than the Tsai–Hill as they account for different behaviors in tension and compression. These characteristics demand approaches different from those used to solve similar problems using the same interpolation material models. In this work, we formulate the stress constraints based on the Tsai–Wu yield criteria. For optimizing the fiber angles, we employ the NDFO-adapt method [19], and for distributing the matrix, we use the well-known SIMP material interpolation model [21]. The optimization process does not solely focus on the material and fibers; it also involves optimizing the \(\beta \) variable of the threshold projection and the penalization parameters for both interpolation material models (NDFO-adapt and SIMP). Although this approach was used in a previous work [19], it is not possible to solve the problem for Tsai–Wu using the same functions that control the continuations in the constraints. So, we are proposing new functions to control these continuations essential to solving the problem for Tsai–Wu yield criteria using the proposed method.

The structure of this paper is as follows. Section 2 presents and discusses the equations of solid mechanics considering linear elasticity, the material models for fiber orientation and material distribution, the topology optimization problem, the augmented Lagrangian version of the optimization problem, and the calculation of sensitivities. Section 3 introduces the finite element mesh refinement, spatial filter, and the optimization procedure. Section 4 showcases the results obtained through the proposed method. Some conclusions are presented in Sect. 5.

2 Theoretical formulation

The optimized distribution for the design variables is performed in a domain \(\Omega \), which is often fixed. The schematic representation in Fig. 1 illustrates the \(\Omega \) domain and its boundaries denoted as \(\partial \Omega \). The boundary conditions of Dirichlet and Neumann are applied in the subdomains \(\Gamma _u\) \(\subset \) \(\partial \Omega \) and \(\Gamma _t\) \(\subset \) \(\partial \Omega \), respectively. Vectors \(\textbf{t}\) and \(\textbf{b}\) symbolize surface and body forces, respectively. The illustration’s purple lines depict randomly oriented fibers within \(\Omega \).

Fig. 1
figure 1

Generic design domain

This study’s theoretical framework operates within the confines of linear elastic behavior. The solution involves determining the field equations of solid mechanics to address the Topology Optimization (TO) problem on domain \(\Omega \). These equations also termed the forward or state equations in the context of Topology Optimization (TO) are represented in their variational form as outlined by [22, 23]:

$$\begin{aligned} \begin{aligned} a(\textbf{u,v}) - L(\textbf{v}) = 0 \end{aligned} \end{aligned}$$
(1)

where \(\textbf{u}\) and \(\textbf{v}\) are the displacements and the virtual displacements, respectively. The functional \(a\)(\(\textbf{u, v}\)) represents the Energy bilinear form, and the functional \(L\)(\(\textbf{v}\)) represents the Load linear form as defined in [22].

$$\begin{aligned} \begin{aligned} a(\textbf{u, v}) = \int _\Omega \sigma _{ij}(\textbf{u}) \; \varepsilon _{ij} (\textbf{v}) \; \textrm{d}\Omega \end{aligned} \end{aligned}$$
(2a)
$$\begin{aligned} \begin{aligned} L(\textbf{v}) = \int _\Omega b_i \; v_i \; \textrm{d}\Omega + \int _{\Gamma _t}t_i \; v_i \; \textrm{d}S \end{aligned} \end{aligned}$$
(2b)

where \(b_i\) is the \(i^{th}\) vector which represents the body forces, \(t_i\) is the \(i^{th}\) component of the vector of surface forces, and \(v_i\) is the \(i^{th}\) component of the virtual displacement vector. The Cauchy stress tensor, denoted by \({\varvec{\sigma }}\), is computed using the strain tensor \({\varvec{\varepsilon }}\), as described in [23].

$$\begin{aligned} \begin{aligned} \sigma _{ij} = C_{ijkl} \, \varepsilon _{kl}(\textbf{u}) \end{aligned} \end{aligned}$$
(3)
$$\begin{aligned} \begin{aligned} \varepsilon _{ij} = \frac{1}{2} \left( \frac{\partial u_i}{\partial x_j} + \frac{\partial u_j}{\partial x_i} \right) \end{aligned} \end{aligned}$$
(4)

In Eq. (3), the term \(C_{ijkl}\) represents the components of \(\textbf{C}\)(\(\phi \)), which represents the constitutive tensor that plays a crucial role in optimization due to its dependency on the fiber angles \(\phi \). The constitutive relationship is formulated based on Eq. (3) to express the components of \(\textbf{C}\)(\(\phi \)). The constitutive relationship, under the assumptions of orthotropic material, for the plane stress formulation, is presented in Voigt notation [24]:

$$\begin{aligned} \begin{aligned} \begin{bmatrix} \varepsilon _{1} \\ \varepsilon _{2} \\ 2\, \varepsilon _{6} \\ \end{bmatrix} = \begin{bmatrix} S_{11} &{} S_{12} &{} 0 \\ S_{12} &{} S_{22} &{} 0 \\ 0 &{} 0 &{} S_{66} \\ \end{bmatrix} \begin{bmatrix} \sigma _1 \\ \sigma _2 \\ \sigma _6 \\ \end{bmatrix} \end{aligned} \end{aligned}$$
(5)

In Eq. (5), \(\textbf{S}\) is called compliance tensor. Despite representing the plane stress, the formulation retains the three-dimensional index, hence the appearance of an index 6. The tensor \(\textbf{C}\) for plane stress is derived by inverting Eq. (5), resulting in:

$$\begin{aligned} \begin{aligned} \begin{bmatrix} \sigma _1 \\ \sigma _2 \\ \sigma _6 \\ \end{bmatrix}= \begin{bmatrix} Q_{11} &{} Q_{12} &{} 0 \\ Q_{12} &{} Q_{22} &{} 0 \\ 0 &{} 0 &{} Q_{66} \\ \end{bmatrix} \begin{bmatrix} \varepsilon _{1} \\ \varepsilon _{2} \\ 2\, \varepsilon _{6} \\ \end{bmatrix} \end{aligned} \end{aligned}$$
(6)

where \(Q_{ij}\) denotes the components of the constitutive tensor \(\textbf{Q}\). It is crucial to emphasize that the components of the constitutive tensor \(\textbf{C}\) are not the same in 3D and in plane stress. Consequently, a distinct symbol \(\textbf{Q}\) is employed for the plane stress situation. \(Q_{ij}\) are computed as follows:

$$\begin{aligned} \begin{aligned} Q_{11} = \frac{S_{22}}{S_{11}\,S_{22} - S_{12}^2} \end{aligned} \end{aligned}$$
(7a)
$$\begin{aligned} \begin{aligned} Q_{12} = -\frac{S_{12}}{S_{11}\,S_{22} - S_{12}^2} \end{aligned} \end{aligned}$$
(7b)
$$\begin{aligned} \begin{aligned} Q_{22} = \frac{S_{11}}{S_{11}\,S_{22} - S_{12}^2} \end{aligned} \end{aligned}$$
(7c)
$$\begin{aligned} \begin{aligned} Q_{66} = \frac{1}{S_{66}} \end{aligned} \end{aligned}$$
(7d)

The components of the \(\textbf{S}\) are:

$$\begin{aligned} \begin{aligned} S_{11} = \frac{1}{E_1} \end{aligned} \end{aligned}$$
(8a)
$$\begin{aligned} \begin{aligned} S_{12} = \frac{-\nu _{12}}{E_1} \end{aligned} \end{aligned}$$
(8b)
$$\begin{aligned} \begin{aligned} S_{22} = \frac{1}{E_2} \end{aligned} \end{aligned}$$
(8c)
$$\begin{aligned} \begin{aligned} S_{66} = \frac{1}{B_{12}} \end{aligned} \end{aligned}$$
(8d)

where \(E_1\) and \(E_2\) represent Young’s modulus in the fiber direction and perpendicular to the fiber direction, respectively, \(\nu _{12}\) represents the Poisson ratio considering an extension in direction 1 and a contraction in direction 2, and \(B_{12}\) is the shear modulus in plane 1–2.

The local constitutive tensor \(\textbf{Q}\) is defined in Eq. (6). However, global coordinates are necessary for fiber angle optimization. Figure 2 depicts a fiber within a generic domain. Here, global coordinates (x, y) are distinguished from local(1,2), axis 1 aligns with the fiber, and the \(\tilde{\tilde{\phi }}\) represents the physical fiber angle. The physical fiber angle is determined as a function of the filtered angle \(\tilde{\phi }\), which in turn is dependent on the design variable for the fiber angle, \(\phi \), that is, \(\tilde{\tilde{\phi }}\) = \(\tilde{\tilde{\phi }}\)(\(\tilde{\phi }\)(\(\phi \))).

Equations (6) and (7) represent the stress–strain relationship in local coordinates. To optimize the fiber angles, it is essential to express this relationship in global coordinates (x, y), as illustrated in Fig. 2.

Fig. 2
figure 2

Local and global axes of a fiber

Under plane stress conditions, the mapping from local stresses to global stresses is established through the transformation tensor \(\textbf{T}\):

$$\begin{aligned} \begin{aligned} \begin{bmatrix} \sigma _x \\ \sigma _y \\ \sigma _{xy} \\ \end{bmatrix} = \textbf{T}^{-1} \begin{bmatrix} \sigma _1 \\ \sigma _2 \\ \sigma _6 \\ \end{bmatrix} \end{aligned} \end{aligned}$$
(9)

where \(\textbf{T}\) is defined as [24]:

$$\begin{aligned} \begin{aligned} \textbf{T}^{-1} = \begin{bmatrix} cos^2(\tilde{\tilde{\phi }}) &{} sin^2(\tilde{\tilde{\phi }}) &{} -2\,sin(\tilde{\tilde{\phi }})\;cos(\tilde{\tilde{\phi }}) \\ sin^2(\tilde{\tilde{\phi }}) &{} cos^2(\tilde{\tilde{\phi }}) &{} 2\,sin(\tilde{\tilde{\phi }})\;cos(\tilde{\tilde{\phi }}) \\ sin(\tilde{\tilde{\phi }}) \, cos(\tilde{\tilde{\phi }}) &{} - sin(\tilde{\tilde{\phi }}) \, cos(\tilde{\tilde{\phi }}) &{} cos^2(\tilde{\tilde{\phi }}) - sin^2(\tilde{\tilde{\phi }}) \end{bmatrix} \end{aligned} \end{aligned}$$
(10a)
$$\begin{aligned} \begin{aligned} \textbf{T} = \begin{bmatrix} cos^2(\tilde{\tilde{\phi }}) &{} sin^2(\tilde{\tilde{\phi }}) &{} 2\,sin(\tilde{\tilde{\phi }})\;cos(\tilde{\tilde{\phi }}) \\ sin^2(\tilde{\tilde{\phi }}) &{} cos^2(\tilde{\tilde{\phi }}) &{} -2\,sin(\tilde{\tilde{\phi }})\;cos(\tilde{\tilde{\phi }}) \\ -sin(\tilde{\tilde{\phi }}) \, cos(\tilde{\tilde{\phi }}) &{} sin(\tilde{\tilde{\phi }}) \, cos(\tilde{\tilde{\phi }}) &{} cos^2(\tilde{\tilde{\phi }}) - sin^2(\tilde{\tilde{\phi }}) \end{bmatrix} \end{aligned} \end{aligned}$$
(10b)

The stress–strain relationship (Eq. (3)) with stresses in global coordinates can be expressed as follows:

$$\begin{aligned} \begin{aligned} \begin{bmatrix} \sigma _x \\ \sigma _y \\ \sigma _{xy} \\ \end{bmatrix}= \textbf{T}^{-1} \; \textbf{Q} \begin{bmatrix} \varepsilon _1 \\ \varepsilon _2 \\ 2\, \varepsilon _6 \\ \end{bmatrix} \end{aligned} \end{aligned}$$
(11)

The mapping between the local strains and global strains can similarly be carried out by \(\textbf{T}\), as indicated by:

$$\begin{aligned} \begin{aligned} \begin{bmatrix} \varepsilon _1 \\ \varepsilon _2 \\ \varepsilon _6 \\ \end{bmatrix} = \textbf{R} \; \textbf{T} \; \textbf{R}^{-1} \begin{bmatrix} \varepsilon _x \\ \varepsilon _y \\ \varepsilon _{xy} \\ \end{bmatrix} \end{aligned} \end{aligned}$$
(12)

where the Reuter matrix \(\textbf{R}\) is calculated as follows:

$$\begin{aligned} \begin{aligned} \textbf{R} = \begin{bmatrix} 1 &{}\quad 0 &{}\quad 0 \\ 0 &{}\quad 1 &{} \quad 0 \\ 0 &{}\quad 0 &{}\quad 2 \\ \end{bmatrix} \end{aligned} \end{aligned}$$
(13)

In conclusion, the global constitutive equation can be defined by incorporating the fiber angle, as presented in [24]:

$$\begin{aligned} \begin{aligned} \begin{bmatrix} \sigma _x \\ \sigma _y \\ \sigma _{xy} \\ \end{bmatrix} = \textbf{T}^{-1} \; \textbf{Q} \; \textbf{R} \; \textbf{T} \; \textbf{R}^{-1} \begin{bmatrix} \varepsilon _x \\ \varepsilon _y \\ \varepsilon _{xy} \\ \end{bmatrix} \end{aligned} \end{aligned}$$
(14)

\(\textbf{C}\)(\(\phi \)) for orthotropic materials under plane stress can be determined through Eq. (14).

$$\begin{aligned} \begin{aligned} \textbf{C}(\tilde{\tilde{\phi }}) = \left( \textbf{T}(\tilde{\tilde{\phi }})\right) ^{-1} \; \textbf{Q} \; \textbf{R} \; \textbf{T}(\tilde{\tilde{\phi }}) \; \textbf{R}^{-1} \end{aligned} \end{aligned}$$
(15)

2.1 Material models

In this paper, the material model Solid Isotropic Material with Penalization (SIMP) [21] is used to determine the material distribution, while the Normal Distribution Fiber Optimization adaptive (NDFO-adapt) deals with the fiber angle optimization. The penalization factors are treated as design variables in both models, dynamically determined through the optimization steps.

2.1.1 SIMP

The SIMP interpolation material model employs a pseudo-density \(\rho \) to modulate the tensor \(\textbf{C}\) resulting in the modified constitutive equation:

$$\begin{aligned} \begin{aligned} \sigma _{ij} = \left( \rho _{\min } + (1 - \rho _{\min }) \tilde{\tilde{\rho }}^{\tilde{p}}\right) \; C_{ijkl} \; \varepsilon _{kl}(\textbf{u}) \end{aligned} \end{aligned}$$
(16)

With a minimum pseudo-density value of \(\rho _{\min }\), the physical pseudo-densities are denoted by \(\tilde{\tilde{\rho }}\)\((\textbf{x})\) and \(\tilde{p}\) represented the field of filtered penalization for the SIMP, as introduced in [19]. The objective of \(\tilde{p}\) is to drive all \(\tilde{\tilde{\rho }}\) values to either 0 or 1, with 1 representing material-filled regions and 0 representing voids. When \(\tilde{p}\) equals 1, the mapping between \(\tilde{\tilde{\rho }}\) and the SIMP is linear. Notwithstanding, for higher values of \(\tilde{p}\), the mapping approaches a 0-1 relationship.

2.1.2 NDFO-adapt material model

Normal Distribution Fiber Optimization adaptive (NDFO-adapt) (Normal Distribution Fiber Optimization adaptive) interpolation material model, introduced in [19], is responsible for optimizing fiber angles. This method employs a weighted sum of candidate angles to determine fiber orientations. The candidate angle strategy, proposed by Stegmann and Lund [9] in their Discrete Material Optimization (DMO) method, effectively addresses the prevalent challenge of multiple local minima associated with Continuous Fiber Angle Optimization (CFAO). Both Discrete Material Optimization (DMO) and Normal Distribution Fiber Optimization adaptive (NDFO-adapt) leverage candidate angles to navigate through these minima, enhancing optimization robustness.

The primary advantage of Normal Distribution Fiber Optimization (NDFO)-based approaches lies in the requirement of only one design variable in each point of interest \((e)\), regardless of the number of candidate angles.

A weighted summation is used to determine the \(\tilde{\tilde{\phi }}\) that represents the physical fiber angles [14]

$$\begin{aligned} \begin{aligned} \tilde{\tilde{\phi }} = \sum _{i = 1}^{N_c}\, w_i\, \phi ^c_i \end{aligned} \end{aligned}$$
(17)

With \({N_c}\) fiber candidate angles, \({{\phi _c}}\) representing the candidate angles, and w denoting the following weight function:

$$\begin{aligned} \begin{aligned} w_i = \frac{\hat{w}_i}{\sum _{j=1}^{N_c}\,\hat{w}_j} \end{aligned} \end{aligned}$$
(18)

\(\hat{w}\) represents a normal distribution function that is calculated as Eq. (18):

$$\begin{aligned} \begin{aligned} \hat{w}_i = \exp \left( {-\frac{(\tilde{\phi }(\textbf{x}) - \phi ^c_i)^2}{2 \; \left( \tilde{p}_n(\textbf{x}) \right) ^2}}\right) \end{aligned} \end{aligned}$$
(19)

The fields of filtered penalizations and filtered fiber angles for the Normal Distribution Fiber Optimization adaptive (NDFO-adapt) are denoted by \(\tilde{p}_n(\textbf{x})\) and \(\tilde{\phi }\)\((\textbf{x})\), respectively, as introduced in Eq. (19).

The idea of using the function \(\hat{w}\) is to apply a continuation in the design variable \(p_n\), which represents the penalization, because the smaller the \(p_n\) value, the closer to angles candidates \(\phi ^c\) will be the variable \(\tilde{\phi }\). This behavior is expected since the \(\tilde{p}_n\) field is responsible for penalizing the interpolation material model, and as its value decreases, the penalization effect becomes more significant. On the other hand, when continuous variable \(\tilde{\phi }\) aligns with a candidate angle \(\phi ^c\), the corresponding \(\hat{w}\) is equal to 1. Conversely, as the variable moves away from the candidate angle, the values of normal distribution function \(\hat{w}\) progressively approach 0.

2.1.3 Threshold projection

It is expected that the results obtained from the Topology Optimization (TO) have a \(\tilde{\tilde{\rho }}\) field consisting exclusively of 0 and 1. However, in certain cases, the result may exhibit values between 0 and 1 that lack clear physical interpretation. To address this, we employ a tanh-based threshold projection as defined in [25,26,27,28]:

$$\begin{aligned} \begin{aligned} \tilde{\tilde{\rho }}(\textbf{x}) = \frac{tanh\left( \tilde{\beta }(\textbf{x}\right) \, \eta ) + tanh\left( \tilde{\beta }\left( \textbf{x})(\tilde{\rho }(\textbf{x}) - \eta \right) \right) }{tanh\left( \tilde{\beta }(\textbf{x}\right) \, \eta ) + tanh\left( \tilde{\beta }(\textbf{x})(1 - \eta )\right) } \end{aligned} \end{aligned}$$
(20)

The variable \(\eta \) in Eq. (20) controls where the point of inflection will occur, and \(\tilde{\beta }\) is the filtered variable that controls the range of the threshold.

Examining Eq. (20), it becomes evident that when \(\tilde{\beta }\) is set to 1, the behavior is almost linear, indicating a mapping that closely aligns \(\tilde{\tilde{\rho }}\) with \(\tilde{\rho }\). Nevertheless, when \(\tilde{\beta }\) is large enough, every \(\tilde{\rho }\) value converges to 0 or 1. Specifically, with a substantial \(\tilde{\beta }\), any \(\tilde{\rho }\) less than \(\eta \) undergoes projection to \(\tilde{\tilde{\rho }}\) = 0, while any \(\tilde{\rho }\) greater than \(\eta \) is projected to \(\tilde{\tilde{\rho }}\) = 1.

2.2 Optimization problem

In this study, the optimization problem aims to minimize the structure volume while considering stress and compliance constraints, as outlined in:

$$\begin{aligned} \begin{aligned} \underset{\rho , \phi , p, p_n, \beta }{{\min }}&\qquad J = \frac{\displaystyle \int _\Omega \tilde{\tilde{\rho }} \; \textrm{d}\Omega }{V} \\ \text {such that}&\qquad F = a(\textbf{u}, \textbf{v}, \tilde{\tilde{\rho }}, \tilde{\tilde{\phi }}) - L(\textbf{v}) = 0 \\&\qquad G_1^{(e)} = f_{\sigma }^{(e)}\; \sigma _{tw}^{(e)} \left( \left( f_{\sigma }^{(e)}\; \sigma _{tw}^{(e)} \right) ^2 + 1 \right) \le 0 \\&\qquad G_2 = \left( \frac{c(\textbf{u}, \tilde{\tilde{\rho }}, \tilde{\tilde{\phi }})}{\alpha _c \; c_{full}(\textbf{u}, \phi _{princ})} - 1\right) \left( \left( \frac{c(\textbf{u}, \tilde{\tilde{\rho }}, \tilde{\tilde{\phi }})}{\alpha _c \; c_{full}(\textbf{u}, \phi _{princ})} - 1\right) ^2 + 1\right) \le 0 \\&\qquad \rho _{\min } \le \rho _{lb}(g, g_{gl}) \le \rho (\textbf{x}) \le \rho _{ub}(g, g_{gl}) \le 1 \\&\qquad \phi _{\min } \le \phi (\textbf{x}) \le \phi _{\max }\\&\qquad p_{\min }(\textbf{x}, g) \le p(\textbf{x}) \le p_{\max }\\&\qquad p_{n_{\min }}(realmin, \phi ^c) \le p_{n_{lb}}(k) \le p_{n}(\textbf{x}) \le p_{n_{\max }}(k)\\&\qquad \beta _{\min }(\textbf{x}, g) \le \beta (\textbf{x}) \le \beta _{\max }\\ \end{aligned} \end{aligned}$$
(21)

where \(J\) represents the objective function, \(F\) denotes the forward problem, \(G_1\)\(^{(e)}\) represents the stress constraint at a designated point \((e)\), \(G_2\) stands for the compliance constraint, and the remaining constraints are box constraints. \(V\) represents the volume of the domain. The energy bilinear form, unlike Eq. (2a), becomes a function of \(\tilde{\tilde{\rho }}\), and \(\tilde{\tilde{\phi }}\).

In the scope of this study, the Tsai–Wu yield criterion is used for stress constraint calculations formulated as per [24]:

$$\begin{aligned} \begin{aligned} H_1\,\sigma _1\,+\,H_2\,\sigma _2\,+\,H_6\,\sigma _6\,+\,H_{11}\,\sigma _1^2\,+\,H_{22}\,\sigma _2^2\,+\,H_{66}\,\sigma _6^2\,+\,2\,H_{12}\,\sigma _1\,\sigma _2 - 1 < 0 \end{aligned} \end{aligned}$$
(22)

where \(H_{[\cdot ]}\) are auxiliary variables to calculate the Tsai–Wu yield criterion, defined as [24]:

$$\begin{aligned}&H_\alpha = \displaystyle \frac{1}{(\sigma _\alpha ^T)_{ult}} - \frac{1}{(\sigma _\alpha ^C)_{ult}} \end{aligned}$$
(23a)
$$\begin{aligned}&H_{\alpha \alpha } = \displaystyle \frac{1}{(\sigma _\alpha ^T)_{ult} \,(\sigma _\alpha ^C)_{ult}} \qquad \alpha = 1, 2 \end{aligned}$$
(23b)
$$\begin{aligned}&H_6 = 0 \end{aligned}$$
(23c)
$$\begin{aligned}&H_{66} = \displaystyle \frac{1}{(\sigma _6)^2_{ult}} \end{aligned}$$
(23d)
$$\begin{aligned}&H_{12} = -\displaystyle \frac{1}{2} \sqrt{\frac{1}{(\sigma _1^T)_{ult}(\sigma _1^C)_{ult}(\sigma _2^T)_{ult}(\sigma _2^C)_{ult}}} \end{aligned}$$
(23e)

In Eq. (23), \((\sigma _1^T)_{ult}\) represents the ultimate longitudinal tensile strength, \((\sigma _1^C)_{ult}\) is the ultimate longitudinal compressive strength, \((\sigma _2^T)_{ult}\) is the ultimate transverse tensile strength, \(\left( \sigma _2^C\right) _{ult}\) is the ultimate transverse compressive strength, and \(\left( \sigma _6\right) _{ult}\) denotes the ultimate in-plane shear strength.

The left-hand side of the Tsai–Wu criterion is used to define \(G_1\) in each point of interesting represented by \((e)\)

$$\begin{aligned} \begin{aligned} \sigma _{tw} = H_1\,\sigma _1\,+\,H_2\,\sigma _2\,+\,H_6\,\sigma _6\,+\,H_{11}\,\sigma _1^2\,+\,H_{22}\,\sigma _2^2\,+\,H_{66}\,\sigma _6^2\,+\,2\,H_{12}\,\sigma _1\,\sigma _2 - 1 \end{aligned} \end{aligned}$$
(24)

The function \(f_{\sigma }\) is introduced to deal with the singularity phenomenon [29]:

$$\begin{aligned} \begin{aligned} f_{\sigma }^{(e)} = \frac{\tilde{\tilde{\rho }}^{(e)}}{\epsilon \left( 1 - \tilde{\tilde{\rho }}^{(e)} \right) + \tilde{\tilde{\rho }}^{(e)}} \end{aligned} \end{aligned}$$
(25)

where \(\epsilon \) = \(\epsilon \) \((\tilde{\tilde{\rho }})\) represents the relaxation parameter. Pereira et al. [30] suggest that smaller \(\epsilon \) values lead to better-defined results. Nevertheless, as noted by Silva et al. [31], excessively small \(\epsilon \) values may result in overestimated stresses at jagged edges, causing strong stress oscillations that impede achieving robust solutions [31]. Silva et al. in their extensive examination [31] investigated the impact of the \(\epsilon \) in Topology Optimization (TO) problems. They discovered that when employing the SIMP with penalization \(p\) set to 3, a favorable balance between the stress accuracy and the minimum stress oscillation is achieved by setting \(\epsilon \) to 0.2. In our current study, we adopt the polynomial function introduced by [19]:

$$\begin{aligned} \begin{aligned} \epsilon (\tilde{\tilde{\rho }}) = \frac{18}{25} \; \tilde{\tilde{\rho }}^2 - \frac{18}{25} \; \tilde{\tilde{\rho }} + 0.2 \end{aligned} \end{aligned}$$
(26)

The compliance constraint, inspired by Bruggi and Duysinx [32], is extended to incorporate fiber contributions. Equation (27) is used to calculate the compliance as a function of \(\tilde{\tilde{\rho }}\) and \(\tilde{\tilde{\phi }}\) in the current iteration of the optimization process. Meanwhile, \(c_{full}\) represents the compliance assuming that \(\tilde{\tilde{\rho }}\) is equal to 1 in whole domain, and aligning the fiber angles with the principal stress direction \(\phi _{princ}\) (refer to Eq. (28)). The parameter \(\alpha _c\) in the compliance constraint acts as a scalar weight for \(c_{full}\).

$$\begin{aligned} \begin{aligned} c(\textbf{u}, \tilde{\tilde{\rho }}, \tilde{\tilde{\phi }}) = \int _{\Omega } \sigma _{ij}(\textbf{u}, \tilde{\tilde{\rho }}, \tilde{\tilde{\phi }}) \; {\varepsilon _{ij} (\textbf{u})} \; \partial \Omega \end{aligned} \end{aligned}$$
(27)
$$\begin{aligned} \begin{aligned} c_{full}(\textbf{u}, \phi _{princ}) = \int _{\Omega } \sigma _{ij}(\textbf{u}, \phi _{princ}) \; {\varepsilon _{ij} (\textbf{u})} \; \partial \Omega \end{aligned} \end{aligned}$$
(28)

In Eq. (21), the parameters \(\rho _{\min }\), \(\phi _{\min }\), \(p_{\min }\), \(p_{n_{\min }}\), and \(\beta _{\min }\) represent the inferior limit for each design variable, while \(\phi _{\max }\), \(p_{\max }\), \(p_{n_{\max }}\), and \(\beta _{\max }\) denote the superior bounds.

The same scheme proposed in [19] is used to determine the lower and upper bounds for the variable \(\rho \)

$$\begin{aligned} \rho _{lb}^{{(e)}^{k+1}} = \max \left( \rho ^{(e)^{k}} - \Delta _{\rho }^{{(e)}^{k+1}}, \,\rho _{\min } \right) \end{aligned}$$
(29a)
$$\begin{aligned} \rho _{ub}^{{(e)}^{k+1}} = \min \left( \rho ^{(e)^{k}} + \Delta _{\rho }^{{(e)}^{k+1}}, \, 1 \right) \end{aligned}$$
(29b)

where \(\Delta _{\rho }\) is an increment calculated as:

$$\begin{aligned} \begin{aligned} \Delta _{\rho }^{{(e)}^{k+1}} = \max \left( g^{(e)^k} \left( g_{gl}^k \right) ^2 {\mathfrak {m}}_\rho , \, {\mathfrak {m}}_{\rho , {\min }}\right) \end{aligned} \end{aligned}$$
(30)

where \({\mathfrak {m}}_\rho \) and \({\mathfrak {m}}_{\rho , {\min }}\) are referred to as the move limit and minimum move limit for \(\rho \), respectively. The amount of gray \(g\) is calculated according to:

$$\begin{aligned} \begin{aligned} g^{(e)} = 4 \; \tilde{\tilde{\rho }}^{(e)} \, \left( 1 - \tilde{\tilde{\rho }}^{(e)}\right) \end{aligned} \end{aligned}$$
(31)

The variable \(g_{gl}\) represents the global amount of gray for the structure, calculated according to the method proposed by Sigmund [28]:

$$\begin{aligned} \begin{aligned} g_{gl} = 4 \; \frac{\sum _{(e)=1}^{N_e} \; \tilde{\tilde{\rho }}^{(e)} \, \left( 1 - \tilde{\tilde{\rho }}^{(e)}\right) }{N_e} \end{aligned} \end{aligned}$$
(32)

where the parameter \(N_e\) denotes the number of points of interest in \(\Omega \). It is noteworthy that for \(\tilde{\tilde{\rho }}\) equal to 0 or 1, the gray measure is equal to zero. In contrast, the maximum contribution occurs when \(\tilde{\tilde{\rho }}\) equals 0.5, representing the grayest region. \(p_{n_{lb}}\) represent the lower bound for the penalization term \(p_n\) at the current iteration. Additionally, \(p_{n_{lb}}\)(k) and \(p_{n_{\max }}\)(k) are dynamically adjusted based on the current iteration (k), illustrating the application of a continuation strategy. In practical terms, as the optimization unfolds, the values of \(p_{n_{lb}}\)(k) decrease progressively through the optimization process, following a predefined function until a certain threshold is reached. This systematic reduction is crucial to guarantee that \(\tilde{p}_n\) becomes sufficiently small, facilitating the mapping of all \(\tilde{\phi }\) to a candidate angles \(\phi ^c\). The continuation process for \(p_{n_{lb}}\) involves iteratively decreasing its value until it reaches a predefined minimum, guided by the following expression:

$$\begin{aligned} \begin{aligned} {p^{k+1}_{n_{lb}}} = \max ({p^{k}_{n_{lb}}} \; p_{n_{red}}, \, p_{n_{\min }}) \end{aligned} \end{aligned}$$
(33)

where \(p_{n_{red}}\) is a reduction factor calculated by using a new proposed function represented by Eq. (34). Differently from previous literature works, the new scheme to determine the reduction factor \(p_{n_{red}}\) defines directly in what iteration \(p_{n_{lb}}\) will reach its minimal value, ensuring that the angles of the fibers will not be stuck in the candidate angles much before the iteration \(n_{it_{\min }}\).

$$\begin{aligned} \begin{aligned} p_{n_{red}} = \left( \frac{p_{n_{\max }}}{p_{n_{\min }}}\right) ^{-\frac{1}{n_{it_{\min }}}} \end{aligned} \end{aligned}$$
(34)

and \(p_{n_{\min }}\) is calculated according to Eq. (35)

$$\begin{aligned} \begin{aligned} p_{n_{\min }}(realmin, \Delta \phi _{c_{\min }}) = \sqrt{\frac{(\Delta \phi _{c_{\min }})^2}{8 \, ln\left( \frac{1}{realmin} \right) }} \end{aligned} \end{aligned}$$
(35)

In Eq. (34), the term \(n_{it_{\min }}\) represents the number of iterations necessary until the value \(p_{n_{\max }}\) reaches \(p_{n_{\min }}\). In Eq. (35), \(realmin\) denotes the smallest positive normalized float-point number, a value contingent upon the specifics of the machine architecture [33] (\(realmin\) = \(2.2251 \cdot 10^{-308}\) for a 64-bit machine). Simultaneously, \(\Delta \phi _{c_{\min }}\) represents the minimum angular separation between two candidate angles.

The continuation for \(p_{n_{\max }}\) follows a similar approach as applied in Eq. (33). However, in updating \(p_{n_{\max }}\), care is taken to ensure that its value remains greater than the value of \(p_{n_{lb}}\):

$$\begin{aligned} \begin{aligned} p_{n_{\max }}^{k + 1} = \max (p_{n_{\max }}^k \; p_{n_{red}}, \, 1.4\,p_{n_{lb}}^{k + 1}) \end{aligned} \end{aligned}$$
(36)

The value 1.4 multiplying \(p_{n_{lb}}\) in Eq. (36) is a safety factor heuristically chosen to ensure that \(p_{n_{\max }}\) is always greater than \(p_{n_{lb}}\).

The parameters \(\beta _{\min }\) and \(p_{\min }\) are updated according to the value of \(g\) in each point of interest. The concept is that the grayer the region, the higher the increment in \(\beta _{\min }\) and \(p_{\min }\). Conversely, these values will remain the same for regions where \(\tilde{\tilde{\rho }}\) is equal to 0 or 1

$$\begin{aligned} \begin{aligned} \left( [\cdot ]_{\min }^{(e)}\right) ^{k+1} = \min \left( \left( [\cdot ]_{\min }^{(e)}\right) ^{k} + [\cdot ]_w\; \left( g^{(e)}\right) ^k, [\cdot ]_{\max } - 0.1 \right) ; \qquad [\cdot ] = \beta , p \end{aligned} \end{aligned}$$
(37)

where \(\beta _w\) and \(p_w\) represent weights for g. \([\cdot ]{\max } - 0.1\) is introduced to ensure that \([\cdot ]{\max }\) is always bigger than \([\cdot ]{\min }\)

In optimizing using the Tsai–Wu criterion, a different behavior emerges compared to the Tsai–Hill criterion in our proposed method, resulting in some instances displaying grayscale in the final optimization outcomes. Addressing this involves increasing the value of \(\beta _w\), which can lead to local minima in the tested examples. We suggest implementing a continuation in \(\beta _w\) to mitigate this, ensuring that, at the outset of the optimization, its values are sufficiently small to avoid undesirable outcomes yet are increased gradually to prevent grayscale in the final results. Besides, unlike the case with the Tsai–Hill yield criterion, a continuation in \(\beta _{\max }\) is necessary for feasibility in Tsai–Wu optimization. In specific scenarios, the optimization directs \(\beta \) values to become exceptionally high in specific domain regions. This results in \(\tilde{\tilde{\rho }}\) being confined to values between 0 and 1 at the beginning of the optimization, impacting sensitivity calculations and potentially leading to undesirable local minima. To address this issue, we propose a new function to enforce a continuation in both \(\beta _{\max }\) and \(\beta _w\), offering control over \(\beta \) values to prevent excessive magnitudes at the start of the optimization and allowing for a smooth continuation as needed:

$$\begin{aligned} \begin{aligned} \beta _{[\cdot ]}^{k+1} = \min (\gamma _{\beta _{[\cdot ]}} \; \beta _{[\cdot ]}^k, \beta _{{[\cdot ]}_{lim}}); \quad [\cdot ] = \max , w \end{aligned} \end{aligned}$$
(38)

where \(\beta _{{[\cdot ]}{lim}}\) represents the limit value for \(\beta {[\cdot ]}\). The calculation of \(\gamma _{[\cdot ]}\) is determined according to

$$\begin{aligned} \begin{aligned} \gamma _{\beta _{[\cdot ]}} = \left( \frac{\beta _{[\cdot ]_{lim}}}{\beta _{[\cdot ]_0}}\right) ^{\frac{1}{n_{it_{\min }}}} \end{aligned} \end{aligned}$$
(39)

where \({\beta _{[\cdot ]0}}\) denotes the initial value for \({\beta {[\cdot ]}}\). The scalar \(\gamma _{\beta _{[\cdot ]}}\), calculated using Eq. (39), ensures that \({\beta _{[\cdot ]}}\) reaches its maximum value by iteration \(n_{it_{\min }}\).

2.3 Augmented Lagrangian

Various methodologies have been proposed to tackle the resolution of Topology Optimization (TO) issues incorporating stress constraints. This work concentrates on the localized measurement of stress. To contend with a substantial number of constraints, we employ the Augmented Lagrangian approach [34], following a methodology akin to that proposed by Fancello and Pereira [35]. In light of this, we reformulate the optimization problem presented in Eq. (21) as follows:

$$\begin{aligned} \begin{aligned} \underset{\rho , \phi , p, p_n, \beta }{{\min }}&\qquad L_a = J + \frac{r_1}{2} \, \sum _{e=1}^{N_e} \left\langle \frac{\lambda _1^{(e)}}{r_1} + G_1^{(e)}\right\rangle ^2 + \frac{r_2}{2} \, \left\langle \frac{\lambda _2}{r_2} + G_2\right\rangle ^2 \\ \text {such that}&\qquad F = a(\textbf{u}, \textbf{v}, \tilde{\tilde{\rho }}) - L(\textbf{v}) = 0 \\&\qquad \rho _{\min } \le \rho _{lb}(g, g_{gl}) \le \rho (\textbf{x}) \le \rho _{ub}(g, g_{gl}) \le 1 \\&\qquad \phi _{\min } \le \phi (\textbf{x}) \le \phi _{\max }\\&\qquad p_{\min }(\textbf{x}, g) \le p(\textbf{x}) \le p_{\max }\\&\qquad p_{n_{\min }}(realmin, \phi ^c) \le p_{n_{lb}}(k) \le p_{n}(\textbf{x}) \le p_{n_{\max }}(k)\\&\qquad \beta _{\min }(\textbf{x}, g) \le \beta (\textbf{x}) \le \beta _{\max }\\ \end{aligned} \end{aligned}$$
(40)

where \(\langle [\cdot ] \rangle \) are the Macaulay brackets:

$$\begin{aligned} \begin{aligned} \langle [\cdot ]\rangle = {\left\{ \begin{array}{ll} 0, &{} [\cdot ] < 0 \\ {}[\cdot ], &{} [\cdot ] \ge 0 \end{array}\right. } \end{aligned} \end{aligned}$$
(41)

The constraints \(G_1\) and \(G_2\) are functions of \(\textbf{u}\), the displacement field that is the solution of \(F\). The variable \(r_i\) represents penalization parameters for the Augmented Lagrangian, while \(\lambda _i\) \(\quad i = 1, 2\) are the Lagrange multipliers. Following each subproblem solution, the values of \(r_i\) undergo updates. A subproblem is considered resolved whenever the optimization problem, as defined in Eq. (40), converges or reaches a predetermined maximum iteration count. The update mechanism for \(r_i\) is articulated as [36]:

$$\begin{aligned} \begin{aligned} r_i^{k+1} = \min \left( \gamma _{r,i}\;r_i^k, \frac{r_{i,\max }}{N_e}\right) \end{aligned} \end{aligned}$$
(42)

where \(\gamma _{r_i}\) works for \(r_i\) in the same way that \(\gamma _{\beta _{[\cdot ]}}\) works for \(\beta _{[\cdot ]}\) in Eq. (38), that is, for a initial value of \(r_i\), denoted by \(r_{i,0}\), the scalar \(\gamma _{r_i}\) ensures that \(r_i\) will reach the maximum value denoted by \(r_{i,\max }\) in iteration \(n_{it_{\min }}\)

$$\begin{aligned} \begin{aligned} \gamma _{r_i} = \left( \frac{r_{i,\max }}{r_{i,0}}\right) ^{\frac{1}{n_{it_{\min }}}} \end{aligned} \end{aligned}$$
(43)

where \(r_{i,0}\) is the initial value for the penalization parameter \(r_i\).

The update of \(\lambda _i\) is performed according to:

$$\begin{aligned} \begin{aligned} \lambda _i^{k+1} = \left\langle r_i\,G_i+\lambda _i^k \right\rangle \end{aligned} \end{aligned}$$
(44)

2.4 Sensitivities

This work employs an algorithm based on gradients to solve Eq. (40). Consequently, it is required to calculate the sensitivities of the Augmented Lagrangian function \(L_a\). So, it is necessary to compute the Gâteaux derivative of the Augmented Lagrangian function concerning all design variables, as follows:

$$\begin{aligned} \begin{aligned} DL_a(m_i)[\delta m_i] = DJ(m_i)[\delta m_i] + r_1 \sum _{e=1}^{N_e} \left\langle \frac{\lambda _1^{(e)}}{r_1} + G_1^{(e)} \right\rangle DG_1^{(e)}(m_i)[\delta m_i] + r_2 \left\langle \frac{\lambda _2}{r_2} + G_2 \right\rangle DG_2(m_i)[\delta m_i] \end{aligned} \end{aligned}$$
(45)

where all design variables are contained in the vector \(\textbf{m}\) = \([\rho (\textbf{x})\; \phi (\textbf{x})\; \; p(\textbf{x})\;\) \(p_n(\textbf{x})\; \beta (\textbf{x})]^T\). The term \(\delta m_i\) signifies an incremental function [37] corresponding to the i-th design variable. The definition of a Gateaux derivative of a functional \({\mathcal {F}}\) is given by [37]:

$$\begin{aligned} \begin{aligned} D{\mathcal {F}}(m_i) [\delta m_i] = \lim _{\alpha \rightarrow 0} \frac{{\mathcal {F}}(m_i + \alpha \; \delta m_i) - {\mathcal {F}}(m_i)}{\alpha } \end{aligned} \end{aligned}$$
(46)

where \(\alpha \) is a scalar value. Here, the derivatives presented in Eq. (45) are computed utilizing the Automatic Differentiation feature of the open-source Dolfin Adjoint project [38].

3 Numerical implementation

In this paper, the forward equation, represented as F in Eq. (40), is addressed using the open-source FEniCS computation platform [39]. This platform employs the FEM to solve the given problem.

The optimization problem involves two types of elements, as illustrated in Fig. 3. Differently than the previous work [19], we opt for a quadratic Lagrange element [39] for the state variable field (displacement \(\textbf{u}\)) in the place of the linear element. This approach helps to deal with numerically induced artificial stiffness, which can account for the formation of checkerboard patterns in optimization problems [40]. Simultaneously, a Discontinuous Lagrange element [39] is chosen for all design variables.

Fig. 3
figure 3

Elements used to solve the optimization problem

3.1 Refinement strategy

A significant portion of the computational costs associated with solving Topology Optimization (TO) problems using Augmented Lagrangian methods arises from the solution of the forward problem \(F\). These costs are directly tied to the scale of the FEM mesh, specifically the number of nodes. A greater number of elements contribute to increased computational overhead in resolving state equations [41]. However, finer meshes offer superior resolution and provide richer gradient information, particularly in regions with stress concentration. This additional data are crucial for facilitating stress relief within the structure. Therefore, an effective optimization strategy for FEM utilization involves meticulous mesh refinement in areas crucial to achieving an enhanced optimization outcome.

Within the FEniCS project, direct implementation of coarse elements within a FEM mesh is not supported. Consequently, the employed strategy focuses exclusively on element refinement. The refinement mesh proposed in a previous work [19] is based on three parameters: a stress error measurement (\(\sigma _{er}\)), the stress constraint \(G_1\), and the variations in \(\tilde{\tilde{\rho }}\) within an element relative to the values of its neighbors, denoted as \(var_{\rho }\). The stress error (\(\sigma _{er}\)) is determined as a function of the Tsai–Wu criterion, derived through a residual displacement, and multiplied by the current \(\tilde{\tilde{\rho }}\), as detailed below:

$$\begin{aligned} \begin{aligned} \sigma _{er}(\textbf{u}_{res}) = \tilde{\tilde{\rho }} \; f_{\sigma } \; \sigma _{th}(\textbf{u}_{res}) \end{aligned} \end{aligned}$$
(47)

The residual \(\textbf{u}_{res}\) in Eq. (47) is determined by evaluating the difference between the current displacement \(\textbf{u}\), computed within the present finite element mesh, and the displacement \(\textbf{u}_{ref}\) obtained from a converged regular mesh, as outlined in the following equation:

$$\begin{aligned} \begin{aligned} \textbf{u}_{res} = \textbf{u} - \textbf{u}_{ref} \end{aligned} \end{aligned}$$
(48)

To identify the converged mesh, the same approach used in the previous work is used [19], that is, the Finite Element Method (FEM) simulation is executed on an initial coarse mesh. The maximum displacement \(u_{\max }\) is measured within this coarse mesh. Subsequently, each direction receives one additional element, and the iterative process continues until the difference between \({u_{\max }^j}\) and \({u_{\max }^{j-1}}\) is smaller than a suitable tolerance, which means that the mesh converged. Defining the refined mesh is performed just once before the optimization process starts. The variable \(var_{\rho }\) denotes the absolute difference between the \(\tilde{\tilde{\rho }}\) values of neighboring elements.

To establish a refinement limit, a minimum value for h is defined for each measure

$$\begin{aligned} \begin{aligned} h_{[\cdot ], \min } = \max \left( \frac{h_0}{n_{ref,[\cdot ]}^i\;2^{\left\lfloor \frac{k}{s_{ref}}\right\rfloor +1}}, \; \frac{h_0}{2^{n_{ref,[\cdot ]}^f}}\right) \qquad [\cdot ] = G_1, \sigma _{er}, var_{\rho } \end{aligned} \end{aligned}$$
(49)

where \(h_0\) represents the longest edge of the elements in the initial mesh, \(n^i_{ref,[\cdot ]}\) is the initial number of refinements, \(n^f_{ref,[\cdot ]}\) is the final number of refinements, and \(s_{ref}\) signifies the step that determines when the number of refinements will increase during iterations.

The variable \(ref\) is used to determine if a refinement is performed according to

$$\begin{aligned} \begin{aligned} ref^{(e)} =&\, \left( G_1^{(e)}> l_{G_1} \text { and } h^{(e)}> h_{G_1, \min }\right) \text { or }\\&\, \left( \sigma _{er}^{(e)}> l_{\sigma _{er}} \text { and } h^{(e)}> h_{\sigma _{er}, \min }\right) \text { or }\\&\, \left( var_{\rho }^{(e)}> l_{var_\rho } \text { and } h^{(e)} > h_{var_\rho , \min }\right) \\ \end{aligned} \end{aligned}$$
(50)

where \(l_{[\cdot ]}, \;\;\; [\cdot ] = G_1, \sigma _{er}, var_{\rho }\) represent the limit for each measure and \(h\)\(^{(e)}\) denotes the height of element e. The element (e) will be refined if any condition specified in Eq. (52) is true.

The refinement procedure uses the FEniCS function named refine(). The \(ref\) variable undergoes evaluation for each element. If \(ref\) holds the value true, the FEniCS refine() function is invoked. In this particular scenario, the original element undergoes subdivision into four distinct parts. The refinement concludes when no further modifications exist in the count of elements over consecutive iterations. Furthermore, refinement occurs continuously throughout the optimization process. When the global gray value exceeds 0.15, refinement starts from the initial mesh configuration. Conversely, when the global gray value decreases to 0.15 or below, refinement begins from the most recently refined mesh.

3.2 Spatial filter

Topology optimization often faces challenges related to mesh dependence and checkerboards. To mitigate these issues, regularization is essential. In this study, we employ the spatial filter proposed by Bruns and Tortorelli [42], a method mathematically validated by Bourdin [43]. This approach addresses concerns related to mesh dependence and checkerboards. The numerical formulation utilized in this work is consistent with the approach presented by Andreassen et al. [44]:

$$\begin{aligned} \begin{aligned} (\tilde{m}_i)_e = \frac{1}{\displaystyle {\sum _{j=1}^{N_e} H_{ej}}}\sum _{j=1}^{N_e} H_{ej} (m_i)_e \end{aligned} \end{aligned}$$
(51)

where \(\tilde{m}_i\) represents the i-th component of the vector of filtered design variables. The index e corresponds to element (e) in the finite element mesh, and \(H_{ei}\) is a weight factor defined as:

$$\begin{aligned} \begin{aligned} H_{ei} = {\max }(0, r_{\min } - \Delta (e, i)) \end{aligned} \end{aligned}$$
(52)

The parameter \(r_{\min }\) represents the spatial filter radius, and \(\Delta (e, i)\) is the distance between the center of the element e and element i.

3.3 Optimization procedure

Solving Eq. (40) follows an iterative approach, as depicted in Fig. 4. Initially, the domain is defined, as well its boundary condition. After that, initial assumptions are made for design variables. Subsequently, mesh refinement follows the procedure outlined in Sect. 3.1. Upon acquiring the updated mesh, the forward problem (Eq. (1)) is addressed through the Finite Element Method (FEM). Subsequently, the objective function J is calculated and evaluated, aided by the FEniCS Project software [39]. The determination of sensitivities is achieved through the utilization of Dolfin adjoint [38].

The optimization process is performed by using the L-BFGS-S algorithm [45]. The iteration process persists until convergence is attained within SciPy [46]. Throughout this process, the design variables are iteratively updated, and this sequence is repeated. Upon achieving convergence within SciPy, stop criteria for the subproblem are evaluated to determine if an optimized design has been reached. If the any stop criteria is reached, the parameters, including \(\rho _{lb}\), \(\rho _{ub}\), \(p_{\min }\), \(p_{n_{lb}}\), \(p_{n_{\max }}\), \(\beta _{\min }\), \(r_1\), \(r_2\), \(\lambda _1\), and \(\lambda _2\), are adjusted, and the process is reiterated.

Fig. 4
figure 4

Optimization flowchart

The criteria to determine the convergence of the subproblems are based on the previous work [19]. The first criterion checks if the maximum stress constraint (\(G_1\)), the compliance constraint (\(G_2\)), and the global gray measure (\(g_{gl}\)) fall below predefined thresholds. Additionally, the convergence of the fiber angles, represented by the variable \(\chi \), is also verified [9].

$$\begin{aligned} \begin{aligned} stop_{crit, 1} = ||G_1||_\infty \le l_{sc} \text { and } G_2 \le l_{cc} \text { and } g \le l_g \text { and } \chi \le w_i, \quad i = 1, 2,..., N_c \end{aligned} \end{aligned}$$
(53)

where \(l_{sc}\) is the allowable limit for the stress constraint, \(l_{cc}\) is the allowable limit for the compliance constraint \(G_2\), and \(l_g\) is the allowable limit for the amount of gray measure \(g_{gl}\). The variable \(\chi \) is defined as [9]

$$\begin{aligned} \begin{aligned} \chi = 0.95 \sqrt{w_1^2 + w_2^2 + \cdots + w_{N_c}^2} \end{aligned} \end{aligned}$$
(54)

The second stopping criterion is based on the objective function values in each subproblem, according to Eq. (55).

$$\begin{aligned} \begin{aligned} \text {condition 1} = \frac{\text {abs} \left( ln \left( \text {abs} \left( \frac{J^k}{J^{k-1}}\right) \right) \right) }{ln\left( \text {abs}\left( J^k\right) \right) } < \xi \end{aligned} \end{aligned}$$
(55a)
$$\begin{aligned} \begin{aligned} \text {condition 5} = \frac{\text {abs} \left( ln \left( \text {abs} \left( \frac{J^{k-1}}{J^{k-2}}\right) \right) \right) }{ln\left( \text {abs}\left( J^{k-1}\right) \right) } < \xi \end{aligned} \end{aligned}$$
(55b)

where \(\xi \) is a suitable scalar number; the optimization process concludes when both conditions outlined in Eq. (55) are met. However, if any of these criteria is satisfied within a specific number of iterations, the stopping criterion becomes the maximum allowable number of iterations.

4 Results

This section presents the results obtained through the NDFO-adapt method considering Tsai–Wu as the stress constraint criterion in solving three distinct problems: an L-bracket, a Wall with opening, and a Hammerhead pier support. The domains are parameterized with a length \(l = 1\) m. The radius for the density filter \(r_{\min }\) is set to 0.07 l. The 46 candidate angles range from \(-\frac{\pi }{2}\) to \(\frac{\pi }{2}\). The vector of candidate angles \(\varvec{\phi }_c\) is represented as \(\varvec{\phi }_c = {x,|, x = -\frac{\pi }{2} + \frac{\pi }{45}, n, n \in {0, 1,..., 45} }\).

For the first iteration, the height of the elements \(h_0\) is set to 0.05l, and \(\lambda _i \quad i = 1, 2\) are equal to 0. The variable \(\xi \), which controls the stop criteria based on the convergence, is equal to \(1 \cdot 10^{-6}\). Additional parameters are shown in Table 1, while mesh refinement parameters are outlined in Table 2

Table 1 Initial parameters shared for all problems
Table 2 Parameters for mesh refinement

4.1 L-bracket problem

The L-bracket serves as a benchmark for problems related to Topology Optimization (Topology Optimization (TO)) while considering stress constraints. In Fig. 5, the domain dimensions and corresponding boundary conditions are depicted. The structure is clamped at the top, and a load of \(t = 6\) MPa is applied along a length of 0.05 l. The move limit for the design variable \(\rho \) is defined as \({\mathfrak {m}}_\rho \) = 0.2. The maximum number of iterations is set to 500, and the variable \(n_{it_{\min }}\) is established at 400.

Fig. 5
figure 5

L-bracket domain

The fields of physical pseudo-densities \(\tilde{\tilde{\rho }}\) and physical angles \(\tilde{\tilde{\phi }}\) are depicted in Fig. 6a. The visualization reveals that the fibers exhibit a consistent continuity, as highlighted in Details 2 and 3, owing to the spatial filter. Additionally, it is evident that the fibers closely follow the path formed by the material distribution. However, in the region emphasized in Detail 1, the fibers appear to be trapped in a local minimum. Wavy patterns are observed mainly in the lowest and oblique 45-degree members, indicating the necessity of increasing the weight of the compliance constraint to address these irregularities more effectively.

As illustrated in Fig. 6b, the stress constraint is satisfied across nearly the entire domain. Although there are certain regions where the stress constraint is not fully satisfied, it is essential to highlight that the maximum stress constraint value is 9.29 \(\cdot 10^{-3}\), which is very close to zero.

Fig. 6
figure 6

L-bracket result

Fig. 7
figure 7

Convergences of the objective function and compliance constraint for the L-bracket result

In Fig. 7, the convergences of the objective function and the compliance constraint for the L-bracket problem are shown. The curves reveal that, after the first iteration, the objective function undergoes a significant decrease from the initial guess of 0.5 to a value smaller than 0.1. However, from the second iteration onwards, the objective function value experiences abrupt increases until iteration 23, reaching approximately 0.27. Subsequently, the objective function exhibits small oscillations between iterations 20 and 90. After iteration 90, the curve forms steps and stabilizes, approaching the value of the variable \(n_{it_{\min }}\). In Fig. 7b, the compliance constraint demonstrates consistent convergence, barring minor oscillations between iterations 20 and 90. This period corresponds to the sharp increase in the objective function value. Notably, the objective function increases to respect the compliance constraint; in other words, increasing the volume of the structure leads to a reduction in compliance. The final value of the objective function is approximately 0.39, while the compliance constraint reaches a final value of 2.8\(\cdot 10^{-2}\). Although the compliance constraint does not precisely reach zero because the maximum number of iterations is reached, it is very close to this threshold, indicating a feasible result.

Table 3 Optimized fields for L-bracket in different iterations

Table 3 presents the evolution of design variable fields  -   \(\tilde{\tilde{\phi }}\), \(\tilde{\tilde{\rho }}\), \(\tilde{p}_n\), \(\tilde{p}\), and \(\tilde{\beta }\)  -   for the L-bracket problem across different iterations. The first row displays the fields after the initial iteration, while subsequent rows showcase the fields after iterations 10, 200, and 500. In the initial iteration, the field of physical pseudo-densities (\(\tilde{\tilde{\rho }}\)) exhibits higher values in the region where fiber discontinuities occur and in the structure joints. By iteration 10, a preliminary glimpse of the final topology emerges. The inferior horizontal members and the vertical supporting members still retain gray values. However, by iteration 200, the structure becomes predominantly free of gray, indicating substantial progress. Despite this, as evident in Fig. 7, the compliance constraint is not yet met in this iteration. In the 500th iteration, the \(\tilde{\tilde{\rho }}\) field closely resembles that of the 200th iteration but with thicker members, ensuring that compliance constraint is satisfied. In the first iteration, the field of penalizations (\(\tilde{p}_n\)) exhibits higher values in specific regions, such as the area close to the clamped part in the center of the vertical section of the "L," where the presence of void (indicated by \(\tilde{\tilde{\rho }}\)) is evident. Other notable regions include the area above the inclined member uniting the two vertical members supporting the structure and in the upper and bottom parts of the horizontal section of the "L." In the first iteration, \(\tilde{p}_n\) ranges from 0.1 to 2.6. By the 10th iteration, the field of penalization (\(\tilde{p}_n\)) remains similar to the first iteration, with the main difference lying in its values, which now range from 8.56 \(\cdot 10^{-2}\) to 3.47. The 200th iteration presents a seemingly homogeneous field for \(\tilde{p}_n\), yet its values vary from 4.27 \(\cdot 10^{-3}\) to 1.64 \(\cdot 10^{-1}\). In the 500th iteration, \(\tilde{p}_n\) values range from 9.90 \(\cdot 10^{-4}\) to 1.29 \(\cdot 10^{-3}\), ensuring convergence of 99.7% of fiber angles (represented by the variable \(\tilde{\tilde{\phi }}\)) to the candidate angles, as depicted in Fig. 8. The field of \(\tilde{p}\) exhibits its highest value in the region between the two vertical members supporting the structure and the left bottom side. However, other areas appear to be homogeneous, although it is noticeable that values are slightly higher in regions where \(\tilde{\tilde{\rho }}\) is higher than zero. In the first iteration, \(\tilde{p}\) ranges from 1.11 to 5.84. By the 10th iteration, the field remains very similar to the first iteration, with values slightly higher in regions where \(\tilde{\tilde{\rho }}\) is higher than zero. Values of \(\tilde{p}\) for the 10th iteration vary from 1.02 to 5.9. In the 200th and 500th iterations, the fields are nearly identical, featuring higher values in the interfaces between solids and voids and values closer to zero in most areas with voids in the structure. In both iterations, the values of \(\tilde{p}\) range from 1.02 to 7.90. The \(\tilde{\beta }\) field appears homogeneous in iterations 1 and 10, but its values range from 2.87 to 10.1 in the first iteration and from 4.37 to 10.7 in the tenth iteration. By the 200th iteration, higher \(\tilde{\beta }\) values concentrate at the interface between the void and the material, mirroring the distribution of \(\tilde{p}\). In this iteration, \(\tilde{\beta }\) varies from 8.18 to 37.4. In the final iteration, prominent \(\tilde{\beta }\) values trace a path starting from the vertical member in the horizontal part of the "L," extending to the end of the inclined member connecting the two vertical members supporting the structure, and at interfaces between the void and the material. In the last iteration, the values of \(\tilde{\beta }\) vary from 8.18 to 101.0.

Fig. 8
figure 8

Convergence of the fibers angles \(\tilde{\tilde{\phi }}\) to the candidate angles

4.2 Wall with opening

The second case involves the Wall with Opening depicted in Fig. 9. This structure possesses dimensions of \(l \times 0.5, l\) and incorporates an opening on the left side at its bottom, with dimensions \(0.35, l \times 0.35, l\). The left side is clamped, while the support on the right side permits unrestricted displacement along the x axis. A load of \(t = 6\) MPa is applied, aligned with a length of 0.1, l, situated on the domain’s top at \(x = 0.65 m\). The move limit for the variable \(\rho \) is established at \({\mathfrak {m}}_\rho \) = 0.2. The maximum number of iterations is 500, with the variable \(n_{it_{\min }}\) specified as 300.

Fig. 9
figure 9

Wall with opening domain

The fields of physical pseudo-densities \(\tilde{\tilde{\rho }}\) and physical fiber orientation \(\tilde{\tilde{\phi }}\) are illustrated in Fig. 10. The resulting structure is supported by two vertically oriented members, with the right member, closer to the load application, being thicker than the left member. It is evident that the fibers tend to follow the path formed by the material distribution, and the result exhibits consistent fiber continuity in most parts of the domain, as highlighted in Detail 3. However, Details 1 and 2 stand out as exceptions, where the fibers undergo abrupt changes in direction. In these regions, the values of \(\tilde{\tilde{\rho }}\) demonstrate an early rise preceding that observed in neighboring regions. This pattern is observed in Table 3. Observing the field of \(\tilde{\tilde{\rho }}\) at the end of the first iteration with deformations 100 times greater (Fig. 11), it is apparent that these regions are under bend solicitation. The value of the objective function in the final iteration is approximately 0.32, and the compliance constraint is equal to approximately \(-\)2.51.

Fig. 10
figure 10

Fiber orientation and material distribution for Wall with opening result

Fig. 11
figure 11

Field of physical pseudo-densities \(\tilde{\tilde{\rho }}\) for the Wall with opening result in the first iteration with the displacements 100 times greater

In Fig. 12, the stress constraint field for the Wall with Opening result is presented. The stress constraint is adhered to in almost the entire domain, with only a few elements showing stress constraint values equal to 5.8 \(\cdot 10^{-3}\), which is quite close to zero.

Fig. 12
figure 12

Stress constraint field for Wall with opening result

As depicted in Fig. 13, the convergence of the objective function exhibits small oscillations in the initial iterations. Starting from iteration 200, the behavior begins to stabilize, demonstrating an almost constant value until reaching iteration 321, where a slight decrease occurs. Subsequently, the objective function convergence maintains an almost constant value.

Fig. 13
figure 13

Convergence of the objective function for the Wall with opening problem

The design variable fields for the Wall with opening problem are showcased in Table 4 at iterations 1, 10, 200, and 500. As mentioned before, it is noticeable that the regions with an abrupt change in fiber direction in both vertical members are among the areas where the value of the pseudo-densities is higher in the first iteration. Subsequent iterations (10, 200, 500) demonstrate that the topology does not change significantly during the iterations. The main difference is the grayscale, which decreases with incrementing iterations. The variable \(\tilde{p}_n\) exhibits similar behavior in iterations 1 and 10, with higher values in the region above the opening and a considerable area in the center of the domain. The range for the values of \(\tilde{p}_n\) is from 0.097 to 1.28 for iteration 1 and from 0.075 to 1.316 for iteration 10. In iteration 200, \(\tilde{p}_n\) appears as a homogeneous field, and the same trend continues in iteration 500. Indeed, in iteration 200, the range of \(\tilde{p}_n\) is from 0.0016 to 0.016, and in iteration 500, the range is from 0.0009 to 0.0013. In iteration 1, the filtered penalization \(\tilde{p}\) is higher in two specific domain regions: at the right side of the opening above the center of the domain and in the superior right corner. In the first iteration, the values of \(\tilde{p}_n\) range from 1.00 to 6.15. In iteration 10, a similar behavior is observed, but the values of \(\tilde{p}_n\) start to become higher in regions where \(\tilde{\tilde{\rho }}\) is higher than 0. The values of \(\tilde{p}_n\) vary from 1.00 to 5.90 in iteration 10. The fields of \(\tilde{p}_n\) are very similar for iterations 200 and 500, with significant values in all regions of the domain where \(\tilde{\tilde{\rho }}\) is higher than zero and even higher values in the area below the application of the force, in the union of the inclined member with the vertical member, and the vertical right member in a region below the center of the domain. The values of \(\tilde{p}_n\) range from 1.08 to 7.85 in iteration 200 and from 1.10 to 7.84 in iteration 500. The variable \(\tilde{\beta }\) seems homogeneous in the iterations 1 and 10. However, its values range from 6.21 to 20.7 in the iteration 1 and from 6.34 to 21.25 in the iteration 10. In iteration 200, the highest values of \(\tilde{\beta }\) are observed in regions where \(\tilde{\tilde{\rho }}\) is higher than 0. In iteration 200, the values of \(\tilde{\beta }\) range from 5.61 to 121.2. In the last iteration, the values of \(\tilde{\beta }\) range from 7.38 to 305.4 and the highest values are in the left lower area. Besides, it is possible to observe values higher than the average in the region above the load application, in the union of the inclined member with the superior horizontal member, in the right lower area, and in the left upper area.

Table 4 Optimized fields for Wall with opening in different iterations

4.3 Hammerhead Pier Support

The last example is the Hammerhead Pier Support, as presented in Fig. 14a. The domain is characterized by a height of 0.75, l and a width of l. Notably, it incorporates two bottom openings, each possessing a height of 0.5, l, thus creating a member with a width of \(\frac{1}{6},l\). The simulation is focused on only half of the domain, with corresponding boundary conditions outlined in Fig. 14b. Four loads, each with a magnitude of \(t = 80, MPa\), are strategically applied at the top to subject the structure to specific loading conditions. Two loads are positioned at the far left and right ends, while the other two are symmetrically spaced, with a separation of \(\frac{1}{3},l\). For this problem, the maximum number of iterations and the variable \(n_{it_{\min }}\) are both set to 300. The initial values for \(r_1\) and \(r_2\) are 10 and \(5 \cdot 10^{-4}\), respectively, while their maximum values, \(r_{1, {{\max }}}\) and \(r_{2, {{\max }}}\) are \(1 \cdot 10^{4}\) and 10. The variable \(\alpha _c\) is equal to 4.

Fig. 14
figure 14

Hammerhead Pier Support domain

Figure 15a shows the optimized material distribution and fiber orientation. Two vertical members support the final topology, with most fibers oriented vertically. An exception can be observed in Detail 2, where a discontinuity arises from a member that initially united the two vertical members, however, vanished in the first iteration. This disappearance led to the entrapment of fibers in a local minimum. In areas where forces are applied, such as in Detail 1 and the superior extremities of the structure, fibers are predominantly oriented vertically. Similar to the previous examples, the fibers tend to follow the path formed by the material distribution, resulting in consistent continuity.

The volume fraction of the optimized structure is approximately 0.39, and the compliance constraint \(G_2\) in the last iteration is equal to \(6.93 \cdot 10^{-3}\).

In Fig. 15b, the stress constraint field in the last iteration for the Hammerhead Pier Support is depicted. The visualization reveals that the constraint is adhered to across nearly the entire domain, with a few exceptions, where the red color signifies values slightly above zero. Nonetheless, the maximum stress constraint value is \(5 \cdot 10^{-3}\), which is extremely close to zero, making the result acceptable.

Fig. 15
figure 15

Hammerhead result

Table 5 shows the design variables field for the optimized structure in iterations 1, 10, 200, and 300. After the initial iteration, the physical pseudo-density \(\tilde{\tilde{\rho }}\) exhibits a predominantly gray appearance, with values ranging from \(1.81 \cdot 10^{-2}\) to 0.64. Notably, the emergence of a member connecting the two vertical supporters of the structure is observed in this iteration, a region where, toward the end of optimization, the fiber is trapped in a local minimum. By the 10th iteration, the physical pseudo-densities are between 0 and 0.99. Notably, elevated values are concentrated in the upper region of the structure, as well as the space previously occupied by the connecting member. The fields are similar in iterations 200 and 300, featuring values from 0 to 1 with almost no gray values. In the same way, as observed in the previous work where the Tsai–Hill yield criterion is considered [19], the last iterations ensure the convergence of 99.4% of the fiber angles to the candidate angles, as shown in Fig. 16. Total convergence can be reached with additional iteration.

Table 5 Optimized fields for Hammerhead Pier Support in different iterations
Fig. 16
figure 16

Fiber convergence for the Hammerhead Pier Support

After the initial iteration, the optimization algorithm assigns \(\tilde{p}_n\) values ranging from 0.0999 to 1.384, with higher values concentrated in the upper part of the domain, coinciding with areas where the final result lacks material. By the 10th iteration, the greater values persist in these regions, but the \(\tilde{p}_n\) values exhibit a different range, spanning from 0.077 to 1.37. Subsequently, in iterations 200 and 300, the \(\tilde{p}_n\) field becomes notably more homogeneous, varying from 0.004 to 0.016 at iteration 200 and from 0.001 to 0.0013 at iteration 300.

After the initial iteration, the variable \(\tilde{p}\) manifests values ranging from 1.01 to 6.96. Higher values are concentrated in the upper part of the structure, proximate to the application points of the extremity forces, as depicted in Table 5. By iteration 10, the values extend from 1.03 to 6.43, with the highest values persisting in the same area observed in the initial iteration. Notably, in regions where the physical pseudo-densities \(\tilde{\tilde{\rho }}\) are greater than zero, the values of \(\tilde{p}\) begin to increase, influenced by the update scheme based on the gray values. The \(\tilde{p}\) fields in iterations 200 and 300 display similarities, with values ranging from 1.06 to 7.9 in both instances.

The variable \(\tilde{\beta }\) starts with a seemingly homogeneous field. Although the first iteration field in Table 5 appears homogeneous, \(\tilde{\beta }\) values range from 5.10 to 9.97. By the end of iteration 10, these values exhibit slight changes, now varying from 5.29 to 10.06. In iteration 200, \(\tilde{\beta }\) values increase in regions with material, ranging from 5.73 to 63.12. By iteration 300, the smallest \(\tilde{\beta }\) does not change. However, the highest value rises to 129.31.

5 Conclusions

This paper proposes the NDFO-adapt method to minimize the structure volume, incorporating compliance and local stress constraints based on the Tsai–Wu yield criterion. The material model integrates penalization as a design variable, eliminating the necessity for heuristically selecting values. This concept is extended to include the projection parameter \(\beta \) and the SIMP penalization p, broadening the solution space and enabling the exploration of new local minima.

Given the constraints previously mentioned, the challenge of minimizing the volume was effectively addressed. Although some examples violated the constraints, the violation was marginal, with a value approaching zero, rendering the results feasible.

New functions to control the continuation for the box constraints of the design variable were proposed to circumvent the local minima obtained with the previous approach applied when the Tsai–Hill yield criteria were considered. These new functions allow better control of the design variable values during optimization, allowing for more feasible results.

The move limit scheme depends on the amount of gray in the structure, providing a strategy to navigate around local minima, albeit without complete assurance. Furthermore, a new function to update the box constraints of \(\beta \) and p is proposed, where the control above the increase in these values was improved, allowing a smooth continuation precisely controlled by a maximum number of iterations.

The fiber orientation exhibited uniform continuity in most regions across all solved problems. However, certain specific regions did present some discontinuities. These discontinuities can be managed effectively by incorporating manufacturing constraints.

Future work will delve into refining the treatment of discontinuities, extending the methodology to 3D domains, and enhancing the method’s robustness.