Background

Colorectal cancer (CRC) is the second most commonly diagnosed cancer in women and the third most commonly diagnosed cancer in men, accounting for approximately 10% of all cancer diagnoses annually and associated mortality (Dekker et al. 2019). Despite advances in treatment modalities such as surgery, chemotherapy, and radiotherapy, CRC management faces significant challenges, including late diagnosis and limited treatment efficacy, resulting in unfavorable outcomes such as toxicity, relapse, and drug resistance (Rejhová et al. 2018). Therefore, there is an urgent need for innovative therapeutic strategies targeting multiple carcinogenic mechanisms to effectively manage CRC.

Nano-drug delivery systems represent a promising approach to overcome the limitations of conventional chemotherapy in cancer treatment. These systems are designed to enhance drug solubility, bioavailability, and targeting, thereby maximizing drug levels at the target site while minimizing systemic toxicity and side effects (Pérez-Herrero and Fernández-Medarde 2015). They provide precise drug delivery to cancer cells, protect healthy tissues, reduce systemic toxicity and side effects, and allow for controlled drug release over time, potentially improving therapeutic outcomes (Naeem et al. 2020; Patra et al. 2018; Adepu and Ramakrishna 2021; Li et al. 2017a).

Boswellic acid derivatives, derived from the resin of the genus Boswellia, have attracted attention for their anticancer properties. Compounds such as α- and β-boswellic acid (BA), 3-O-acetyl-α- and β-boswellic acid (ABA), 11-keto-β-boswellic acid (KBA), and 3-O-acetyl-11-keto-β-boswellic acid (AKBA) have demonstrated anti-inflammatory, anti-arthritic, and anticancer effects in various preclinical studies (Efferth and Oesch 2022). Their antiproliferative effect against various types of cancer has been extensively studied in both in vitro and in vivo settings (Roy et al. 2016). However, their therapeutic potential is limited by poor solubility and bioavailability (Hussain et al. 2017; Abdel-Tawab et al. 2011).

Researchers have explored alternative delivery methods to address the solubility and bioavailability challenges associated with BA derivatives, including naturally derived polymeric nanoparticles such as sodium alginate (SA) and chitosan (CS). In particular, sodium alginate is a hydrogel-based natural polymer with exceptional properties including chelation, biocompatibility, immunogenicity, and mucoadhesion (Choukaife et al. 2020). Chitosan, another natural polymer, is well suited for advanced drug delivery systems due to its advantageous properties, including low immunogenicity, mucoadhesiveness, biocompatibility, biodegradability, and non-toxicity (Fathi et al. 2018). In particular, the pH sensitivity of chitosan-based nanocapsules allows for targeted drug release, enhancing drug bioavailability while minimizing off-target effects (Wang et al. 2020). Calcium chloride (CaCl2) has been investigated as a cross-linking agent to form stable CS-SA nanoparticles by linking the amino and carboxyl groups of chitosan and alginate chains, respectively (Takka and Gürel 2010). This cross-linking reaction is crucial for maintaining the stability and integrity of the nanoparticle, preventing rapid drug release, and enhancing mucoadhesive properties, thereby improving drug release kinetics and bioavailability (Khan et al. 2020; Szekalska et al. 2018; Wang et al. 2023).

Despite the progress in the use of CS-SA-CaCl2 nanoparticles for various drugs, their application as a delivery system for BA derivatives remains unexplored. Considering the reported anticancer activity of BA derivatives and their solubility problem, the investigation of CS-SA-CaCl2 nanoparticles as a delivery system provides an opportunity to overcome these limitations and improve their therapeutic efficacy in cancer therapy due to their biocompatibility, biodegradability, mucoadhesiveness, and controlled drug release (Kiti and Suwantong 2020). Therefore, this study aims to prepare CS-SA-CaCl2 nanoparticles for the delivery of AKBA and evaluate their antiproliferative effects on cancer cells.

Results

BA extraction using three-phase partitioning (TPP) followed by column chromatography

The three-phase partitioning (TPP) method was used to extract AKBA from the gum resin of B. serrata. AKBA-containing fractions (BA-Ex) were isolated and confirmed via high-performance liquid chromatography (HPLC). Finally, the amount of AKBA in the extract (BA-Ex) was quantified through UV detection at 260 nm (Fig. 1, blue color). The extraction efficiency was calculated to be 12.64% based on the standard curve of the AKBA standard (Y = 13348X − 16142, R2 = 0.9984).

Fig. 1
figure 1

Representative chromatograms of AKBA standard (in green color), BA-Ex (in blue color), and nano-BA-Ex (in black color)

Preparation and characterization of BA-extract-loaded nanoparticles (nano-BA-Ex(

Dual cross-linked nanoparticles loaded with BA-Ex were synthesized by ionic pregelation and polyelectrolyte complexation methods using sodium alginate and chitosan. The loading capacity and encapsulation efficiency of nano-BA-Ex were determined to be 6.9% and 75.16%, respectively, based on Eqs. (1) and (2). Field emission scanning electron microscopy (FE-SEM) revealed particle structures with an average diameter of about 100 nm (Fig. 2A). Further characterization of the nanoparticles included measurement of mean size, polydispersity index (PDI), and zeta potential (ζ-potential) using dynamic light scattering (DLS) and zeta sizer (Fig. 2B, C). Fourier transform infrared (FTIR) spectra were used to analyze the chemical and crystalline properties of nano-BA-Ex, demonstrating the presence of BA and its derivatives (Fig. 2D). Specifically, the absorption peak at 1243 cm−1 indicated the stretching vibration of C–O bonds commonly found in ester groups (–COO–), carboxylic acids (-COOH), or ethers (–C–O–C–), and in some cases, C–N bonds. In addition, the peak observed at 2921 cm−1 corresponded to the stretching vibration of C–H bonds in aliphatic hydrocarbons, reflecting the symmetric stretching of CH2 and CH3 groups, with intensity variations depending on hydrogenation and chemical environment.

Fig. 2
figure 2

Characterization of BA-Ex loaded nanocapsules: A FE-SEM micrographs of nano-BA-Ex, magnification: 30 KX; B DLS: particle size (mean): 104.5 ± 9.2; C zeta potential (mean): − 37.7 mV, electrophoretic mobility (mean): − 0.000292 cm2/Vs; D Fourier transform infrared (FTIR) spectra of BA-Ex (black), nanocarrier (red), and nano-BA-Ex (Blue)

The pH-responsive behavior is a key feature of the chitosan-sodium alginate–calcium chloride (CS-SA-CaCl2) nanoparticle system we have developed. In order to investigate pH-responsive release profile, AKBA release from nano-BA-Ex was investigated at two different pH values (2.5 and 5) at room temperature. As shown in Fig. 3, the drug released amounts in the 5 pH was higher than 2.5 in 4 h. This could be explained by the formation and elimination of ionic interaction between AKBA and nanocarrier with increase of pH.

Fig. 3
figure 3

Profile of AKBA release from nano-BA-Ex at two different pH values (2.5 and 5)

In vitro

Nano-BA-Ex inhibited cellular growth in HT-29 cells

The cytotoxicity of AKBA, BA-Ex, nano-BA-Ex, 5-Fluorouracil (5-FU, as a positive control), and blank nanocarrier on HT29 human colon adenocarcinoma cells and normal fibroblast cells was evaluated using the MTT assay. After 48 h of each treatment, both cell lines showed a dose-dependent decrease in cell viability for each treatment compared to untreated cells (Fig. 4). However, normal fibroblasts exhibited a smaller decrease in cell viability for each treatment than HT29 cells (p < 0.001). Nano-BA-Ex, even at one-tenth concentration, resulted in a significantly greater decrease in cell viability compared to AKBA and BA-Ex, when compared to 5-FU (as positive control), even in one-tenth of concentration (p < 0.001).

Fig. 4
figure 4

Relative cell viability of HT29 (solid colors) and fibroblast cells (shaded colors) after treatment with 5-FU (15, 30, and 60 µM), AKBA (50, 100, and 150 µM), BA-Ex (50, 100, and 150 µM), nano-BA-Ex (5, 10, and 20 µM), and nanocarrier (5, 10, and 20 µM) after 48 h

Nano-BA-Ex enhanced apoptosis in HT-29

Annexin V staining

The annexin V staining assay was employed to assess apoptosis induction following a 48-h treatment with different test compounds. Flow cytometry analysis revealed distinct populations based on fluorescence intensity in the FITC and PI channels. Cell distribution in distinct quadrants was determined using quadrant statistics. The administration of each test compound led to differences in cell populations within specific quadrants, indicating varying levels of apoptosis induction. Figure 5A illustrates the distribution of cells in various quadrants following treatment with 5-FU (30 µM), AKBA (100 µM), BA-Ex (100 µM), and nano-BA-Ex (10 µM), compared to the control group, as determined using scatter plots. The distribution of quadrants and the early/late apoptotic rate for each treatment compound are presented in Fig. 4B, respectively. As shown in Fig. 5B, the apoptosis rate in nano-BA-Ex was higher than 5-FU.

Fig. 5
figure 5

A Flow cytometry scatter plots of each treatment on HT29 cells, B column chart of early and late apoptotic rate in each compound treatment

Cell cycle analysis

The effect of various test compounds on cell cycle progression was evaluated. After 48 h of treatment, flow cytometry was performed on cells and the resulting cell fractions were sorted by DNA content. Treatment with 5-FU (30 µM), AKBA (100 µM), BA-Ex (100 µM), and nano-BA-Ex (10 µM) produced noticeable changes in the distribution of cell cycle phases. Figure 6A illustrates representative histograms that display the post-treatment cell cycle distribution with the compounds mentioned. Furthermore, the sub-G1 cell fraction, an indicator of apoptotic cells, was assessed based on fluorescence intensity (Fig. 6B). Notably, the nano-BA-Ex, AKBA, and BA-Ex showed increased sub-G1 cell fractions compared to both the control and 5-FU.

Fig. 6
figure 6

A Representative histograms of cell cycle distribution, B column chart of Sub-G1 cell fraction in each compound treatment

In vivo

Nano-BA-Ex inhibits colitis-related mouse colon tumorigenesis

Pathological characteristics of colon lesions including necrotic index, levels of mitosis, and aberrant crypt foci (ACFs) were scored and compared among different groups (e.g., as the average mitoses amount in 10 high-power fields at the lesion) (Fig. 7). The results demonstrated a significant difference between the negative control, placebo, 5-FU, BA-Ex, nano-BA-Ex, and nano-BA-Ex/5-FU receiving groups regarding levels of necrosis, mitosis, and ACF. After scoring and staging, the groups that received 5-FU, BA-Ex, nano-BA-Ex, and nano-BA-Ex/5-FU showed a significant decrease in mitosis compared to the placebo group (p < 0.001) (Table 1). Although the nano-BA-Ex groups exhibited a higher reduction in mitosis rate compared to 5-FU and BA-Ex, the difference was not statistically significant.

Fig. 7
figure 7

Comparison of colon tissue indices of the different groups. Normal tissue presents in the control group. Aberrant crypt foci (ACF) with dysplasia are shown with the down arrow. The epithelium displays nuclear stratification with rounded nuclei. There is a marked depletion of goblet cells in the dysplastic crypts. Mitosis (right arrow) was observed in DMH and therapeutic groups. Magnification 40 × , H&E stain

Table 1 Comparison between the colon tissue necrosis, mitotic, and Aberrant Crypt Foci (ACFs) indices of the different groups against placebo

ACF was significantly reduced in the groups treated with 5-FU, BA-Ex, nano-BA-Ex, and nano-BA-Ex/5-FU compared to those treated with placebo (p < 0.001) (Table 1). Furthermore, the nano-BA-Ex receiving group showed a significantly lower number of ACF compared 5-FU and BA-Ex groups (p < 0.001). The difference was even more pronounced in the group that received a combination of nano-BA-Ex and 5-FU (p < 0.001).

Expression of mRNAs related to inflammation, survival, proliferation, invasion, and angiogenesis

The mRNA expression levels of several proteins involved in tumor cell processes including Bcl2, Cox2, CyclinD1, Mmp9, and Vegf were assessed using quantitative real-time polymerase chain reaction (qRT-PCR). Treatment with various formulations led to a minor alteration in gene expression in comparison to the placebo group. The differential fold-change values for each of these mRNAs between each group are presented in Table 2.

Table 2 Comparison of mRNA gene expression between each study groups against placebo

Discussion

This study aimed to utilize CS-SA-CaCl2 nanoparticles as a delivery system for BA derivatives and to evaluate their antiproliferative activity against cancer cells. The AKBA extraction from B. serrata gum resin using the TPP method followed by column chromatography was successful as confirmed by HPLC with an extraction efficiency of 12.64%. The preparation of pH-sensitive nano-BA-Ex using sodium alginate and chitosan with dual cross-linking via ionic pregelation and polyelectrolyte complexation resulted in approximately 100 nm-sized nanocapsules with a loading capacity of 6.9% and encapsulation efficiency of 75.16%.

Previous studies have extensively investigated the cytotoxic effect of AKBA on various colorectal cancer cell lines in vitro (Liu et al. 2006; Takahashi et al. 2012; Shen et al. 2012). However, the in vitro results of the present study revealed that nano-BA-Ex exhibited superior cytotoxicity against HT29 cells compared to BA-Ex, free AKBA, and 5-FU, even at one-tenth the concentration. It suggests that the nano-delivery system enhances the bioavailability and therapeutic efficacy of BA derivatives. In addition, nano-BA-Ex induced apoptosis in HT29 cells more effectively than the other treatments as evidenced by Annexin V staining and cell cycle phase analysis. These results suggest that nano-BA-Ex not only inhibits cell growth but also promotes apoptosis, which is a favorable outcome in cancer therapy.

Limited research has been conducted on the anticancer effects of nanoforms of AKBA. One study evaluated the effects of BAs and their nanoparticles on the proliferation of human hepatoma HepG2 cells. Two different preparation techniques were used: nanoprecipitation and an oil-in-water emulsion/solvent evaporation method. The particles prepared via nanoprecipitation exhibited significantly higher cytotoxicity (Elnawasany et al. 2023). Additionally, other structural modifications, such as replacing the acetoxy group with a beta-amino group, introducing an electron-withdrawing group at C-2, or adding a nitrogen-containing substitution in the A-ring, can improve the anticancer effects of AKBA and KBA (Li et al. 2017b, c; Csuk et al. 2015; Huang et al. 2018). In a recent study, 26 new derivatives of AKBA were synthesized using ethylenediamine as the linking chain on the carboxyl group. The derivatives that targeted mitochondria demonstrated the most effective antiproliferative action, up to 20 times stronger than AKBA (Li et al. 2022). Additionally, Shamraiz et al. synthesized heterocyclic derivatives of BAs and new monomers of AKBA and KBA, as well as bis-AKBA and KBA homodimers, and AKBA-KBA heterodimers (Shamraiz et al. 2020). The antiproliferative effects of these dimers were evaluated on several human cancer cell lines, including HT29. The study found that the homodimer of KBA was the most effective, followed by pyrazine derivatives of AKBA, monomers of AKBA, KBA, and homodimers of AKBA. Additionally, a formulation of KBA nanoparticles (KBA-NPs) was designed using the emulsion-diffusion-evaporation method based on poly-DL-lactide-co-glycolide in a separate study. The study evaluated the oral bioavailability and anti-inflammatory effects of KBA and KBA-NPs in the rat paw edema model. The results indicated that KBA-NPs had a sevenfold increase in bioavailability and a 1.7-fold increase in anti-inflammatory activity compared to KBA alone (Bairwa and Jachak 2016).

A self-nanoemulsifying drug delivery system (SNES) has been developed to enhance the oral bioavailability of B. serrata extract (Ting et al. 2018). SNES forms emulsions composed of isotropic mixtures of oil, surfactant, and/or co-surfactant with droplet sizes less than 200 nm. These emulsions are effective for oral delivery of bioactive compounds. Both KBA and AKBA exhibited a more than two-fold increase in aqueous solubility and bioaccessibility. In addition, pharmacokinetic evaluations demonstrated that SNES significantly improved the oral bioavailability of KBA and AKBA in mice, resulting in more than twice the bioavailability of the bulk oil solution.

The in vivo investigation of the present study showed that nano-BA-Ex inhibited colon tumorigenesis induced by DMH in mice and significantly reduced the number of aberrant crypt foci (ACFs). Molecular studies showed that nano-BA-Ex had a small effect on the expression of several proteins, including Bcl-2, a marker of tumor survival; COX-2, a mediator of inflammation; cyclin D1, associated with cell proliferation; MMP-9, a protein involved in invasion processes; and VEGF, a protein associated with angiogenesis. However, this effect was not statistically significant. In a previous study, V.R. Yadav et al. reported that AKBA significantly suppressed the expression of several biomarkers in orthotopically implanted CRC tumors in mice. These biomarkers included pro-inflammatory COX2, tumor survival markers bcl-2, bcl-xL, IAP-1 and survivin, proliferative cyclin D1, invasive MMP-9 and ICAM-1, and angiogenic VEGF and CXCR4 (Yadav et al. 2012). Later that year, this was confirmed by H. Liu et al. in the adenomatous polyposis coli multiple intestinal neoplasia (APCMin/+) mice (Liu et al. 2013). The study found that AKBA reduced over 60% of colon polyps and prevented their malignant progression. The chemo-preventive effect of AKBA is attributed to its multifaceted activities, which include antiproliferative, apoptosis-inducing, antiangiogenic, and anti-inflammatory effects achieved through the Wnt/β-catenin and NF-κB/cyclooxygenase-2 pathways inhibition.

Nanoformulations and their size can significantly influence gene expression, which may be relevant to therapeutic outcomes. A study demonstrated that a novel pH-responsive hesperidin nanoformulation exhibited anticancer effects on lung adenocarcinoma cells by targeting the Akt/mTOR and MEK/ERK pathways. This nanoformulation decreased the expression of signaling proteins and enhanced the expression of apoptotic markers by upregulating Bax and p53 genes and downregulating the anti-apoptotic gene Bcl-2 (Pavan and Prabhu 2024). Additionally, research indicated that Ag nanoparticles impacted the viability of MCF-7 and Vero cell lines and altered the expression of apoptotic genes. These nanoparticles selectively induced apoptosis and upregulated tumor suppressor genes (Sangour et al. 2021). The combination of our pH-sensitive nanoformulation with standard chemotherapy agents could enhance therapeutic outcomes, similar to the findings reported by E I. Salim, where egg serum albumin nanoparticles improved the efficacy of cetuximab against Caco-2 colon cancer cells while reducing systemic toxicity (Salim et al. 2023). Their study demonstrated that cetuximab-loaded albumin nanoparticles exhibited significant antitumor activity and induced apoptosis through the upregulation of pro-apoptotic genes (Caspase3 and Bax) and downregulation of the anti-apoptotic gene Bcl2, reinforcing the potential of targeted drug delivery systems in cancer therapy. These findings suggest that the combination of our Nano-BA-Ex formulation with chemotherapeutic agents may synergistically enhance anticancer effects by modulating key signaling pathways and apoptotic genes.

Conclusions

This study introduces CS-SA-CaCl2 nanoparticles as a novel delivery system for AKBA with potential anticancer properties. By addressing the limitations of conventional chemotherapy, the research lays the foundation for an effective drug delivery system that enhances the therapeutic efficacy of these derivatives. However, further research is necessary to optimize its formulation, safety, and efficacy, highlighting nano-BA-Ex's potential for future preclinical and clinical evaluation in CRC therapy.

Methods

Materials

The gum resin of Boswellia serrata was acquired from a local market in our region. The pure reference standard of AKBA was purchased from Wuhan ChemFaces Biochemical, China (Cat. No. CFN90531). Analytical grade N,N′-Dimethylhydrazine dihydrochloride (DMH), as well as all other chemicals, was purchased from Merck Chemicals located in Darmstadt, Germany. The HPLC-grade solvents were obtained from Dae-Jung, Korea.

Extraction and purification of B. serrata gum resin

AKBA was extracted from the gum resin of B. serrata using the TPP method, as previously described (Niphadkar et al. 2017), followed by column chromatographic purification. Briefly, the gum resin of B. serrata was ground in a mill and one gram of sifted powder was soaked in 20 mL of distilled water and stirred gently. Next, ammonium sulfate, at a concentration of 40% (w/v), was added, and the pH was adjusted to 6.0. Twenty milliliters of tert-butyl alcohol (t-butanol) was then added to the mixture prepared at 50 °C for 180 min. The centrifugation process was performed at 8000g for 20 min, resulting in the formation of three layers. A two mL sample was then slowly taken from the upper organic phase, and the concentrated extract was dried in a ventilated oven at 55 °C. The resulting extract was stored as a powder until further analysis.

To achieve greater extraction efficiency, the extraction product underwent column chromatography purification. The powder was suspended in methanol and then fractionated through column chromatography on silica gel. The column was eluted using a combination of hexane, chloroform, and methanol (1:1:0.2, v/v) as the elution solvent. Fractions containing AKBA were collected and analyzed using thin-layer chromatography (TLC), while extraction efficiency was assessed through the use of high-performance liquid chromatography (HPLC).

The extraction efficiency was evaluated with HPLC (Knauer, Germany) using a C18 column (4.6 × 250 mm, Waters). To measure the amount of AKBA in the resulting extract, 0.001 g of the extract was weighed, dissolved in 1 mL of methanol (1000 ppm), and subsequently filtered through an HPLC filter (0.45 µm). The resulting extract was then injected into the HPLC machine in a final volume of 20 µl. The mobile phase consisted of methanol, acetonitrile, acidified water, and orthophosphoric acid in a ratio of 55: 40:4.5: 0.5% (v/v) adjusted to pH = 4 with glacial acetic acid, and was used at a flow rate of 1 mL/min. Analysis was conducted via ultraviolet detection at 260 nm, with all experiments repeated thrice.

Preparation of pH-sensitive BA-extract-loaded nanocapsules: (Nano-BA-Ex)

BA-Ex-loaded dual cross-linking nanocapsules were synthesized via the ionic pregelation and polyelectrolyte complexation method using two natural macromolecules, sodium alginate, and chitosan. The procedure was modified from the method outlined by Choukaife et al. (2020). First, 100 mL of 0.06% sodium alginate was prepared with deionized water under mechanical stirring for half an hour, following which the pH was adjusted to 4.9. Then, 1 mL of methanol containing 25 mg of BA-Ex powder was added to the mixture. After stirring for 30 min, 20 mL of 0.067% calcium chloride was sprayed into the above and further stirred for 30 min to form calcium alginate nanoparticles. Then, 15 mL of 0.005% low molecular weight chitosan (pH adjusted to 4.6) was sprayed into the mixture under continuous stirring for an additional 30 min to form double cross-linked particles. The final homogeneous suspension was dried under a vacuum at 50 °C. The loading capacity and encapsulation efficiency of nano-BA-Ex were determined using Eqs. (1) and (2) (Massella et al. 2018):

$${\text{Loading}}\,{\text{Capacity}}\% \, = \,\left( {W_{{\text{total BA}} - {\text{Ex added}}} --W_{{\text{free non}} - {\text{entrapped BA}} - {\text{Ex}}} /W_{{\text{total nano}} - {\text{BA}} - {\text{Ex}}} } \right)\, \times \,{1}00,$$
(1)
$${\text{Encapsulation}}\,{\text{Efficiency}}\% \, = \,\left( {W_{{\text{loaded BA}} - {\text{Ex}}} /W_{{\text{total BA}} - {\text{Ex added}}} } \right)\, \times \,{1}00.$$
(2)

To confirm the pH-sensitive behavior of the Nano-BA-Ex formulation, the controlled release of AKBA from the nanoparticles under simulated gastric (low pH: 2.5) and intestinal (higher pH: 5) conditions was conducted. Nano-BA-Ex nanosuspension was provided with different pH values (2.5 and 5) and injected to the HPLC at distinct time intervals (15, and 30 min, 1, 2, and 4 h). The amount of AKBA was quantified through UV detection at 260 nm.

Nano-BA-Ex characterization

The nanocapsules loaded with BA-Ex were characterized based on their zeta potential and average size. The average size was determined via DLS, and the PDI was also measured. The zeta potential was obtained by analyzing the electrophoretic mobility of the nanocapsules in an aqueous suspension, using a Zetasizer model NANO-flex 180 DLS from Particle Metrix (Germany). Nanocapsules were observed utilizing a scanning electron microscope (FE-SEM; ZEISS Sigma model Sigma VP, Germany). The FTIR spectra of the BA-Ex, nanocarrier, and BA-Ex-loaded nanocapsules were acquired using the Nicolet Avatar 360 FTIR System (Thermo, USA) with a scanning range of 4000–400 cm−1.

In vitro analysis

Cell lines, growth medium, and treatment conditions

Human normal primary fibroblasts were obtained from the circumcision site, and the HT29 colon adenocarcinoma cell line was acquired from the Pasteur Institute of Iran. Fetal bovine serum (FBS), DMEM, and RPMI-1640 medium were purchased from BioIdea (Tehran, Iran). Annexin V, propidium iodide (PI) labeling kit, and MTT cell proliferation reagent, 3-(4,5-dimethylthiazole-2-yl)-2,5-diphenyltetrazolium bromide, were purchased from Roche Diagnostics (Mannheim, Germany).

Human normal primary fibroblasts and HT29 cells were cultured in DMEM and RPMI-1640 medium, supplemented with 10% FBS, respectively, and maintained at 37 °C in a 5% CO2 incubator. For treatment, cells were harvested using 0.05% trypsin/0.02% EDTA when they reached approximately 80% confluence and were sub-cultured in the corresponding medium. After overnight incubation for attachment, cells were treated with different concentrations of 5-FU, standard AKBA, BA-Ex, and nano-BA-Ex. The medium containing 5% DMSO was used as a control.

Cell proliferation assay (MTT)

Cell viability was assessed using the MTT assay, which quantifies the reduction of tetrazolium salt to formazan by viable cells (Mosmann 1983). The cells were seeded in 96-well plates at a density of 105 cells per well with 200 µL of medium. After the cells reached the desired confluency, the medium was removed and cells were incubated with different concentrations of each treatment for 48 h. Cells were then incubated with 50 µL of 0.5 mg/mL MTT for 3 h at 37 °C. The intracellular formazan crystals were dissolved in 150 µL of dimethyl sulfoxide, and the resulting solution was subjected to spectrophotometric analysis at 570 nm to determine the optical density. By comparing the absorbance of treated and untreated cells, the percentage of dissolved colored formazan products was calculated. Triplicate assays were performed, and each experiment was repeated three times for accuracy.

Flow-cytometric determination of apoptosis

Annexin V staining

The apoptosis assay was performed utilizing annexin V and PI staining. In summary, cells were treated with each test compound for 48 h, washed with cold PBS, and stained with Annexin V-FITC antibody and PI for 15 min at room temperature. Subsequently, the cells were assessed by flow cytometry, and fluorescence intensity was measured in the FITC and PI channels. The proportions of the cell population in different quadrants were analyzed with quadrant statistics.

Cell cycle phase analysis

Cell cycle analyses were performed by sorting different cell fractions using flow cytometry. Cells were treated with different test compounds, including 5-FU, AKBA, BA-Ex, and nano-BA-Ex, for 48 h. After trypsinization, the cells were suspended in ice-cold 70% ethanol in PBS and stored at − 20 °C until analysis. Then, the cells were fixed and incubated with RNase A (100 mg/mL) for 30 min at 37 °C, followed by staining with PI (50 mg/mL) in the dark on ice for another 30 min. Nuclear DNA content was quantified with a BD-LSR flow cytometer (Becton Dickinson, USA) equipped with electronic doublet discrimination using blue (488 nm) excitation from an Argon laser. Cell cycle phase distribution was analyzed using Flowjo software. The fluorescence intensity of the sub-G1 cell fraction indicated the population of apoptotic cells.

In vivo analysis

Animals and conditions

Male BALB/c mice aged 6–8 weeks were purchased from the Pasteur Institute of Iran and kept under standard conditions in animal care facilities. Five mice per cage were housed with unlimited access to food and water under constant environmental conditions, including equal periods of light and dark, at 20–22 °C with 60–70% relative humidity, for 7 days prior to the start of the experiment. All animal experiments were performed according to ARRIVE 2.0 guidelines (Sert et al. 2020).

Cancer induction

Cancer was induced by administering weekly intraperitoneal injections of 20 mg/Kg N,N′-dimethylhydrazine dihydrochloride (DMH) (Abe et al. 2009). After 15 weeks of weekly DMH injections, the mice were randomly assigned to different treatments (Table 3). Each treatment was administered for 8 weeks with weekly weighing and daily monitoring of disease symptoms. At week 24, all mice were intraperitoneally anesthetized using a combination of ketamine–xylazine (K, 75 mg/Kg; X, 25 mg/Kg), and blood was collected by cardiac puncture. Animals were then sacrificed, and colon tissues were removed. The colon of each mouse was collected, washed, and dissected to assess both macroscopic and microscopic inflammatory and neoplastic lesions. After macroscopic evaluation, the colons were divided. One part was fixed in 10% buffered formalin, embedded in paraffin, and subjected to H&E staining for further histopathological examination. The second was preserved in RNA Later at − 80 °C for molecular analysis.

Table 3 Different treatment for each group of mice

Histopathology

Colon tissues were soaked in 10% formalin buffer as described. They were then washed with normal saline, fixed, dehydrated, and embedded in paraffin. Blocks were cooled, and 5 μm sections were stained with H&E, then examined using an Olympus CX23 light microscope (Japan). The mitotic index, necrosis, and ACF were recorded for each sample and are expressed as the mean ± SD per sample.

RNA extraction, complementary DNA synthesis, and quantitative real-time polymerase chain reaction (qRT-PCR)

The mRNA expression levels of several proteins involved in tumor cell processes, including Cox-2 (related to inflammation), cyclin D1 (associated with proliferation), Mmp-9 (involved in invasion), Vegf (related to angiogenesis), and Bcl-2 (associated with apoptosis), were evaluated utilizing qRT-PCR after treatment with various formulas. Molecular transcript levels were quantified to estimate mRNA synthesis by calculating the fold change relative to the control. RNA was extracted from 20 mg of each colon tissue sample using the Parstous Total RNA Extraction Kit (Mashhad, Iran). Two-step reverse transcription-PCR was then conducted using first-strand complementary DNA, produced with the cDNA Synthesis Kit from Tehran, Iran. The final reaction volume was 20 µL and included a 1X concentration of SyberGreen gene expression assay in RealQ Plus 2X Master Mix Green without ROX (Stenhuggervej, Denmark). To serve as a negative control, RNase-free water was used in each run. Specific thermal cycler conditions were applied using a StepOne™ Real-Time PCR system. Primer sequences are displayed in Table 4.

Table 4 Primer sequences for qPCR

Statistical analysis

Statistical analysis was performed using the SPSS software 23 (SPSS, Inc., Chicago, Illinois) and GraphPad Prism 8 (GraphPad Software). Data are expressed as mean ± standard deviation (SD). A one-way ANOVA was used to make comparisons between groups, followed by Tukey's multiple comparison tests. The level of statistical significance was set at a p value below 0.05 (two tailed).