No Smooth Julia Sets for Polynomial Diffeomorphisms of ${\mathbb C}^2$ with Positive Entropy

Abstract

For any polynomial diffeomorphism f of ${\mathbb C}^2$ with positive entropy, the Julia set of f is never $C^1$ smooth as a manifold-with-boundary.

1 Introduction

There are several reasons why the polynomial diffeomorphisms of ${\mathbb C}^2$ form an interesting family of dynamical systems. One of these is the fact that there are connections with two other areas of dynamics: polynomial maps of ${\mathbb C}$ and diffeomorphisms of ${\mathbb R}^2$, which have each received a great deal of attention. Among the endomorphisms of ${\mathbb P}^k$, certain ones have more special, and regular, geometric structure.

The question arises whether, among the polynomial diffeomorphisms of ${\mathbb C}^2$, are there analogous special maps with special geometry? The Julia set of such a special map would be expected to have some smoothness. Here we show that this does not happen.

More generally, we consider a holomorphic mapping $f:X\rightarrow X$ of a complex manifold X. The Fatou set of f is defined as the set of points $x\in X$ where the iterates $f^n:=f\circ \cdots \circ f$ are locally equicontinuous. If X is not compact, then in the definition of equicontinuity, we consider the one point compactification of X; in this case, a sequence which diverges uniformly to infinity is equicontinuous. By the nature of equicontinuity, the dynamics of f is regular on the Fatou set. The Julia set is defined as the complement of the Fatou set, and this is where any chaotic dynamics of f will take place. The first nontrivial case is where $X={\mathbb P}^1$ is the Riemann sphere, and in this case Fatou (see [17]) showed that if the Julia set J is a smooth curve, then either J is the unit circle or J is a real interval. If J is the circle, then f is equivalent to $z\mapsto z^d$, where d is an integer with $|d|\ge 2$; if J is the interval, then f is equivalent to a Chebyshev polynomial. These maps with smooth J play special roles, and this sparked our interest to look for smooth Julia sets in other cases. (The higher dimensional case is discussed, for instance, in Nakane [19] and Uchimura [23, 24].)

Here we address the case where $X={\mathbb C}^2$, and f is a polynomial automorphism, which means that f is biholomorphic, and the coordinates are polynomials. Since f is invertible, there are two Julia sets: $J^+$ for iterates in forward time, and $J^-$ for iterates in backward time. Polynomial automorphisms have been classified by Friedland and Milnor [12]; every such automorphism is conjugate to a map which is either affine or elementary, or it belongs to the family $\mathcal{H}$. The affine and elementary maps have simple dynamics, and $J^\pm$ are (possibly empty) algebraic sets (see [12]).

Thus we will restrict our attention to the maps in $\mathcal{H}$, which are finite compositions $f:=f_k\circ \cdots \circ f_1$, where each $f_j$ is a generalized Hénon map, which by definition has the form $f_j(x,y)=(y, p_j(y)-\delta _j x)$, where $\delta _j\in {\mathbb C}$ is nonzero, and $p_j(y)$ is a monic polynomial of degree $d_j\ge 2$. The degree of f is $d:=d_1\cdots d_k$, and the complex Jacobian of f is $\delta :=\delta _1\cdots \delta _k$. In [12] and [22], it is shown that the topological entropy of f is $\log d>0$. The dynamics of such maps is complicated and has received much study, starting with the papers [3, 11, 14, 15].

For maps in $\mathcal{H}$, we can ask whether $J^+$ can be a manifold. For any saddle point q, the stable manifold $W^s(q)$ is a Riemann surface contained in $J^+$. Thus $J^+$ would have to have real dimension at least two. However, $J^+$ is also the support of a positive, closed current $\mu ^+$ with continuous potential, and such potentials cannot be supported on a Riemann surface (see [3, 11]). On the other hand, since $J^+=\partial K^+$ is a boundary, it cannot have interior. Thus dimension 3 is the only possibility for $J^+$ to be a manifold. In fact, there are examples of f for which $J^+$ has been shown to be a topological 3-manifold (see [8, 11, 16, 20]). Fornæss and Sibony [11] have shown that $J^+$ cannot be smooth for a generic element of $\mathcal{H}$.

The purpose of this paper is to prove the following:

Theorem

For any polynomial automorphism of ${\mathbb C}^2$ of positive entropy, neither $J^+$ nor $J^-$ is smooth of class $C^1$, in the sense of manifold-with-boundary.

We may interchange the roles of $J^+$ and $J^-$ by replacing f by $f^{-1}$, so there is no loss of generality if we consider only $J^+$.

In an Appendix, we discuss the nonsmoothness of the related sets J, $J^*$, and K.

2 No Boundary

Let us start by showing that if $J^+$ is a $C^1$ manifold-with-boundary, then the boundary is empty. Recall that if $J^+$ is $C^1$, then for each $q_0\in J^+$ there is a neighborhood $U\ni q_0$ and $r,\rho \in C^1(U)$ with $dr\wedge d\rho \ne 0$ on U, such that $U\cap J^+=\{r=0, \rho \le 0\}$. If $J^+$ has boundary, it is given locally by $\{r=\rho =0\}$. For $q\in J^+$, the tangent space $T_qJ^+$ consists of the vectors that annihilate dr. This contains the subspace $H_q\subset T_qJ^+$ consisting of the vectors that annihilate $\partial r$. $H_q$ is the unique complex subspace inside $T_qJ^+$, so if $M\subset J^+$ is a complex submanifold, then $T_qM=H_q$.

We start by showing that if $J^+$ is $C^1$, then it carries a Riemann surface lamination.

Lemma 2.1

If $J^+$ is $C^1$ smooth, then $J^+$ carries a Riemann surface foliation $\mathcal{R}$ with the property that if $W^s(q)$ is the stable manifold of a saddle point q, then $W^s(q)$ is a leaf of $\mathcal{R}$. If $J^+$ is a $C^1$ smooth manifold-with-boundary, then $\mathcal{R}$ extends to a Riemann surface lamination of $J^+$. In particular, any boundary component is a leaf of $\mathcal{R}$.

Proof

Given $q_0\in J^+$, let us choose holomorphic coordinates (z, w) such that $dr(q_0) = dw$. We work in a small neighborhood which is a bidisk $\Delta _\eta \times \Delta _\eta$. We may choose $\eta$ small enough that $|r_z/r_w|<1$. In the (z, w)-coordinates, the tangent space $H_q$ has slope less than 1 at every point $\{|z|,|w|<\eta \}$. Now let $\hat{q}$ be a saddle point, and let $W^s(\hat{q})$ be the stable manifold, which is a complex submanifold of ${\mathbb C}^2$, contained in $J^+$. Let M denote a connected component of $W^s(\hat{q})\cap (\Delta _\eta \times \Delta _{\eta /2})$. Since the slope is <1, it follows that there is an analytic function $\varphi :\Delta _\eta \rightarrow \Delta _\eta$ such that $M\subset \Gamma _\varphi :=\{(z,\varphi (z)):z\in \Delta _\eta \}$. Let $\Phi$ denote the set of all such functions $\varphi$. Since a stable manifold can have no self-intersections, it follows that if $\varphi _1,\varphi _2\in \Phi$, then either $\Gamma _{\varphi _1}=\Gamma _{\varphi _2}$ or $\Gamma _{\varphi _1}\cap \Gamma _{\varphi _2}=\emptyset$. Now let $\hat{\Phi }$ denote the set of all normal limits (uniform on compact subsets of $\Delta _\eta$) of elements of $\Phi$. We note that by Hurwitz’s Theorem, the graphs $\Gamma _\varphi$, $\varphi \in \hat{\Phi }$ have the same pairwise disjointness property. Finally, by [4], $W^s(q_0)$ is dense in $J^+$, so the graphs $\Gamma _\varphi$, $\varphi \in \hat{\Phi }$ give the local Riemann surface lamination.

If $q_1$ is another saddle point, we may follow the same procedure and obtain a Riemann surface lamination whose graphs are given locally by $\varphi \in \hat{\Phi }_1$. However, we have seen that the tangent space to the foliation at a point q is given by $H_q$. Since these two foliations have the same tangent spaces everywhere, they must coincide.

We have seen that all the graphs are contained in $J^+$, so if $J^+$ has boundary, then the boundary must coincide locally with one of the graphs. $\square$

We will use the observation that $K^+\subset \{(x,y)\in {\mathbb C}^2:|y|>\max (|x|,R)\}$. Further, we will use the Green function $G^-$ which has many properties, including

(i):

$G^-$ is pluriharmonic on $\{G^->0\}$,

(ii):

$\{G^-=0\}=K^-$, and

(iii):

$G^-\circ f = d^{-1}G^-$.

Further, the restriction of $G^-$ to $\{|y|\le \max (|x|,R)\}$ is a proper exhaustion.

Lemma 2.2

Suppose that $J^+$ is a $C^1$ smooth manifold-with-boundary, and M is a component of the boundary of $J^+$. Then M is a closed Riemann surface, and $M\cap K\ne \emptyset$.

Proof

We consider the restriction $g:=G^-|_{M}$. If $M\cap K=\emptyset$, then g is harmonic on M. On the other hand, g is a proper exhaustion of M, which means that $g(z)\rightarrow \infty$ as $z\in M$ leaves every compact subset of M. This means that g must assume a minimum value at some point of M, which would violate the minimum principle for harmonic functions. $\square$

Lemma 2.3

Suppose that $J^+$ is a $C^1$ smooth manifold-with-boundary. Then the boundary is empty.

Proof

Let M be a component of the boundary of $J^+$. By Lemma 2.2, M must intersect $\Delta _R^2$. Since $J^+$ is $C^1$, there can be only finitely many boundary components of $J^+\cap \Delta _R^2$. Thus there can be only finitely many components M, which must be permuted by f. If we take a sufficiently high iterate $f^N$, we may assume that M is invariant. Now let $h:=f^N|_{M}$ denote the restriction to M. We see that h is an automorphism of the Riemann surface M, and the iterates of all points of M approach $K\cap M$ in forward time. It follows that M must have a fixed point $q\in M$, and $|h'(q)|<1$. The other multiplier of Df at q is $\delta /h'(q)$.

We consider three cases. First, if $|\delta /h'(q)|>1$, then q is a saddle point, and $M=W^s(q)$. On the other hand, by [4], the stable manifold of a saddle points is dense in $J^+$, which makes it impossible for M to be the boundary of $J^+$. This contradiction means that there can be no boundary component M.

The second case is $|\delta /h'(q)|<1$. This case cannot occur because the multipliers are less than 1, so q is a sink, which means that q is contained in the interior of $K^+$ and not in $J^+$.

The last case is where $|\delta /h'(q)|=1$. In this case, we know that f preserves $J^+$, so Df must preserve $T_q(J^+)$. This means that the outward normal to M inside $J^+$ is preserved, and thus the second multiplier must be +1. It follows that q is a semi-parabolic/semi-attracting fixed point. It follows that $J^+$ must have a cusp at q and cannot be $C^1$ (see Ueda [25] and Hakim [13]). $\square$

3 Maps that Do Not Decrease Volume

We note the following topological result (see Samelson [21] for an elegant proof): If M is a smooth 3-manifold (without boundary) of class $C^1$ in ${\mathbb R}^4$, then it is orientable. This gives:

Proposition 3.1

For any $q\in M$, there is a neighborhood U about q so that $U-M$ consists of two components $\mathcal{O}_1$ and $\mathcal{O}_2$, which belong to different components of ${\mathbb R}^4-M$.

Proof

Suppose that $\mathcal{O}_1$ and $\mathcal{O}_2$ belong to the same component of ${\mathbb R}^4-M$. Then we can construct a simple closed curve $\gamma \subset {\mathbb R}^4$ which crosses M transversally at q and has no other intersection with M. It follows that the (oriented) intersection is $\gamma \cdot M=1$ (modulo 2). But the oriented intersection modulo 2 is a homotopy invariant (see [18]), and $\gamma$ is contractible in ${\mathbb R}^4$, so we must have $\gamma \cdot M=0$ (modulo 2). $\square$

Corollary 3.2

If $J^+$ is $C^1$ smooth, then f is an orientation preserving map of $J^+$.

Proof

$U^+:={\mathbb C}^2-K^+$ is a connected (see [15]) and thus it is a component of ${\mathbb C}^2-J^+$. Since f preserves $U^+$, it also preserves the orientation of $J^+$, which is $\pm \partial U^+$. $\square$

We recall the following result of Friedland and Milnor:

Theorem

([12]) If $|\delta |>1$, then $K^+$ has zero Lebesgue volume, and thus $J^+=K^+$. If $|\delta |=1$, then ${\mathrm{int}}(K^+)={\mathrm{int}}(K^-) = {\mathrm{int}}(K)$. In particular, there exists R such that $J^+=K^+$ outside $\Delta _R^2$.

Proof of Theorem in the case $|\delta |\ge 1$. Let $q\in J^+$ be a point outside $\Delta ^2_R$, as in the Theorem above. Then near q there must be a component $\mathcal{O}$, which is distinct from $U^+={\mathbb C}^2-K^+$. Thus $\mathcal{O}$ must belong to the interior of $K^+$. But by the Theorem above, the interior of $K^+$ is not near q. $\square$

4 Volume Decreasing Maps

Throughout this section, we continue to suppose that $J^+$ is $C^1$ smooth, and in addition we suppose that $|\delta |<1$. For a point $q\in J^+$, we let $T_q:=T_q(J^+)$ denote the real tangent space to $J^+$. We let $H_q:=T_q\cap i T_q$ denote the unique (one-dimensional) complex subspace inside $T_q$. Since $J^+$ is invariant under f, so is $H_q$, and we let $\alpha _q$ denote the multiplier of $D_qf|_{H_q}$.

Lemma 4.1

Let $q\in J^+$ be a fixed point. There is a $D_qf$-invariant subspace $E_q\subset T_q({\mathbb C}^2)$ such that $H_q$ and $E_q$ generate $T_q$. We denote the multiplier of $D_qf|_{E_q}$ by $\beta _q$. Thus $D_qf$ is linearly conjugate to the diagonal matrix with diagonal elements $\alpha _q$ and $\beta _q$. Further, $\beta _q\in {\mathbb R}$ and $\beta _q>0$.

Proof

We have identified an eigenvalue $\alpha _q$ of $D_qf$. If $D_qf$ is not diagonalizable, then it must have a Jordan canonical form $\left( \begin{array}{cc}\alpha _q &{} 1\\ 0&{} \alpha _q\end{array}\right)$. The determinant is $\alpha _q^2=\delta$, which has modulus less than 1. Thus $|\alpha _q|<1$, which means that q is an attracting fixed point and thus in the interior of $K^+$, not in $J^+$. Thus $D_qf$ must be diagonalizable, which means that $H_q$ has a complementary invariant subspace $E_q$. Since $E_q$ and $T_q$ are invariant under $D_qf$, the real subspace $E_q\cap T_q\subset E_q$ is invariant, too. Thus $\beta _q\in {\mathbb R}$. By Corollary 3.2, $D_qf$ will preserve the orientation of $T_q$, and so $\beta _q>0$. $\square$

Let us recall the Riemann surface foliation of $J^+$ which was obtained in Lemma 2.1. For $q\in J^+$, we let $R_q$ denote the leaf of $\mathcal{R}$ containing q. If q is a fixed point, then f defines an automorphism $g:=f|_{R_q}$ of the Riemann surface $R_q$. Since $R_q\subset K^+$, we know that the iterates of $g^n$ are bounded in a complex disk $q\in \Delta _q\subset R_q$. Thus the derivatives $(Dg)^n= D(g^n)$ are bounded at q. We conclude that $|\alpha _q|=|D_q(g)|\le 1$. If $|\alpha _q|=1$, then $\alpha _q$ is not a root of unity. Otherwise g is an automorphism of $R_q$ fixing q, and $Dg^n(q) =1$ for some n. It follows that $g^n$ must be the identity on $R_q$. This means that Rq would be a curve of fixed points for $f^n$, but by [FM] all periodic points of f are isolated, so this cannot happen.

Lemma 4.2

If $q\in J^+$ is a fixed point, then q is a saddle point, and $\alpha _q=\delta /d$, and $\beta _q=d$.

Proof

First we claim that $|\alpha _p|<1$. Otherwise, we have $|\alpha _q|=1$, and by the discussion above, this means that $\alpha _q$ is not a root of unity. Thus the restriction $g = f|_{R_q}$ is an irrational rotation. Let $\Delta \subset R_q$ denote a g-invariant disk containing q. Since $|\delta |=|\alpha _q\beta _q|=|\beta _q|$ has modulus less than 1, we conclude that f is normally attracting to $\Delta$, and thus q must be in the interior of $K^+$, which contradicts the assumption that $q\in J^+$.

Now we have $|\alpha _q|<1$, so if $|\beta _q|=1$, we have $\beta _q=1$, since $\beta _q$ is real and positive. This means that q is a semi-parabolic, semi-attracting fixed point for f. We conclude by Ueda [25] and Hakim [13] that $J^+$ has a cusp at q and thus is not smooth. Thus we conclude that $|\beta _q|>1$, which means that q is a saddle point.

Now since $E_q$ is transverse to $H_q$, it follows that $W^u(q)$ intersects $J^+$ transversally, and thus $J^+\cap W^u(q)$ is $C^1$ smooth. Let us consider the uniformization

$$\begin{aligned} \phi :{\mathbb C}\rightarrow W^u(q)\subset {\mathbb C}^2, \ \ \ \phi (0)=q, \ \ \ f\circ \phi (\zeta )=\phi (\lambda ^u\zeta ). \end{aligned}$$

The pre-image $\tau :=\phi ^{-1}(W^u(q)\cap J^+)\subset {\mathbb C}$ is a $C^1$ curve passing through the origin and invariant under $\zeta \mapsto \lambda ^u\zeta$. It follows that $\lambda ^u\in {\mathbb R}$, and $\tau$ is a straight line containing the origin. Further, $g^+:=G^+\circ \phi$ is harmonic on ${\mathbb C}-\tau$, vanishing on $\tau$, and satisfying $g^+(\lambda ^u\zeta )=d\cdot g^+(\zeta )$. Since $\tau$ is a line, it follows that $g^+$ is piecewise linear, so we must have $\lambda ^u=\pm d$. Finally, since f preserves orientation, we have $\lambda ^u=d$.

Lemma 4.3

There can be at most one fixed point in the interior of $K^+$. There are at least $d-1$ fixed points contained in $J^+$, and at each of these fixed points, the differential Df has multiplier of d.

Proof

Suppose that q is a fixed point in the interior of $K^+$. Then q is contained in a recurrent Fatou domain $\Omega$, and by [4], $\partial \Omega =J^+$. If there is more than one fixed point in the interior of $K^+$, we would have $J^+$ simultaneously being the boundary of more than one domain, in addition to being the boundary of $U^+={\mathbb C}^2-K^+$. This is not possible if $J^+$ is a topological submanifold of ${\mathbb C}^2$.

By [FM] there are exactly d fixed points, counted with multiplicity. By Lemma 4.3, the fixed points in $J^+$ are of saddle type, so they have multiplicity 1. Thus there are at least $d-1$ of them. $\square$

5 Fixed Points with Given Multipliers

If $q=(x,y)$ is a fixed point for $f=f_n\circ \cdots \circ f_1$, then we may represent it as a finite sequence $(x_j,y_j)$ with $j\in {\mathbb Z}/n{\mathbb Z}$, subject to the conditions $(x,y)=(x_1,y_1)=(x_{n+1},y_{n+1})$ and $f_j(x_j,y_j)=(x_{j+1},y_{j+1})$. Given the form of $f_j$, we have $x_{j+1}=y_j$, so we may drop the $x_j$’s from our notation and write $q=(y_n,y_1)$. We identify this point with the sequence $\hat{q}=(y_1,\dots ,y_n)\in {\mathbb C}^n$, and we define the polynomials

$$\begin{aligned} \varphi _1&:= p_1(y_1)-\delta _1 y_n - y_2 \\ \varphi _2&:=p_2(y_2) - \delta _2 y_1 - y_3\\&\dots \dots \dots \\ \varphi _n&:= p_n(y_n) - \delta _n y_{n-1} - y_1. \end{aligned}$$

The condition to be a fixed point is that $\hat{q}=(y_1,\dots ,y_n)$ belongs to the zero locus $Z(\varphi _1,\dots ,\varphi _n)$ of the $\varphi _i$’s. We define $q_i(y_i):=p_i(y_i) - y_i^{d_i}$ and $Q_i := q_i(y_i) - y_{i+1}-\delta _i y_{i-1}$, so

$$\begin{aligned} \varphi _i = y_i^{d_i} + q_i(y_i) - y_{i+1} - \delta _i y_{i-1} = y_i^{d_1} + Q_i \end{aligned}$$

(*)

Since $p_j$ is monic, the degrees of $q_i$ and $Q_i$ are $\le d_i-1$.

By the Chain Rule, the differential of f at $q=(y_n,y_1)$ is given by

$$\begin{aligned} Df(q)=\left( \begin{array}{cc} 0&{}1\\ -\delta _n&{} p'_n(y_n)\end{array}\right) \cdots \left( \begin{array}{cc}0&{}1\\ -\delta _1&{} p'_n(y_1)\end{array}\right) \end{aligned}$$

We will denote this by $M_n=M_n(y_1,\dots ,y_n):= \left( \begin{array}{cc}m_{11}^{(n)} &{} m_{12}^{(n)}\\ m_{21}^{(n)} &{} m_{22}^{(n)} \end{array}\right)$.

We consider special monomials in $p'_j=p'_j(y_j)$ which have the form $(p')^L:= p'_{\ell _1}\cdots p'_{\ell _s}$, with $L=\{\ell _1,\dots ,\ell _s\}\subset \{1,\dots ,n\}$. Note that the factors $p'_{\ell _i}$ in $(p')^L$ are distinct. Let us use the notation |L| for the number of elements in L and $H_\mathbf{m}$ for the linear span of $\{(p')^L: |L|= m-2k, 0\le k\le {n/ 2} \}$. With this notation, m indicates the maximum number of factors of $p'_j$ in any monomial, and in every case the number of factors differs from m by an even number.

Lemma 5.1

The entries of $M_n$:

• (1) $m_{11}^{(n)}$ and $m_{22}^{(n)}- p'_1(y_1)\cdots p'_n(y_n)$ both belong to $H_\mathbf{n-2}$.

• (2) $m_{12}^{(n)}, m_{21}^{(n)} \in H_\mathbf{n-1}$.

Proof

We proceed by induction. The case $n=1$ is clear. If $n=2$,

$$\begin{aligned} M_2=\left( \begin{array}{cc}0&{}1\\ -\delta _2 &{} p'_2\end{array}\right) \left( \begin{array}{cc}0 &{} 1\\ -\delta _1 &{} p'_1\end{array}\right) = \left( \begin{array}{cc}-\delta _1 &{} p'_1\\ -\delta _1p_2'&{} p'_1p_2'-\delta _2\end{array}\right) \end{aligned}$$

which satisfies (1) and (2). For $n>2$, we have

$$\begin{aligned} M_n = \left( \begin{array}{cc}0&{}1\\ -\delta _n &{} p'_n\end{array}\right) M_{n-1} = \left( \begin{array}{cc}m_{21}^{(n-1)} &{} m_{22}^{(n-1)}\\ -\delta _n m_{11}^{(n-1)} + m_{21}^{(n-1)} p'_n &{} -\delta _n m_{12}^{(n-1)} + p'_n m_{22}^{(n-1)}\end{array}\right) \end{aligned}$$

which gives (1) and (2) for all n. $\square$

The condition for Df to have a multiplier $\lambda$ at q is $\Phi (\hat{q})=0$, where

$$\begin{aligned} \Phi = \det \left( M_n-\left( \begin{array}{cc}\lambda &{} 0\\ 0&{} \lambda \end{array}\right) \right) \end{aligned}$$

Lemma 5.2

$\Phi -p'_1(y_1)\cdots p'_n(y_n) \in H_\mathbf{n-2}$.

Proof

The formula for the determinant gives

$$\begin{aligned} \Phi = \lambda ^2 - \lambda {\mathrm{Tr}}(M_n) + {\mathrm{det}}(M_n) = \lambda ^2 - \lambda (m_{11}^{(n)} + m_{22}^{(n)}) + \delta \end{aligned}$$

since $\delta$ is the Jacobian determinant of Df. The Lemma now follows from Lemma 5.1. $\square$

The degree of the monomial $y^a:= y_1^{a_1}\cdots y_n^{a_n}$ is ${\mathrm{deg}}(y^a)=a_1+\cdots + a_n$. We will use the graded lexicographical order on the monomials in $\{y_1,\dots ,y_n\}$. That is, $y^a> y^b$ if either ${\mathrm{deg}}(y^a)>{\mathrm{deg}}(y^b)$, or if ${\mathrm{deg}}(y^a)={\mathrm{deg}}(y^b)$ and $a_i>b_i$, where $i = \min \{1\le j\le n: a_j\ne b_j\}$. If $f\in {\mathbb C}[y_1,\dots ,y_n]$, we denote LT(f) for the leading term of f, LC(f) for the leading coefficient, and LM(f) for the leading monomial.

Lemma 5.3

With the graded lexicographical order, $G:=\{\varphi _1,\dots ,\varphi _n\}$ is a Gröbner basis.

Proof

We will use Buchberger’s Algorithm (see [10, Chapter 2]). For each $i=1,\dots ,n$, LT$(\varphi _i)= \mathrm{LM}(\varphi _i)=y_i^{d_i}$, so for $i\ne j$, the least common multiple of the leading terms is $\mathrm{L.C.M.} = y_i^{d_i}y_j^{d_j}$. The S-polynomial is

$$\begin{aligned} S(\varphi _i,\varphi _j) := {\mathrm{L.C.M.} \over \mathrm{LM}(\varphi _j)}\varphi _i-{\mathrm{L.C.M.}\over \mathrm{LM}(\varphi _i)}\varphi _j = y_j^{d_j}Q_i-y_i^{d_i}Q_j = \varphi _jQ_i-Q_j\varphi _i \end{aligned}$$

where we use the $Q_j$ from (4.1) and cancel terms. Now let $\mu _i:={\mathrm{deg}}(Q_i)$. Since $\mu _i<d_i$ for all i, the monomials LM$(\varphi _jQ_i)=y_j^{d_j}y_i^{\mu _i}$ and LM$(\varphi _iQ_j)=y_i^{d_i}y_j^{\mu _j}$ are not equal in our monomial ordering. Thus LM$(S(\varphi _i,\varphi _j)\ge \max (\mathrm{LM}(\varphi _j Q_i), \mathrm{LM}(\varphi _iQ_j))$. It follows from Buchberger’s Algorithm that $\{\varphi _1,\dots ,\varphi _n\}$ is a Gröbner basis. $\square$

We will use the Multivariable Division Algorithm, by which any polynomial $g\in {\mathbb C}[y_1,\dots ,y_n]$ may be written as $g=A_1\varphi _1 + \cdots + A_n\varphi _n + R$ where LM$(g)\ge \mathrm{LM}(A_j\varphi _j)$ for all $1\le j\le n$, and R contains no terms divisible by any LM$(\varphi _j)$. An important property of a Gröbner basis is that g belongs to the ideal $\langle \varphi _1,\dots ,\varphi _n\rangle$ if and only if $R=0$ (see, for instance, [10] or [1]).

If all fixed points have the same value of $\lambda$ as multiplier, then it follows that $\Phi$ must vanish on the whole zero set $Z(\varphi _1,\dots ,\varphi _n)$. Since we have a Gröbner basis, we easily determine the following:

Corollary 5.4

$\Phi \notin \langle \varphi _1,\dots ,\varphi _n\rangle$.

Proof

The leading monomial of $\Phi$ is $y_1^{d_1-1}\cdots y_n^{d_n-1}$, but this is not divisible by any of the leading monomials LM$(\varphi _j)=y_j^{d_j}$. Since $\{\varphi _1,\dots ,\varphi _n\}$ is a Gröbner basis, it follows that $\Phi$ does not belong to the ideal $\langle \varphi _1,\dots ,\varphi _n\rangle$. $\square$

6 Proof of the Theorem

In this section we prove the Theorem, which will follow from 4.3, in combination with:

Proposition 6.1

Suppose $F=f_n\circ \cdots \circ f_1$, $n\ge 3$, is a composition of generalized Hénon maps with $|\delta |<1$. Suppose that F has $d=d_1\cdots d_n$ distinct fixed points. It is not possible that $d-1$ of these points have the same multipliers.

Proof that Proposition 6.1 implies the Theorem

To prove the Theorem, it remains to deal with the case $|\delta |<1$. If $f=f_1$ is a single generalized Hénon map, we consider $F=f_1\circ f_1\circ f_1$ with $n=3$ and the same Julia set. Lemma 3.4 asserts that if $J^+$ is $C^1$, there are $d-1$ saddle points with unstable multiplier $\lambda =d$. So by Proposition 4.1, we conclude that $J^+$ cannot be $C^1$ smooth. $\square$

We give the proof of Proposition 6.1 at the end of this section. For $J\subset \{1,\dots ,n\}$, we write

$$\begin{aligned} \Lambda _J:=\{(p')^L: L\subset J, |L|=|J|-2k, {\mathrm{for~some }}, 1\le k\le |J| / 2\}, \end{aligned}$$

We let $H_J$ denote the linear span of $\Lambda _J$. To compare with our earlier notation, we note that $H_J\subset H_\mathbf{|J|-2}$ and that $(p')^J\notin H_J$. The elements of $H_J$ depend only on the variables $y_j$ for $j\in J$. Now we formulate a result for dividing certain terms by $\varphi _j$:

Lemma 6.2

Suppose that $J\subset \{1,\dots ,n\}$ and $h\in H_J$. Then for each $j\in J$ and $\alpha \in {\mathbb C}$, we have

where $\rho _1,\rho _2\in H_{J-\{j\}}$, and $B=\eta _j(y_j) + d_j y_{j+1} + d_j\delta _j y_{j-1}$ with

The leading monomials satisfy

$$\begin{aligned} LM \left( (y_j-\alpha ) \left( (p')^J+h \right) \right) =LM(A(y)\varphi _j) \end{aligned}$$

Proof

Let us start with the case $J=\{1,\dots ,m\}$, $m\le n$, and $j=1$, so $J-\{j\}=J_{\hat{1}}=\{2,\dots ,n\}$. We divide by $p'_1$ and remove any factor of $p'_1$ in h. This gives

$$\begin{aligned} (p')^J + h = p'_1(y_1)\mu _1 + \rho _2 \end{aligned}$$

where $\mu _1=(p')^{J_{\hat{1}}} +\rho _1$, $\rho _1,\rho _2\in H_{\{2,\dots ,m\}}$, and $\mu _1$, $\rho _1$, $\rho _2$ are independent of the variable $y_1$. Thus

$$\begin{aligned} (y_1-\alpha )\left( (p')^J + h\right)&= (y_1-\alpha )(d_1 y_1^{d_1-1} + q'_1(y_1)) \mu _1 + (y_1-\alpha )\rho _2 \\&= d_1 y_1^{d_1}\mu _1 + (y_1 q'_1(y_1) - \alpha p_1'(y_1))\mu _1 + (y_1-\alpha )\rho _2\\&= (d_1\mu _1)\varphi _1 + (\eta _1(y_1) + d_1 y_2 + d_1\delta _1 y_n) \mu _1 + (y_1-\alpha ) \rho _2 \end{aligned}$$

where in the last line we substitute $\eta _1$ defined by $(\ddag )$. Using $(*)$, we see that this gives $(\dag )$.

It remains to look at the leading terms of $T_1:= (y_1-\alpha )\left( (p')^J + h) \right)$ and $T_2:= d_1\mu _1\varphi _1$. We see that $T_1$ and $T_2$ both contain nonzero multiples of $y_j \prod _{i=1}^m y_i^{d_i -1}$, and all other monomials in $T_1$ and $T_2$ have lower degree. Thus we have $LM(T_1)=LM(T_2)$ for the graded ordering, independent of any ordering on the variables $y_1,\dots ,y_n$. The choices of $J=\{1,\dots ,m\}$ and $j=1$ just correspond to a permutation of variables, and this does not affect the conclusion that $LM(T_1)=LM(T_2)$. $\square$

Lemma 6.3

For any $\alpha \in {\mathbb C}$, $(y_1-\alpha )\Phi \notin \langle \varphi _1,\dots ,\varphi _n\rangle$.

Proof

By [FM], we may assume that $p_j(y_j)=y_j^{d_j} + q_j(y_j)$ and ${\mathrm{deg}}(q_j)\le d_j-2$. We consider two cases. The first case is that there is at least one j such that $\eta _j$ is not the zero polynomial. If we conjugate by $f_{j-1}\circ \cdots \circ f_1$, we may “rotate” the maps in f so that the factor $f_j$ becomes the first factor. If there exists a j for which $\eta _j(y_j)$ is nonconstant, we choose this for $f_1$. Otherwise, if all the $\eta _j$ are constant, we choose $f_1$ to be any factor such that $\eta _1\ne 0$.

We will apply the Multivariate Division Algorithm on $(y_1-\alpha )\Phi$ with respect to the set $\{\varphi _1,\dots ,\varphi _n\}$. We will find that there is a nonzero remainder, and since $\{\varphi _1,\dots ,\varphi _n\}$ is a Gröbner basis, it will follow that $(y_1-\alpha )\Phi$ does not belong to the ideal $\langle \varphi _1,\dots ,\varphi _n\rangle$.

We start with Lemma 5.2, according to which $\Phi = p_1'\cdots p_n' + h$, where $h\in H_\mathbf{n-2} = H_{\{1,\dots ,n\}}$. The leading monomial of $(y_1-\alpha )\Phi$ is $y_1^{d_1}\prod _{i=2}^n y_i^{d_i-1}$, and $\varphi _1$ is the only element of the basis whose leading monomial divides this. Thus we apply Lemma 6.2, with $J=\{1,\dots ,n\}$, $j=1$, and $J_{\hat{1}}:= J-\{j\}=\{2,\dots ,n\}$. This gives

$$\begin{aligned}&(y_1-\alpha )\Phi = A_1 \varphi _1 + (\eta _1(y_1) + d_1 y_2 + d_1\delta _1 y_n) \left( \prod _{i=2}^n p_i'(y_i) +\rho _1 \right) + (y_1-\alpha )\rho _2\\&\quad = A_1 \varphi _1 +\left[ d_1y_2 \left( (p')^{J_{\hat{1}}}+\rho _1 \right) \right] \\&\qquad + \left[ d_1\delta _1 y_n\left( (p')^{J_{\hat{1}}}+\rho _1 \right) \right] + \left[ \eta _1\left( (p')^{J_{\hat{1}}}+\rho _1 \right) \right] + \ell .o.t \\&\quad =A_1 \varphi _1 + T_2 + T_n + R_1 + \ell .o.t \end{aligned}$$

where $\rho _1,\rho _2\in H_{\{2,\dots ,n\}}$. In particular, $T_2$ and $T_n$ depend on $y_2,\dots ,y_n$ but not on $y_1$. We note that $T_2$ (respectively, $T_n$) contains a term divisible by LM$(\varphi _2)$ (respectively, LM$(\varphi _n)$). We view $R_1$ as a remainder term, and note that LM$(R_1)$ is divisible by $y_2^{d_2-1}\cdots y_n^{d_n-1}$, as well as the largest power of $y_1$ in $\eta _1(y_1)$. By “$\ell .o.t.$,” we mean that none of its monomials is divisible by LM$(R_1)$ or by any of the LM$(\varphi _j)$.

Now we apply Lemma 6.2 to $T_2$, this time with $J=\{2,\dots ,n\}$ and $j=2$, with $J-\{2\} = J_{\hat{1}\hat{2}} = \{3,\dots ,n\}$. We have

$$\begin{aligned} T_2 =&A_2\varphi _2 + d_2y_3((p')^{J_{\hat{1}\hat{2}}}+ \rho _1^{(2)}) + d_2\delta _2 y_1 \left( (p')^{J_{\hat{1}\hat{2}} } + \rho _1^{(2)} \right) + \eta _2(y_2) (p')^{J_{\hat{1}\hat{2}}} + \ell .o.t.\\ =&A_2\varphi _2 + T_2^{(2)} + R_1^{(2)} + R_2^{(2)} + \ell .o.t. \end{aligned}$$

We see that $T_2^{(2)}$ contains terms that are divisible by LM$(\varphi _3)$, but the monomials in $R_1^{(2)}$ and $R_2^{(2)}$ are not divisible by LM$(\varphi _i)$ for any i. The remainder term here is $R_1^{(2)}+R_2^{(2)}$, and we observe that this cannot cancel the largest term in $R_1$. This is because LM$(R_1^{(2)})$ lacks a factor of $y_2$, and LM$(R_2^{(2)})$ is equal to $y_3^{d_3-1}\cdots y_n^{d_n-1}$ times the largest power of $y_2$ in $\eta _2(y_2)$, and by $(\ddag )$, this power is no bigger than $d_2-1$. If $\eta _1$ is not constant, then we see that LM$(R_1)>\mathrm{LM}(R_2^{(2)})$. If $\eta _1$ is constant, then $\eta _2$ must be constant, too, and again we have LM$(R_1)>\mathrm{LM}(R_2^{(2)})$. Thus, with our earlier notation, $R_1^{(2)} + R_2^{(2)} = \ell .o.t.$

We do a similar procedure with $T_n$, $T_2^{(2)}$, etc., and again find that the remainder term does not contain a multiple of the leading monomial of $R_1$. We see that each time we do this process, the size of the exponent L decreases in the term $(p')^L$. When we have $L=\emptyset$, there are no terms that can be divided by any LM$(\varphi _j)$. Thus we end up with

$$\begin{aligned} (y_1-\alpha )\Phi =A_1\varphi _1 + \cdots + A_n\varphi _n + R_1 + \ell .o.t. \end{aligned}$$

and LT$((y_1-\alpha )\Phi )\ge \mathrm{LT}(A_j\varphi _j)$ for all $1\le j\le n$, and none of the remaining terms is divisible by any of the leading monomials of $\varphi _j$. Thus we have now finished the Multivariate Division Algorithm, and we have a nonzero remainder. Thus $(y_1-\alpha )\Phi$ does not belong to the ideal of the $\varphi _j$’s.

Now we turn to the second case, in which $\eta _j=0$ for all j. By [12], we may assume that ${\mathrm{deg}}(q_j)\le d_j-2$. It follows that $\alpha =0$ and $q_j=0$. Thus $p_j = y_j^{d_j}$ for all $1\le j\le n$, so $p_j' = d_j y_j^{d_j-1}$, and $H_J$ consists of linear combinations of products $(p')^I= y_{i_1}^{d_{i_1}-1}\cdots y_{i_k}^{d_{i_k}-1}$ for $I = \{i_1,\dots ,i_k\}\subset J$, for even $k\le |J|-2$. We will go through the Multivariate Division Algorithm again. The principle is the same as before, but the details are different; in the first case we needed $n\ge 2$, and now we will need $n\ge 3$.

Again, it is only $\varphi _1$ which has a leading monomial which can divide some terms in $(y_1-\alpha )\Phi$. As before, we apply Lemma 6.2 with $J=\{1,\dots ,n\}$, $j=1$, and $J-\{1\}=J_{\hat{1}}=\{2,\dots ,n\}$. The polynomial in $(\ddag )$ becomes $B=d_jy_{j+1} + d_j\delta _j y_{j-1}$, and we have

$$\begin{aligned} y_1\Phi&= A_1\varphi _1 + d_1y_2 \left( (p')^{J_{\hat{1}}} + \rho _1\right) + d_1\delta _1 y_n\left( (p')^{J_{\hat{1}}}+ \rho _1\right) + y_1\rho _2\\&= A_1\varphi _1 + T_2 + T_n + \ell .o.t. \end{aligned}$$

where $\rho _1,\rho _2\in H_{\{2,\dots ,n\}}$. Now we apply Lemma 6.2 to divide $T_2$ (respectively, $T_n$) by $\varphi _2$ (respectively, $\varphi _n$). This yields

$$\begin{aligned} y_1\Phi =A_1\varphi _1+ A_2\varphi _2 + A_n\varphi _n + T_3+T_n+ R+\ell .o.t., \end{aligned}$$

where

$$\begin{aligned} T_3= d_1d_2y_3\left( (p')^{J_{\hat{1}\hat{2}} }+ \tilde{\rho }_3 \right) , \ \ \ T_n= d_1d_n\delta _1\delta _n y_{n-1} \left( (p')^{J_{\hat{1}\hat{n}}} + \tilde{\rho }_n \right) \end{aligned}$$

with $\tilde{\rho }_3\in H_{\{3,\dots ,n\}}$ and $\tilde{\rho }_n\in H_{\{2,\dots ,n-1\}}$, and

$$\begin{aligned} R = \left( d_1d_2\delta _2 y_1 y_n^{d_n-1} + d_1d_n\delta _1 y_1 y_{2}^{d_2-1} \right) \prod _{i=3}^{n-1} y_i^{d_i-1} \end{aligned}$$

Since $n>2$, R is not the zero polynomial. We will continue the Multivariate Division Algorithm by dividing $T_3$ by $\varphi _3$ and $T_n$ by $\varphi _n$, but we see that any terms created cannot cancel R. Thus when we finish the Multivariable Division Algorithm, we will have a nonzero remainder. As in the previous case, we conclude that $y_1\Phi$ is not in the ideal $\langle \varphi _1,\dots ,\varphi _n\rangle$.

Proof of Proposition 6.1

The fixed points of f coincide with the elements of $Z(\varphi _1,\dots ,\varphi _n)$, which is a variety of pure dimension zero. Saddle points have multiplicity 1, and since there are $d-1$ of these, and since the total multiplicity is d, there must be one more fixed point, also of multiplicity 1. It follows that the ideal $I:=\langle \varphi _1,\dots ,\varphi _n\rangle$ is equal to its radical (see [1]). Since the saddle points all have multiplier $\lambda$, $\Phi$ must vanish at all the saddle points. If $(\alpha ,\beta )$ is the other fixed point, we conclude that $(y_1-\alpha )\Phi$ vanishes at all the fixed points. Thus $(y_1-\alpha )\Phi$ belongs to the radical of I, and thus I itself. This contradicts Lemma 6.3, which completes the proof of Proposition 6.1. $\square$

Appendix: Nonsmoothness of J, \(J^*\), and K

Let us turn our attention to other dynamical sets for polynomial diffeomorphisms of positive entropy. These are $J:=J^+\cap J^-$, $K:= K^+\cap K^-$, and the set $J^*$, which coincides with the closure of the set of periodic points of saddle type. (See [3, 5], and [2] for other characterizations of $J^*$.) We have $J^*\subset J\subset K$. We note that none of these sets can be a smooth 3-manifold: otherwise, for any saddle point p, it would be a bounded set containing $W^s(p)$ or $W^u(p)$, which is the holomorphic image of ${\mathbb C}$. The following was suggested by Remark 5.9 of Cantat in [9]; we sketch his proof:

Proposition 6.1

If $J=J^*$, then it is not a smooth 2-manifold.

Proof

Let p be a saddle point, and let $W^u(p)$ be the unstable manifold. The slice $J \cap W^u(p)$ is smooth and invariant under multiplication by the multiplier of Df. This means that in fact, the multiplier must be real, and the restriction of $G^+$ to the slice must be linear on each (half-space) component of $W^u(p)- J$.

The identity $G^+\circ f = d\cdot G^+$ means that the canonical metric (defined in [6]) is multiplied by d. Thus f is quasi-expanding on $J^*$. Now, applying this argument to $f^{-1}$ we get that f is quasi-hyperbolic. Further, $J^*=J$, so it is quasi-hyperbolic on J. If f fails to be hyperbolic, then by [7] there will be a one-sided saddle point, which cannot happen since J is smooth.

Now that f is hyperbolic on J, there is a splitting $E^s \oplus E^u$ of the tangent bundle, so we conclude that J is a 2-torus. The dynamical degree must be the spectral radius of an invertible 2-by-2 integer matrix, but this means it is not an integer, which contradicts the fact the dynamical degree of a Hénon map is its algebraic degree.

Proposition 6.2

Suppose that the complex Jacobian is not equal to $\pm 1$. Then for each saddle (periodic) point p and each neighborhood U of p, neither $J\cap U$ nor $J^*\cap U$ nor $K\cap U$ is a $C^1$ smooth 2-manifold.

Proof

Let us write $M:=J\cap U$ and $g:=f|_M$. (The following argument works, too, if we take $M=J^*\cap U$ or $M=K\cap U$.) The tangent space $T_pM$ is invariant under Df. The stable/unstable spaces $E^{s/u}\subset T_p{\mathbb C}^2$ are invariant under $D_pf$. The space $E^s$ (or $E^u$) cannot coincide with $T_pM$, for otherwise the complex stable manifold $W^s(p)$ (or $W^u(p)$) would be locally contained in M, and thus globally contained in J. But the $W^{s/u}$ are uniformized by ${\mathbb C}$, whereas J is bounded. We conclude that p is a saddle point for g, and thus the local stable manifold $W^s_{\mathrm{loc}}(p; g)$ is a $C^1$-curve inside the complex stable manifold $W^s(p)$. As in Lemma 4.3, we conclude that the multiplier for $D_pf|_{E^u_p}$ is $\pm d$ and the multiplier for $D_pf|_{E^s_p}$ is $\pm 1/d$. Thus the complex Jacobian is $\delta =\pm 1$.

Solenoids The two results above concern smoothness, but no example is known where J, $J^*$, or K is even a topological 2-manifold. In the cases where $J^+$ has been shown to be a topological 3-manifold (see [8, 11, 16, 20]) it also happens that J is a (topological) real solenoid, and in these cases it is also the case that $J=J^*$. Further, for every saddle (periodic) point p, there is a real arc $\gamma _p=W^u_{\mathrm{loc}}(p)\cap J$. If we apply the argument of Proposition 6.2 to this case, we conclude that $\gamma _p$ is not $C^1$ smooth.