1 Introduction

Throughout this paper, unless specifically stated, R always denotes a prime ring of characteristic different from 2. Let U be a Utumi ring of quotients and C be its center known as the extended centroid of R. An additive mapping \(d: R \rightarrow R\) is said to be a derivation on R if \(d(xy)=d(x)y+xd(y)\) for all \(x,y \in R.\) Motivated by elementry operators in the theory of operators Algebra, Bresar [3] has introduced the concept of generalized derivations, which is a generalization of derivation. A generalized derivation F is an additive mapping on R with \(F(xy)= F(x)y+xd(y)\) for all \(x,y \in R,\) where d is a derivation on R. Clearly, every derivation is generalized derivation but not conversely. A polynomial \(f = f(x_1,\ldots ,x_n) \in \mathbb {Z}<X>\) is said to be multilinear if it is linear in every \(x_i,\) \(1 \le i \le n,\) where \(\mathbb {Z}\) is the set of integers.

In [11], Giambruno and Herstien proved that if R is a prime ring and d is a derivation on R such that \(d(x)^n=0\) for all \(x\in R,\) where n is a fixed positive integer, then \(d=0.\) Bresar et al. [2] has extended Herstien result by taking a sequence of different derivations in place of single derivation. Precisely, it is proved that, Let R be a prime ring with infinite extended centroid. If derivations \(d_1,d_2,d_3,\ldots ,d_n\) of R satisfy \(d_1(x)d_2(x)...d_n(x)=0\), for all \(x \in R,\) then \(d_i=0\) for some i. Later, similar situations considered in [18, 19].

In this sequence, Fosner and Vukman [10], have proved that if \(F_1\) and \(F_2\) are generalized derivations of a prime ring R of characteristic different from 2,  such that\(F_1(x)F_2(x) = 0\) for all \(x \in R\), then there exist elements p,  q of the Martindale quotient ring Q of R such that \(F_1(x) = xp\) and \(F_2(x)=qx\) for all \(x \in R\) and \(pq=0\) except when at least one \(F_i\) is zero. Moreover, above identity studied by Carini et al [5] by taking multilinear polynomial. They have proved the following:

Let R be a non-commutative prime ring of characteristic different from 2 with Utumi quotient ring U and extended centroid C,  \(f(x_1,x_2,\ldots ,x_n)\) a multilinear polynomial over C which is not an identity for R, F and G two non-zero generalized derivations on R. If \(F(u)G(u) = 0\) for all \(u\in f(R) = \{f(r_1,r_2,\ldots ,r_n) : r_i \in R \},\) then one of the following holds:

  1. (1)

    There exist \(a,c \in U\) such that \(~ac = 0\) and \(F(x) = xa, G(x) = cx\) for all \(x \in R\);

  2. (2)

    \(f(x_1,x_2,\ldots ,x_n)^2\) is central valued on R and there exist \(a,c \in U\) such that \(ac = 0\) and \(F(x) = ax,\) \(G(x) = xc\) for all \(x\in R\);

  3. (3)

    \(f(x_1,x_2,\ldots ,x_n)\) is central valued on R and there exist \(a, b, c, q \in U\) such that \((a + b)(c + q) = 0\) and \(F(x) = ax + xb,\) \(G(x) = cx + xq\) for all \(x \in R.\)

Here in this article, we have studied the identity \(G^2(u)d(u)=0,\) for all \(u \in f(R)= \{f(r_1,r_2,\ldots ,r_n) : r_i \in R \}\), where G is a generalized derivation and d is a non zero derivation on prime ring R of characteristic different from 2. More preisely, we have proved the following:

Theorem 1.1

Let R be a prime ring of characteristic different from 2 with Utumi quotient ring U and extended centroid C, \(f(x_1,\ldots ,x_n)\) be a multilinear polynomial over C, which is not central valued on R. Suppose that d is a nonzero derivation of R and G is a generalized derivation on R. If \(G^2(u)d(u)=0\) for all \(u\in f(R)\), then one of the following holds:

  1. (i)

    there exists \(a\in U\) such that \(G(x)=ax\)  for all \(x\in R\)  with \(a^2=0\),

  2. (ii)

    there exists \(a\in U\) such that \(G(x)=xa\)  for all \(x\in R\)  with \(a^2=0\).

2 Preliminaries

We will use frequently some important theory of generalized polynomial identities and differential identities. We recall some of the facts.

Fact-1: Every derivation d of R can be uniquely extended to a derivation of U (see Proposition 2.5.1 [1]).

Fact-2: If I is a two-sided ideal of R, then R, I and U satisfy the same differential identities ( [15]).

Fact-3: If I is a two-sided ideal of R, then R, I and U satisfies the same generalized polynomial identities with coefficients in U ([6]).

Fact-4: (Kharchenko [13, Theorem 2]) Let R be a prime ring, d a nonzero derivation on R and I a nonzero ideal of R. If I satisfies the differential identity

$$\begin{aligned} f(r_1,r_2,\ldots ,r_n,d(r_1),d(r_2),\ldots ,d(r_n))=0 \end{aligned}$$

for any \(r_1,r_2,\ldots ,r_n \in I\), then either

  1. (i)

    I satisfies the generalized polynomial identity

    $$\begin{aligned} f(r_1,r_2,\ldots ,r_n,x_1,x_2,\ldots ,x_n)=0 \end{aligned}$$

    or

  2. (ii)

    d is Q-inner i.e., for some \(q \in Q,\ d(x)=[q,x]\) and I satisfies the generalized polynomial identity

    $$\begin{aligned} f(r_1,r_2,\ldots ,r_n,[q,r_1],[q,r_2],\ldots ,[q,r_n])=0. \end{aligned}$$

Fact-5: We shall use the following notation:

$$\begin{aligned} f(x_1,\ldots ,x_n)=x_1x_2\ldots x_n+\sum \limits _{\sigma \in S_n, \sigma \ne id} \alpha _\sigma x_{\sigma (1)}x_{\sigma (2)} \ldots x_{\sigma (n)} \end{aligned}$$

for some \(\alpha _\sigma \in C\) and \(S_n\) the symmetric group of degree n.

Let d be a derivation. We denote by \(f^d(x_1,\ldots ,x_n)\), \(f^{d^2}(x_1,\ldots ,x_n)\) the polynomials obtained from \(f(x_1,\ldots ,x_n)\) replacing each coefficients \(\alpha _\sigma\) with \(d(\alpha _\sigma )\) and \(d^2(\alpha _\sigma )\) respectively. Then we have

$$\begin{aligned} d(f(x_1,\ldots ,x_n))=f^d(x_1,\ldots ,x_n)+\sum \limits _if (x_1,\ldots ,d(x_i),\ldots ,x_n) \end{aligned}$$

and

$$\begin{aligned}&d^2(f(x_1,\ldots ,x_n))=f^{d^2}(x_1,\ldots ,x_n) +2\sum \limits _if^d(x_1,\ldots ,d(x_i),\ldots ,x_n)\\&\quad +\sum \limits _if(x_1,\ldots ,d^2(x_i),\ldots ,x_n)+\sum \limits _{i\ne j}f(x_1, \ldots , d(x_i), \ldots , d(x_j),\ldots , x_n). \end{aligned}$$

3 The case when d and G are an inner

First, we study the situation when both d and G are an inner. Let \(d(x)=[P,x]\) for all \(x\in R\) be an inner derivation on R and \(G(x)=ax+xb\) for all \(x\in R\) be an inner generalized derivation on R for some \(P, a, b \in U\). Then \(G^2(f(r))d(f(r))= 0\) for all \(r=(r_1,\ldots ,r_n) \in R^n\) implies

$$\begin{aligned}&(a^2f(r)+2af(r)b+f(r)b^2)Pf(r)-(a^2f(r)+2af(r)b+f(r)b^2)f(r)P=0. \end{aligned}$$

This gives

$$\begin{aligned}&a'f(r)Pf(r)+2af(r)b'f(r)+f(r)b''f(r)-a'f(r)^2P\\&\quad -2af(r)bf(r)P-f(r)cf(r)P=0 \end{aligned}$$

for any \(r=(r_1,\ldots ,r_n) \in R^n\), where \(a'=a^2, b'=bP, b''=b^2P, c=b^2\).

To prove main results, we need the following.

Lemma 3.1

[7, Lemma 1] Let C be an infinite field and \(m \ge 2\). If \(A_1, \ldots , A_k\) are not scalar matrices in \(M_m(C)\) then there exists some invertible matrix \(B \in M_m(C)\) such that any matrices \(BA_1B^{-1}, \ldots , BA_kB^{-1}\) have all non-zero entries.

The following lemma is a particular case of Theorem 1.1 of [4].

Lemma 3.2

Let R be a prime ring of characteristic different from 2, \(Q_r\) its right Martindale quotient ring, and C its extended centroid. Suppose that F is a generalized derivation and d is a non zero derivation on R and \(f(x_1,\ldots ,x_n)\) a noncentral multilinear polynomial over C with n noncommuting variables, such that \(F(f(r_1,\ldots ,r_n))d(f(r_1,\ldots ,r_n))=0\) for all \(r_1,r_2,\ldots , r_n \in R\), then \(F=0\).

Proposition 3.3

Let \(R=M_m(C)\) be the ring of all \(m \times m\) matrices over the field C, \(f(x_1, \ldots , x_n)\) a non-central multilinear polynomial over C and \(a, b, c, P, a', b', b'' \in R\). If  \(a'f(r)Pf(r)+2af(r)b'f(r)+f(r)b''f(r)-a'f(r)^2P-2af(r)bf(r)P-f(r)cf(r)P=0\) for all \(r=(r_1, \ldots , r_n)\in R^n\), then either P or a or b is central.

Proof

By our assumption, R satisfies the generalized identity

$$\begin{aligned}&a'f(r_1, \ldots , r_n)Pf(r_1, \ldots , r_n)+2af(r_1, \ldots , r_n)b'f(r_1, \ldots , r_n)\nonumber \\&\quad +f(r_1, \ldots , r_n)b''f(r_1, \ldots , r_n)-a'f(r_1, \ldots , r_n)^2P-2af(r_1, \ldots , r_n)bf(r_1, \ldots , r_n)P\nonumber \\&\quad -f(r_1, \ldots , r_n)cf(r_1, \ldots , r_n)P=0. \end{aligned}$$
(1)

We shall prove it by contradiction. Suppose that \(a\notin Z(R)\), \(b\notin Z(R)\) and \(P\notin Z(R)\).

Case-I: Suppose that C is infinite field. Since \(a \notin Z(R)\), \(b \notin Z(R)\) and \(P \notin Z(R)\), by Lemma 3.1 there exists a C-automorphism \(\phi\) of \(M_m(C)\) such that \(a_1=\phi (a)\), \(b_1=\phi (b)\) and \(P_1=\phi (P)\) have all non-zero entries. Clearly \(a_1\), \(b_1\), \(P_1\), \(c_1=\phi (c)\), \(a'_1=\phi (a')\), \(b'_1=\phi (b')\) and \(b''_1=\phi (b'')\) must satisfy the condition (1). Without loss of generality we may replace \(a, b, c, P, a', b', b''\) with \(a_1, b_1, c_1, P_1, a'_1, b'_1, b''_1\) respectively.

Here \(e_{ij}\) denotes the matrix whose (i, j)-entry is 1 and rest entries are zero. Since \(f(x_1, \ldots , x_n)\) is not central, by [15] (see also [16]), there exist \(u_1, \ldots , u_n \in M_m(C)\) and \(\gamma \in C-\{0\}\) such that \(f(u_1, \ldots , u_n)=\gamma e_{st}\), with \(s \ne t\). Moreover, since the set \(\{f(r_1, \ldots , r_n) : r_1, \ldots , r_n \in M_m(C)\}\) is invariant under the action of all C-automorphisms of \(M_m(C)\), then for any \(i \ne j\) there exist \(r_1, \ldots , r_n \in M_m(C)\) such that \(f(r_1, \ldots , r_n)=e_{ij}\). Hence by (1) we have

$$\begin{aligned} a'e_{ij}Pe_{ij}+2ae_{ij}b'e_{ij}+e_{ij}b''e_{ij} -2ae_{ij}be_{ij}P-e_{ij}ce_{ij}P=0. \end{aligned}$$

Right and left multiplying by \(e_{ij}\), we obtain \(2a_{ji}b_{ji}P_{ji}e_{ij}=0\). Since char \((R)\ne 2\), thus we have \(a_{ji}b_{ji}P_{ji}e_{ij}=0\). It implies either \(a_{ji}=0\) or \(b_{ji}=0\) or \(P_{ji}=0\). By Lemma 3.1, it gives a contradiction, since a, b and P have all non-zero entries. Thus we conclude that either a or b or P is central.

Case-II: Suppose C is finite field. Let K be an infinite field which is an extension of the field C. Let \(\overline{R}=M_m(K)\cong R\otimes _C K\). Notice that the multilinear polynomial \(f(x_1, \ldots , x_n)\) is central-valued on R if and only if it is central-valued on \(\overline{R}\). Suppose that the generalized polynomial \(Q(r_1,\ldots ,r_n)\) such that

$$\begin{aligned}&Q(r_1,\ldots ,r_n)= a'f(r_1, \ldots , r_n)Pf(r_1, \ldots , r_n)+2af(r_1, \ldots , r_n)b'f(r_1, \ldots , r_n)\nonumber \\&\quad +f(r_1, \ldots , r_n)b''f(r_1, \ldots , r_n)-a'f(r_1, \ldots , r_n)^2P-2af(r_1, \ldots , r_n)\nonumber \\&\quad bf(r_1, \ldots , r_n)P-f(r_1, \ldots , r_n)cf(r_1, \ldots , r_n)P \end{aligned}$$
(2)

is a generalized polynomial identity for R.

Moreover, it is a multi-homogeneous of multi-degree \((2,\ldots ,2)\) in the indeterminates \(r_1, \ldots , r_n\). Hence the complete linearization of \(Q(r_1, \ldots , r_n)\) is a multilinear generalized polynomial \(\Theta (r_1, \ldots , r_n, x_1, \ldots , x_n)\) in 2n indeterminates, moreover

$$\begin{aligned} \Theta (r_1, \ldots , r_n, r_1, \ldots , r_n)=2^n Q(r_1, \ldots , r_{n}). \end{aligned}$$

It is clear that the multilinear polynomial \(\Theta (r_1, \ldots , r_n, x_1, \ldots , x_n)\) is a generalized polynomial identity for both R and \(\overline{R}\). For assumption \(char(R)\ne 2\) we obtain \(Q(r_1, \ldots , r_{n})=0\) for all \(r_1, \ldots , r_{n} \in \overline{R}\) and then conclusion follows from Case-I.

Lemma 3.4

Let R be a prime ring of characteristic different from 2 with Utumi quotient ring U and extended centroid C, and \(f(x_1,\ldots , x_n)\) a multilinear polynomial over C, which is not central valued on R. Suppose that for some \(a, b, c, P, a', b', b'' \in R\), \(a'f(r)Pf(r)+2af(r)b'f(r)+f(r) b''f(r)-a'f(r)^2P-2af(r)bf(r)P-f(r)cf(r)P=0\) for all \(r=(r_1, \ldots , r_n)\in R^n\), then either a or b or P is central.

Proof

Let \(P\notin C\), \(a \notin C\) and \(b\notin C\). By hypothesis, we have

$$\begin{aligned}&h(x_1,\ldots ,x_n)= a'f(x_1, \ldots , x_n)Pf(x_1, \ldots , x_n)+2af(x_1, \ldots , x_n)b'f(x_1, \ldots , x_n)\nonumber \\&+f(x_1, \ldots , x_n)b''f(x_1, \ldots , x_n)-a'f(x_1, \ldots , x_n)^2P-2af(x_1, \ldots , x_n)\nonumber \\&bf(x_1, \ldots , x_n)P-f(x_1, \ldots , x_n)cf(x_1, \ldots , x_n)P=0 \end{aligned}$$
(3)

for all \(x_1, \ldots , x_n \in R\). Since R and U satisfy same generalized polynomial identity (GPI) (see [6]), U satisfies \(h(x_1,\ldots ,x_n)=0.\) Suppose that \(h(x_1,\ldots ,x_n)\) is a trivial GPI for U. Let \(T=U*_CC\{x_1,x_2,\ldots ,x_n\}\), the free product of U and \(C\{x_1,\ldots ,x_n\}\), the free C-algebra in noncommuting indeterminates \(x_1,x_2,\ldots , x_n\). Then, \(h(x_1,\ldots ,x_n)\) is zero element in \(T=U*_CC\{x_1,\ldots ,x_n\}\). Since \(P\notin C\), \(a\notin C\) and \(b\notin C\), the term \(2af(x_1, \ldots , x_n)bf(x_1, \ldots , x_n)P\) appears nontrivially in \(h(x_1,\ldots ,x_n)\). This gives a contradiction.

Next, suppose that \(h(x_1,\ldots ,x_n)\) is a non-trivial GPI for U. In case C is infinite, we have \(h(x_1,\ldots ,x_n)=0\) for all \(x_1,\ldots ,x_n \in U \otimes _C\overline{C}\), where \(\overline{C}\) is the algebraic closure of C. Since both U and \(U \otimes _C \overline{C}\) are prime and centrally closed [8, Theorems 2.5 and 3.5], we may replace R by U or \(U \otimes _C \overline{C}\) according to C finite or infinite. Then R is centrally closed over C and \(h(x_1,\ldots ,x_n)=0\) for all \(x_1,\ldots ,x_n \in R\). By Martindale’s theorem [17], R is then a primitive ring with nonzero socle soc(R) and with C as its associated division ring. Then, by Jacobson’s theorem [12, p.75], R is isomorphic to a dense ring of linear transformations of a vector space V over C. Assume first that V is finite dimensional over C, that is, \(\hbox {dim}_C V = m\). By density of R, we have \(R\cong M_m(C)\). Since \(f(r_1,\ldots ,r_n)\) is not central valued on R, R must be noncommutative and so \(m\ge 2\). In this case, by Proposition ??, we get that either a or b or P is in C, a contradiction. If V is infinite dimensional over C, then for any \(e^2=e\in soc(R)\) we have \(eRe\cong M_t(C)\) with \(t=\)dim\(_CVe\). Since P, a and b are not in C, there exist \(h_1,h_2,h_3\in soc(R)\) such that \([P,h_1]\ne 0\), \([a,h_2]\ne 0\) and \([b,h_3]\ne 0\). By Litoff’s Theorem [9], there exists idempotent \(e\in soc(R)\) such that \(Ph_1, h_1P, ah_2, h_2a, bh_3, h_3b, h_1, h_2, h_3\in eRe\). Since R satisfies generalized identity

$$\begin{aligned}&e\{a'f(ex_1e, \ldots , ex_ne)Pf(ex_1e, \ldots , ex_ne) +2af(ex_1e, \ldots , ex_ne)b'f(ex_1e, \ldots , ex_ne)\\&+f(ex_1e, \ldots , ex_ne)b''f(ex_1e, \ldots , ex_ne) -a'f(ex_1e, \ldots , ex_ne)^2P\\&-2af(ex_1e, \ldots , ex_ne)bf(ex_1e, \ldots , ex_ne)P\\&-f(ex_1e, \ldots , ex_ne)cf(ex_1e, \ldots , ex_ne)P\}e, \end{aligned}$$

the subring eRe satisfies

$$\begin{aligned}&ea'ef(x_1, \ldots , x_n)ePef(x_1, \ldots , x_n)+2eaef(x_1, \ldots , x_n)eb'ef(x_1, \ldots , x_n)\\&\quad +f(x_1, \ldots , x_n)eb''ef(x_1, \ldots , x_n)-ea'ef(x_1, \ldots , x_n)^2ePe-2eaef(x_1, \ldots , x_n)\\&\quad ebef(x_1, \ldots , x_n)ePe-f(x_1, \ldots , x_n)ecef(x_1, \ldots , x_n)ePe=0. \end{aligned}$$

Then by the above finite dimensional case, either ePe or eae or ebe is central element of eRe. This leads a contradiction, since \(Ph_1=(ePe)h_1=h_1ePe=h_1P\), \(ah_2=(eae)h_2=h_2(eae)=h_2a\) and \(bh_3=(ebe)h_3=h_3(ebe)=h_3b\). Thus, we have proved that either P or a or b is in C.

Lemma 3.5

Let R be a prime ring of characteristic different from 2 with Utumi quotient ring U and extended centroid C, \(f(x_1,\ldots ,x_n)\) be a multilinear polynomial over C, which is not central valued on R. Suppose that for some \(P, a, b\in U\), \(d(x)=[P,x]\) for all \(x\in R\) is a nonzero inner derivation of R and \(G(x)=ax+xb\) for all \(x\in R\) is an inner generalized derivation of R. If \(G^2(f(r))d(f(r))=0\) for all \(r=(r_1,\ldots ,r_n)\in R^{n}\), then one of the following holds:

  1. (i)

    \(G(x)=(a+b)x\) for all \(x\in R\) with \((a+b)^2=0\),

  2. (ii)

    \(G(x)=x(a+b)\) for all \(x\in R\) with \((a+b)^2=0\).

Proof

By hypothesis, we have

$$\begin{aligned}&\big (a^2f(r)+2af(r)b+f(r)b^2\big )Pf(r)- \big (a^2f(r)+2af(r)b+f(r)b^2\big )f(r)P=0. \end{aligned}$$
(4)

That is

$$\begin{aligned}&a^2f(r)Pf(r)+2af(r)bPf(r)+f(r)b^2Pf(r)-a^2f(r)^2P\\&-2af(r)bf(r)P-f(r)b^2f(r)P=0 \end{aligned}$$

for all \(r=(r_1,\ldots ,r_n)\in R^n\). Since \(d\ne 0\), so \(P\notin C\), then by Lemma 3.4, either \(a\in C\) or \(b\in C\).

If \(a\in C\), then \(G(x)=x(a+b)\) for all \(x \in R\). Then by hypothesis, we have

$$\begin{aligned} f(r)(a+b)^2[P,f(r)]=0 \end{aligned}$$

for all \(r=(r_1,\ldots ,r_n) \in R^n\). Since \(d\ne 0\) so \(P\notin C\), from Lemma 3.2, it implies that \((a+b)^2=0\), which is our conclusion (ii).

If \(b\in C\), then \(G(x)=(a+b)x\). Hence hypothesis becomes

$$\begin{aligned} (a+b)^2f(r)[P, f(r)]=0 \end{aligned}$$

for all \(r=(r_1,\ldots ,r_n)\in R^n\). Since \(d\ne 0\) so \(P\notin C\), from Lemma 3.2, it implies that \((a+b)^2=0\), which gives our conclusion (i).

Lemma 3.6

Let R be a prime ring of characteristic different from 2 with Utumi quotient ring U and extended centroid C, \(f(x_1,\ldots ,x_n)\) be a multilinear polynomial over C, which is not central valued on R. Suppose that for some \(a, b\in U\), d is a nonzero derivation of R, and \(G(x)=ax+xb\) for all \(x\in R\) is an inner generalized derivation of R. If \(G^2(f(r))d(f(r))=0\) for all \(r=(r_1,\ldots ,r_n)\in R^{n}\), then one of the following holds:

  1. (i)

    \(G(x)=(a+b)x\) for all \(x\in R\) with \((a+b)^2=0\),

  2. (ii)

    \(G(x)=x(a+b)\) for all \(x\in R\) with \((a+b)^2=0\).

Proof

If d is an inner derivation, then by Lemma 3.5 we get our conclusions. Suppose d is not an inner derivation. Then hypothesis implies that

$$\begin{aligned} G^2(f(r_1,\ldots ,r_n))d(f(r_1,\ldots ,r_n))=0. \end{aligned}$$

That is

$$\begin{aligned}&\big (a^2f(r_1,\ldots ,r_n)+2af(r_1,\ldots ,r_n)b+f(r_1,\ldots ,r_n) b^2\big )d(f(r_1,\ldots ,r_n))=0. \end{aligned}$$
(5)

Since

$$\begin{aligned} d(f(r_1,\ldots ,r_n))=f^d(r_1,\ldots ,r_n)+ \sum \limits _if(r_1,\ldots ,d(r_i),\ldots ,r_n), \end{aligned}$$

by applying Kharchenko’s theorem (see Fact 4) to (5), we can replace \(d(f(r_1,\ldots ,r_n))\) with \(f^d(r_1,\ldots ,r_n)+\sum \limits _i f(r_1,\ldots ,y_i,\ldots , r_n)\) and then U satisfies

$$\begin{aligned}&\Big (a^2f(r_1,\ldots ,r_n)+2af(r_1,\ldots ,r_n) b+f(r_1,\ldots ,r_n)b^2\Big )\Big (f^d(r_1,\ldots ,r_n)\nonumber \\&+\sum \limits _i f(r_1,\ldots ,y_i,\ldots , r_n)\Big )=0. \end{aligned}$$
(6)

Hence U satisfies blended component

$$\begin{aligned}&\Big (a^2f(r_1,\ldots ,r_n)+2af(r_1,\ldots ,r_n)b+f(r_1,\ldots ,r_n) b^2\Big )\nonumber \\&\Big (\sum \limits _i f(r_1,\ldots ,y_i,\ldots , r_n)\Big )=0. \end{aligned}$$
(7)

Replacing \(y_i\) with \([q, r_i]\) for some \(q\notin C\), U satisfies

$$\begin{aligned}&\Big (a^2f(r_1,\ldots ,r_n)+2af(r_1,\ldots ,r_n) b+f(r_1,\ldots ,r_n)b^2\Big )\nonumber \\&\Big [q, f(r_1,\ldots , r_n)\Big ]=0. \end{aligned}$$
(8)

Equation (8) is same as Eq. (4). Hence from Lemma 3.5, we conclude our results. \(\square\)

Theorem 3.7

Let R be a prime ring of characteristic different from 2 with Utumi quotient ring U and extended centroid C, \(f(x_1,\ldots ,x_n)\) be a multilinear polynomial over C, which is not central valued on R. Suppose that d is a nonzero derivation of R and G is a generalized derivation of R. If \(G^2(u)d(u)=0\) for all \(u\in f(R)\), then one of the following holds:

  1. (i)

    there exists \(a\in U\) such that \(G(x)=ax\) for all \(x\in R\) with \(a^2=0\),

  2. (ii)

    there exists \(a\in U\) such that \(G(x)=xa\) for all \(x\in R\) with \(a^2=0\).

Proof

If G is an inner generalized derivation, then by Lemma 3.6 we get desired results.

Next we assume that G is not an inner generalized derivation. By [14, Theorem 3], we may assume that there exist derivations \(\delta\) on U, \(a\in U\) such that \(G(x)=ax+\delta (x)\). Since R and U satisfy the same generalized polynomial identities (see [6]) as well as the same differential identities (see [15]), without loss of generality, we have

$$\begin{aligned}&\Big ((a^2+\delta (a))f(r_1,\ldots ,r_n)+2a\delta (f(r_1,\ldots ,r_n)) +\delta ^2(f(r_1,\ldots ,r_n))\Big )\nonumber \\&d(f(r_1,\ldots ,r_n))=0 \end{aligned}$$
(9)

for all \(r_1,\ldots ,r_n \in U\). Now we consider two cases:

Cases-I: Let d and \(\delta\) be C-dependent modulo inner derivations of U, that is \(\alpha d+\beta \delta =ad_q\), where \(\alpha ,\beta \in C\), \(q\in U\) and \(ad_q(x)=[q,x]\) for all \(x\in U\). If \(\alpha =0\), then \(\delta =[q', x]\) for all \(x\in R\), where \(q'=\beta ^{-1} q\), which implies that \(\delta\) is an inner derivation. It implies that G is an inner generalized derivation, a contradiction. Hence \(\alpha \ne 0\), and hence \(d = \lambda \delta +ad_p\), where \(\lambda = \alpha ^{-1}\beta\) and \(p = \alpha ^{-1}q\). Then by hypothesis, we have

$$\begin{aligned}&\Big ((a^2+\delta (a))f(r_1,\ldots ,r_n)+2a\delta (f(r_1,\ldots ,r_n)) +\delta ^2(f(r_1,\ldots ,r_n))\Big )\nonumber \\&\Big (\lambda \delta (f(r_1, \ldots , r_n))+\Big [p, f(r_1,\ldots , r_n)\Big ]\Big )=0 \end{aligned}$$
(10)

for all \(r_1,\ldots ,r_n \in U\).

Since \(\delta (f(r_1,\ldots ,r_n))=f^{\delta }(r_1,\ldots ,r_n) +\sum \limits _if(r_1,\ldots ,\delta (r_i),\ldots ,r_n)\) and \(\delta ^2(f(r_1, \ldots , r_n))\) \(=f^{\delta ^2}(r_1,\ldots ,r_n)+2\sum \limits _if^\delta (r_1,\ldots ,\delta (r_i),\ldots ,r_n)+\sum \limits _if (r_1,\ldots ,\delta ^2(r_i),\ldots ,r_n) +\sum \limits _{i\ne j}f(r_1, \ldots , \delta (r_i), \ldots , \delta (r_j), \ldots , r_n)\). Hence our hypothesis becomes

$$\begin{aligned}&\Big ((a^2+\delta (a))f(r_1,\ldots ,r_n)+2af^{\delta }(r_1,\ldots ,r_n) +2a\sum \limits _if(r_1,\ldots ,\delta (r_i),\ldots ,r_n)\nonumber \\&+f^{\delta ^2}(r_1,\ldots ,r_n)+2\sum \limits _if^\delta (r_1,\ldots ,\delta (r_i),\ldots ,r_n)+\sum \limits _if (r_1,\ldots ,\delta ^2(r_i),\ldots ,r_n)\nonumber \\&+\sum \limits _{i\ne j}f(r_1, \ldots , \delta (r_i), \ldots , \delta (r_j), \ldots , r_n)\Big )\Big (\lambda f^{\delta }(r_1,\ldots ,r_n)\nonumber \\&+\lambda \sum \limits _if(r_1,\ldots ,\delta (r_i),\ldots ,r_n)+\Big [p, f(r_1,\ldots , r_n)\Big ]\Big )=0. \end{aligned}$$
(11)

By Kharchenko’s theorem (see Fact-4), we can replace \(\delta (r_i)\) with \(y_i\), \(\delta ^2(r_i)\) with \(z_i\) in (10), then U satisfies the blended component

$$\begin{aligned}&\Big (\sum \limits _if(r_1,\ldots ,z_i,\ldots ,r_n)\Big )\Big (\lambda f^{\delta }(r_1,\ldots ,r_n)\nonumber \\&+\lambda \sum \limits _if(r_1,\ldots ,y_i,\ldots ,r_n)+\Big [p, f(r_1,\ldots , r_n)\Big ]\Big )=0. \end{aligned}$$
(12)

In particular for \(y_i=0\) for all \(i=1,2,\ldots ,n\), U satisfies the blended component

$$\begin{aligned}&\Big (\sum \limits _if(r_1,\ldots ,z_i,\ldots ,r_n)\Big )\Big (\lambda \sum \limits _if(r_1,\ldots ,y_i,\ldots ,r_n)\Big )=0. \end{aligned}$$
(13)

Replacing \(y_i\) with \([q,r_i]\) for some \(q\notin C\) and \(z_1=r_1\), \(z_2=\cdots =z_n=0\), we have that U satisfies

$$\begin{aligned} f(r_1,\ldots ,r_n)\lambda \Big [q, f(r_1,\ldots ,r_n)\Big ]=0. \end{aligned}$$
(14)

Since \(q\notin C\), it implies that \(\lambda =0\). Hence (11) reduces to

$$\begin{aligned}&\Big ((a^2+\delta (a))f(r_1,\ldots ,r_n)+2af^{\delta } (r_1,\ldots ,r_n)+2a\sum \limits _if(r_1,\ldots , \delta (r_i),\ldots ,r_n)\nonumber \\&+f^{\delta ^2}(r_1,\ldots ,r_n)+2\sum \limits _if^\delta (r_1,\ldots ,\delta (r_i),\ldots ,r_n)+\sum \limits _if (r_1,\ldots ,\delta ^2(r_i),\ldots ,r_n)\nonumber \\&+\sum \limits _{i\ne j}f(r_1, \ldots , \delta (r_i), \ldots , \delta (r_j), \ldots , r_n)\Big )\Big [p, f(r_1,\ldots , r_n)\Big ]=0. \end{aligned}$$
(15)

Again using Kharchenko’s theorem (see Fact-4) and using Fact-5, U satisfies the blended component

$$\begin{aligned} \sum \limits _if(r_1,\ldots ,z_i,\ldots ,r_n)\Big [p, f(r_1,\ldots , r_n)\Big ]=0. \end{aligned}$$
(16)

Replacing \(z_i\) with \([q,r_i]\) for some \(q\notin C\), we have that U satisfies

$$\begin{aligned} \Big [q,f(r_1,\ldots ,\ldots ,r_n)\Big ]\Big [p, f(r_1,\ldots , r_n)\Big ]=0. \end{aligned}$$
(17)

Since \(q\notin C\), by Lemma 3.6, it gives \(p\in C\). It implies \(d=0\), a contradiction.

Case-II: Let d and \(\delta\) be C-independent modulo inner derivations of U. By Kharchenko’s theorem (see Fact-4) and using Fact-5, we can replace \(\delta (f(r_1,\ldots ,r_n))\) with \(f^{\delta }(r_1,\ldots ,r_n)+\sum \limits _if(r_1,\ldots ,z_i,\ldots ,r_n)\), \({d}(f(r_1,\ldots ,r_n))\) with \(f^{d}(r_1,\ldots ,r_n)+\sum \limits _i f(r_1,\ldots ,y_i,\ldots ,r_n)\) and \({\delta }{^2}(f(r_1,\ldots ,r_n))\) with \(f^{{\delta }^2}(r_1,\ldots ,r_n)+ 2\sum \limits _if^{\delta }(r_1,\ldots ,z_i,\ldots ,r_n) +\sum \limits _if(r_1,\ldots ,x_i,\ldots ,r_n)+\sum \limits _{i\ne j}f(r_1, \ldots , z_i, \ldots , z_j, \ldots , r_n)\), where \(\delta (r_i)=z_i\), \(d(r_i)=y_i\) and \(\delta ^2(r_i)=x_i\) in (9) and then U satisfies

$$\begin{aligned}&\Big ((a^2+\delta (a))f(r_1,\ldots ,r_n)+2af^{\delta } (r_1,\ldots ,r_n)+2a\sum \limits _if(r_1,\ldots ,z_i,\ldots ,r_n)\nonumber \\&+f^{{\delta }^2}(r_1,\ldots ,r_n) +2\sum \limits _if^{\delta }(r_1,\ldots ,z_i, \ldots ,r_n)+\sum \limits _if(r_1,\ldots ,x_i,\ldots ,r_n)\nonumber \\&+\sum \limits _{i\ne j}f(r_1, \ldots , z_i, \ldots , z_j, \ldots , r_n)\Big )\Big (f^{d}(r_1,\ldots ,r_n)\nonumber \\&+\sum \limits _i f(r_1,\ldots ,y_i,\ldots ,r_n)\Big )=0 \end{aligned}$$
(18)

for all \(r_1,\ldots ,r_n \in U\). In particular U satisfies the blended component

$$\begin{aligned}&\sum \limits _if(r_1,\ldots ,x_i,\ldots ,r_n)\sum \limits _i f(r_1,\ldots ,y_i,\ldots ,r_n)=0. \end{aligned}$$
(19)

Replacing \(x_i\) with \([q,r_i]\) and \(y_i\) with \([p,r_i]\) for some \(q\notin C\) and \(p\notin C\), we have that U satisfies

$$\begin{aligned}&\Big [q,f(r_1,\ldots ,\ldots ,r_n)\Big ]\Big [p, f(r_1,\ldots ,\ldots ,r_n\Big ]=0. \end{aligned}$$
(20)

This is same as equation (17). In this case, we get a contradiction. \(\square\)

In particular for \(G=I\) in theorem 3.7, I denotes an identity function on R, we have the following corollaries.

Corollary 3.8

Let R be a noncommutative prime ring of characteristic different from 2 with Utumi quotient ring U and extended centroid C, \(f(x_1,\ldots ,x_n)\) be a multilinear polynomial over C, which is not central valued on R. Suppose that d is a derivation on R such that \(f(x_1,\ldots , x_n)d(f(x_1, \ldots , x_n))=0\) for all \((x_1,\ldots , x_n)\in R^n\), then \(d=0\)

Corollary 3.9

Let R be a prime ring of characteristic different from 2, I a non zero ideal of R and d be a non zero derivation on R. If \(xd(x)=0\) for all \(x\in I\), then R is a commutative.

Again for \(G=g\), where g is a derivation on R, we have the following.

Corollary 3.10

Let R be a prime ring of characteristic different from 2, I a non zero ideal of R. Suppose that g and d a non zero derivations on R. If \(g^2(x)d(x)=0\) for all \(x\in I\), then R is a commutative.

4 Open problems

In this section, we will give some open problems. In the Theorem 3.7, we have studied the identity \(F^2(u)g(u)=0\) for all \(u\in f(R)\), where F is a generalized derivation and g is a derivation on prime ring R. The natural question will arise that what will happen if we replace derivation g with generalized derivation G on prime ring R? More precisely, the statement is given below.

Proposition 4.1

Let R be a prime ring and G and F are two generalized derivations on R. Let U be Utumi ring of quotient of R with extended centroid C. Suppose  \(f(x_1,\ldots ,x_n)\) is a multilinear polynomial over C which is not central valued on R such that  \(F^2(u)G(u)=0\) for all \(u\in f(R)\). Then find the structure of these additive mappings as well as prime ring R.

If we replace generalized derivation G with \(G^2\) in above problem, we have the following.

Proposition 4.2

Let R be a prime ring and G and F are two generalized derivations on R. Let U be Utumi ring of quotient of R with extended centroid C. Suppose \(f(x_1,\ldots ,x_n)\) is a multilinear polynomial over C which is not central valued on R such that \(F^2(u)G^2(u)=0\) for all \(u\in f(R)\). Then find the structure of these additive mappings as well as prime ring R.

Proposition 4.3

Let R be a prime ring and G and F are two generalized derivations on R. Let U be Utumi ring of quotient of R with extended centroid C. Suppose \(f(x_1,\ldots ,x_n)\) is a multilinear polynomial over C which is not central valued on R such that \(F^2(u)G(u)\in C\) for all \(u\in f(R)\). Then find the structure of these additive mappings as well as prime ring R.

Since we know that identity mapping is a generalized derivation on R. If we replace \(G=id\), where id is the identity mapping on R, in problem 4.3, then it will be [ Eroǧlu and Argaç Canad. Math. Bull. 2017; 60: 721–735].

Proposition 4.4

Let R be a prime ring and G and F are two generalized derivations on R. Let U be Utumi ring of quotient of R with extended centroid C. Suppose \(f(x_1,\ldots ,x_n)\) is a multilinear polynomial over C which is not central valued on R such that \(F^n(u)G^m(u)\in C\) (or \(F^n(u)G^m(u)=0\)) for all \(u\in f(R)\), where m and n are positive integers. Then find the structure of these additive mappings as well as prime ring R.