Picture Scanning Automata

Abstract

We are systematically discussing finite automata working on rectangular-shaped arrays (i.e., pictures), reading them with different scanning strategies. We show that all 32 different variants only describe two different classes of array languages. We also introduce and discuss Mealy machines working on pictures and show how these can be used in a modular design of picture processing devices.

1 Introduction

Syntactic considerations of digital images have a tradition of about five decades. They should (somehow) reflect methods applied to picture processing. However, two basic methods of scanning pictures in practice have not been thoroughly investigated from a more theoretical point of view: that of scanning pictures line by line and that of using space-filling curves. Here, we continue such an investigation with what can be considered as the most simple way of defining space-filling curves: scanning line after line of an image, alternating the direction of movement every time when the image boundary is encountered, as well as by scanning line by line following the same direction each time. We consider finite automata that work the described ways, extending previous studies [2]. This shows tighter connections between finite automata working on pictures and array grammars of different types; for overviews on this topic, we refer to [4, 5].

Our main contributions can be summarized as follows: (a) We show that an amazing variety of picture scanning devices basically only describe two different array language families. (b) This result is obtained by making use of connections to the theory of dihedral groups. (c) Further closure properties of array language families are derived. (d) We also introduce Mealy picture machines and show how they could be useful in a modular design of picture processing automata.

General Definitions. In the following, we briefly recall some standard definitions and notations regarding two-dimensional words and languages. A two-dimensional word (also called picture, matrix or array) over $\varSigma$ is a tuple

$$\begin{aligned} W := ((a_{1,1}, a_{1,2}, \ldots , a_{1,n}), (a_{2,1}, a_{2,2}, \ldots , a_{2,n}), \ldots , (a_{m,1}, a_{m,2}, \ldots , a_{m,n})), \end{aligned}$$

where $m, n \in \mathbb {N}$ and, for every i, $1 \le i \le m$, and j, $1 \le j \le n$, $a_{i, j} \in \varSigma$. We define the number of columns (or width) and number of rows (or height) of W by $|W|_c := n$ and $|W|_r := m$, respectively. We also denote W by $[a_{i, j}]_{m, n}$ or by a matrix in a more pictorial form. If we want to refer to the $j^{\text {th}}$ symbol in row i of the picture W, then we use $W[i, j] = a_{i, j}$. By $\varSigma ^{++}$, we denote the set of all (non-empty) pictures over $\varSigma$. Every subset $L \subseteq \varSigma ^{++ }$ is a picture language.

2 Operations on Pictures and Picture Languages

2.1 Binary Operations

Let $W := [a_{i, j}]_{m, n}$ and $W' := [b_{i, j}]_{m', n'}$ be two pictures over $\varSigma$. The column concatenation of W and $W'$, denoted by , is undefined if $m \ne m'$ and is otherwise obtained by writing W to the left of $W'$, yielding the picture

$$\begin{aligned} \begin{matrix} a_{1,1} &{} a_{1,2} &{} \ldots &{} a_{1,n} &{} b_{1,1} &{} b_{1,2} &{} \ldots &{} b_{1,n'} \\ a_{2,1} &{} a_{2,2} &{} \ldots &{} a_{2,n} &{} b_{2,1} &{} b_{2,2} &{} \ldots &{} b_{2,n'} \\ \vdots &{} \vdots &{} \ddots &{} \vdots &{} \vdots &{} \vdots &{} \ddots &{} \vdots \\ a_{m,1} &{} a_{m,2} &{} \ldots &{} a_{m,n} &{} b_{m',1} &{} b_{m',2} &{} \ldots &{} b_{m',n'} \end{matrix} \end{aligned}$$

The row concatenation of W and $W'$, denoted by , is undefined if $n \ne n'$ and is otherwise obtained by writing W above of $W'$. The row and column catenation operations can be also viewed as operations on languages. Also, we can define n-fold iterations (powers) of column catenation as $W^n$ and n-fold iterations (powers) of row catenation as $W_n$. Accordingly, $\varSigma ^n_m$ is understood, as well as $\varSigma ^+_m=\bigcup _{n\ge 1}\varSigma ^n_m$ and similarly $\varSigma _+^n$. In this sense, $\varSigma ^{++}=\varSigma _+^+$.

2.2 Unary Operations and Connections to Group Theory

As pictures are (also) geometrical objects, several further unary operations can be introduced [7]: quarter-turn (rotate clockwise by $90^{\circ }$) Q, half-turn (rotate by $180^{\circ }$) H, anti-quarter-turn (rotate anti-clockwise by $90^{\circ }$ (or rotate clockwise by $270^{\circ }$)) $Q^{-1}$, transpose T (reflection along the main diagonal), anti-transpose $T'$ (reflection along the anti-diagonal), $R_b$ (reflection along a horizontal (base) line), $R_v$(reflection along a vertical line). Together with the identity I, these (now eight) operators form a non-commutative group (with respect to composition), the well-known dihedral group $D_4$ [1]; see Table 1a.

In Table 1a, $\circ$ is the function composition. So, if $f: X \rightarrow Y$ and $g: Y \rightarrow Z$ are two functions, then $g \circ f: X \rightarrow Z$ is defined by $(g \circ f)(x) = g(f(x))$ for all $x \in X$. How Table 1a works is shown below for $W := [a_{i, j}]_{m, n}$ in the following.

$$ (T\circ R_v)(W)= T(R_v(W)) = T \begin{pmatrix} &{} a_{1,n} &{} \ldots &{} a_{1,2} &{} a_{1,1} \\ &{} a_{2,n} &{} \ldots &{} a_{2,2} &{} a_{2,1} \\ &{} \vdots &{} \vdots &{} \ddots &{} \vdots \\ &{} a_{m,n} &{} \ldots &{} a_{m,2} &{} a_{m,1} \end{pmatrix} = \begin{matrix} &{} a_{1,n} &{} a_{2,n} &{} \ldots &{} a_{m,n} \\ &{} \vdots &{} \ddots &{} \vdots &{} \vdots \\ &{} a_{1,2} &{} a_{2,2} &{} \ldots &{} a_{m,2} \\ &{} a_{1,1} &{} a_{2,1} &{} \ldots &{} a_{m,1} \end{matrix} = Q^{-1}(W). $$

Table 1. Unary operators

Let $\mathcal {O} = \{I, Q^{-1}, H, Q, R_v, R_b, T, T'\}$ be the set of these 8 unary operators comprising $D_4$. The operators in $D_4$ are usually partitioned into the four rotations (including the identity) $\{I, Q^{-1}, H, Q\}$, which form the subgroup $D_2$ of $D_4$, and four reflections $\{R_v, R_b, T, T'\}$. These operations can be also applied (picture-wise) to picture languages and (language-wise) to families of picture languages. It is interesting to add the fact that one single rotation Q generates all rotations (as a subgroup of $D_4$) and that all of $D_4$ are generated by one rotation Q and one reflection T. In Table 1b we make explicit how any operator can be written using the composition of the two mentioned operators. Table 1b can be deduced from Table 1a, for instance $Q^{-1}=Q \circ (Q \circ Q)$ since $Q^{-1}=Q \circ H$ and $H=Q\circ Q$. This simple observation helps simplify several of our arguments.

For instance, we can combine sequences of catenation and unary operations from $D_4$ to obtain Table 2, starting out from the four simple observations

(1) (however, mind the sequence of arguments), (2) , (3) and

(4) . For example,

Table 2. Table of operators.

3 General Boustrophedon Finite Automata

We now give one of the main definitions of this paper, introducing a new, parameterized automaton model for picture processing.

A general boustrophedon finite automaton, or GBFA for short, can be specified as an 8-tuple , where Q is a finite set of states, where Q is partitioned into $Q_f$ and $Q_b$, $\varSigma$ is an input alphabet, $R\subseteq Q\times (\varSigma \cup \{\#\})\times Q$ is a finite set of rules. A rule $(q,a,p)\in R$ is usually written as $qa\rightarrow p$. We impose some additional restrictions. If $q\in Q_f$ and $a\in \varSigma$, then $qa\rightarrow p\in R$ is only possible if $p\in Q_f$. Such transitions are also called forward transitions and collected within $R_f$. Similarly, if $q\in Q_b$ and $a\in \varSigma$, $qa\rightarrow p\in R$ is only possible if $p\in Q_b$ (backward transitions, collected in $R_b$). Finally, border transitions (collected in $R_\#$) are of the form $q\#\rightarrow p$ with $q\in Q_f$ iff $p\in Q_b$. Namely, the special symbol $\# \notin \varSigma$ indicates the border of the rectangular picture that is processed, $s\in Q_f$ is the initial state, F is the set of final states, and $D\in \mathcal {D}$ indicates the move directions. Here,

$$\mathcal {D}=\left\{ \left( \begin{matrix}s\rightarrow &{} \downarrow \\ \downarrow &{} \leftarrow \end{matrix}\right) , \left( \begin{matrix}s\downarrow &{} \rightarrow \\ \rightarrow &{} \uparrow \end{matrix}\right) , \left( \begin{matrix}\downarrow &{} \leftarrow s\\ \rightarrow &{} \downarrow \end{matrix}\right) , \left( \begin{matrix}\leftarrow &{} \downarrow s\\ \uparrow &{} \leftarrow \end{matrix}\right) , \left( \begin{matrix}\rightarrow &{} \downarrow \\ s \uparrow &{} \rightarrow \end{matrix}\right) , \left( \begin{matrix}\uparrow &{} \leftarrow \\ s \rightarrow &{} \uparrow \end{matrix}\right) , \left( \begin{matrix}\downarrow &{} \leftarrow \\ \leftarrow &{} \uparrow s \end{matrix}\right) , \left( \begin{matrix}\rightarrow &{} \uparrow \\ \uparrow &{} \leftarrow s \end{matrix}\right) \right\} $$

We are now going to discuss the notions of configurations, valid configurations and an according configuration transition to formalize the work of GBFAs, based on snapshots of their work.

Let be a new symbol indicating an erased position and let . Then is the set of configurations of M. Hence, the first and last columns and the first and last rows are completely filled with $\#$, and these are the only positions that contain $\#$.

The initial configuration is determined by the input array $A\in \varSigma ^{++}$. More precisely, if A has m rows and n columns, then

shows the according initial configuration $c_{init}(A)$. Similarly, a final configuration is then given by

for some $q_f\in F$ and $d\in \{b,f\}$.

The processing of the automaton is then crucially depending on $D\in \mathcal {D}$. The arrow that appears together with s indicates the direction of the forward processing of the first, third, fifth etc. line. For instance, the first listed direction contains $s\rightarrow$, determining that the odd-numbered rows of the input array are scanned left to right. Similarly, the second listed direction contains $s\downarrow$, determining that the odd-numbered columns of the input array are scanned top to bottom. When the automaton encounters a border symbol, it processes the next line in the reversed way (backward processing). This is also indicated in the little pictures that describe $D\in \mathcal {D}$. For instance, in the first case, the $\downarrow$ in the first row indicates that after hitting the border, the automaton moves downwards, processing (as indicated by the $\leftarrow$ in the last row) now from right to left, until the border is hit again, which means to move downwards one more row, as indicated by the $\downarrow$ in the last row. The other $D\in \mathcal {D}$ can be interpreted in a similar fashion, as explained below. Let us now formalize this description. Notice that an odd-numbered row of the input array corresponds to an even-numbered row if we consider the input array bordered by a $\#$-layer.

• If (p, A, f) and $(q,A',f)$ are two configurations such that A and $A'$ are identical but for one position (i, j), $1\le i\le m+2$, $1\le j\le n+2$, where while $A[i,j]\in \varSigma$, then $(p,A,f)\vdash _M (q,A',f)$ if $p A[i,j]\rightarrow q\in R_f$. Moreover, i is even.

• Conversely, if (p, A, f) and $(q,A',f)$ are two configurations such that A and $A'$ are identical but for one position (i, j), $1\le i\le m+2$, $1\le j\le n+2$, where while $A[i,j]\in \varSigma$, then $(p,A,b)\vdash _M (q,A',b)$ if $p A[i,j]\rightarrow q\in R_b$. Moreover, i is odd.

• If (p, A, f) and (q, A, b) are two configurations, then $(p,A,f)\vdash _M (q,A,b)$ or $(p,A,b)\vdash _M (q,A,f)$ if $p\#\rightarrow q\in R_\#$.

The reflexive transitive closure of the relation $\vdash _M$ is denoted by $\vdash _M^*$. $A\in \varSigma ^{++}$ is accepted by a GBFA M with direction $D_{BFA}:=\left( {\begin{matrix}s\rightarrow &{} \downarrow \\ \downarrow &{} \leftarrow \end{matrix}}\right)$ if $c_{init}(A)\vdash _M^* c$ such that c is a final configuration.

The following illustrates how such a GBFA scans some input picture and how a picture of a valid configuration looks like; it can be seen that the sequence of only indicates how far the input has been processed:

It should be also clear that the representation on the right-hand side of the previous picture contains all information necessary to describe a configuration apart from the state. Now, we define the other modes by applying transformations according to the following table.

A is accepted by a GBFA M with a different direction D if $f_D(A)$ is accepted by the GBFA $M_{BFA}$ that coincides with M in every detail except for the direction, which is now $D_{BFA}$. This means, for instance, in the case of ${D}= \left( {\begin{matrix}s\downarrow &{} \rightarrow \\ \rightarrow &{} \uparrow \end{matrix}}\right)$, that instead of scanning the input array A column by column, the first column top-down, the second bottom-up, and so forth, we rather transpose A and then scan the transposed array row by row, the first row left-right, the second right-left, and so forth.

The GBFA M is deterministic, or a GBDFA for short, if for each $p\in Q$ and $a\in \varSigma \cup \{\#\}$, there is at most one $q\in Q$ with $pa\rightarrow q\in R$.

This way, we define language classes like $\mathcal {L}_D(\mathrm {GBFA})$ of those array languages accepted by GBFAs working with direction D, as well as

$$\mathcal {L}(\mathrm {GBFA}):=\bigcup _{D\in \mathcal {D}}\mathcal {L}_D(\mathrm {GBFA}).$$

Of course, the interesting question is if the eight language families that we can obtain in these ways are really different from each other or not. Also, the situation of $\mathcal {L}(\mathrm {GBFA})$ needs to be investigated, as well as the role of determinism. We will see that the group-theoretic excursion from above will simplify our reasoning a lot. From the definitions themselves, we can immediately derive the following characterization result.

Theorem 1

The class $\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})$ coincides with the following classes of picture languages: $T\left( \mathcal {L}_{\left( {\begin{matrix}s\downarrow &{} \rightarrow \\ \rightarrow &{} \uparrow \end{matrix}}\right) }(\mathrm {GBFA})\right)$, $R_v\left( \mathcal {L}_{\left( {\begin{matrix}\downarrow &{} \leftarrow s\\ \rightarrow &{} \downarrow \end{matrix}}\right) }(\mathrm {GBFA})\right)$,

$Q^{-1}\left( \mathcal {L}_{\left( {\begin{matrix}\leftarrow &{} \downarrow s\\ \uparrow &{} \leftarrow \end{matrix}}\right) }(\mathrm {GBFA})\right)$, $Q\left( \mathcal {L}_{\left( {\begin{matrix}\rightarrow &{} \downarrow \\ s \uparrow &{} \rightarrow \end{matrix}}\right) }(\mathrm {GBFA})\right)$, $R_b\left( \mathcal {L}_{\left( {\begin{matrix}\uparrow &{} \leftarrow \\ s \rightarrow &{} \uparrow \end{matrix}}\right) }(\mathrm {GBFA})\right)$,

$T'\left( \mathcal {L}_{\left( {\begin{matrix}\downarrow &{} \leftarrow \\ \leftarrow &{} \uparrow s \end{matrix}}\right) }(\mathrm {GBFA})\right)$, and $H\left( \mathcal {L}_{\left( {\begin{matrix}\rightarrow &{} \uparrow \\ \uparrow &{} \leftarrow s \end{matrix}}\right) }(\mathrm {GBFA})\right)$.

Due to the connections to group theory described above, we can easily infer from the previous theorem characterizations of the seven other classes, referring back to $\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})$. We collect these results in the following theorem.

Theorem 2

We obtain the following list of characterizations.

• $\mathcal {L}_{\left( {\begin{matrix}s\downarrow &{} \rightarrow \\ \rightarrow &{} \uparrow \end{matrix}}\right) }(\mathrm {GBFA})=T\left( \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\downarrow &{} \leftarrow s\\ \rightarrow &{} \downarrow \end{matrix}}\right) }(\mathrm {GBFA})=(Q\circ T)\left( \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\rightarrow &{} \downarrow \\ s \uparrow &{} \rightarrow \end{matrix}}\right) }(\mathrm {GBFA})=(Q\circ (Q\circ Q))\left( \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\leftarrow &{} \downarrow s\\ \uparrow &{} \leftarrow \end{matrix}}\right) }(\mathrm {GBFA})=Q\left( \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\uparrow &{} \leftarrow \\ s \rightarrow &{} \uparrow \end{matrix}}\right) }(\mathrm {GBFA})=(T\circ Q)\left( \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\downarrow &{} \leftarrow \\ \leftarrow &{} \uparrow s \end{matrix}}\right) }(\mathrm {GBFA})=(Q\circ (Q\circ T))\left( \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\rightarrow &{} \uparrow \\ \uparrow &{} \leftarrow s \end{matrix}}\right) }(\mathrm {GBFA})=(Q\circ Q)\left( \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})\right)$.

These characterizations are also valid for the corresponding deterministic classes.

In [2], boustrophedon finite automata have been introduced that basically work as GBFAs do when working in mode $D_{BFA}$, apart from the fact that we have integrated in our definition of GBFA that the automaton ‘knows’ if it processes an even- or odd-numbered row. In other words, they are aware of the direction of their movement (left to right or right to left). However, as the basic model is finite automata, it is not difficult to show BFAs can be turned into one that is direction-aware, by counting the number of reads of $\#$-symbols modulo two. Hence, we can easily profit from results in [2]. We give one example now.

Theorem 3

For each direction mode D, we know: $\mathcal {L}_{D}(\mathrm {GBFA})= \mathcal {L}_{D}(\mathrm {GBDFA})$.

Proof

By the previous reasoning, we can derive from [2] that $\mathcal {L}_{D}(\mathrm {GBFA})= \mathcal {L}_{D}(\mathrm {GBDFA})$ is true for $D=D_{BFA}$. By Theorem 2, we have characterizations of $\mathcal {L}_{D}(\mathrm {GBFA})$ in terms of $\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})$. These characterizations are also valid for the corresponding deterministic classes.    $\square$

4 Generalized Returning Finite Automata

A useful auxiliary model introduced in [2] was so-called returning finite automata (RFA). We are going to generalize their work in the following, again by introducing working modes for them. Now, a pair of directions like $D=\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right)$ is sufficient, indicating that an input array is always processed row by row, top down, where each row is scanned from left to right; moreover, the procedure is (here) started at the upper left corner of the array, as indicated by the position of s. The processing mode just describes coincide with that of RFAs from [2]. Leaving out the formal definition for now, we arrive at language families like $\mathcal {L}_{D}(\mathrm {GRFA})$. It is sufficient to understand that there is no need to give any direction information. There are (again) eight natural processing modes D:

$$\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right) ,\ \left( {\begin{matrix} s\downarrow&\rightarrow \end{matrix}}\right) ,\ \left( {\begin{matrix} \downarrow&\leftarrow s\end{matrix}}\right) ,\ \left( {\begin{matrix} \leftarrow&\downarrow s\end{matrix}}\right) ,\ \left( {\begin{matrix} s\rightarrow&\uparrow \end{matrix}}\right) ,\ \left( {\begin{matrix} s\uparrow&\rightarrow \end{matrix}}\right) ,\ \left( {\begin{matrix} \uparrow&\leftarrow s\end{matrix}}\right) ,\ \left( {\begin{matrix} \leftarrow&\uparrow s\end{matrix}}\right) .$$

These can be naturally partioned into the row-first modes $\mathcal {D}_{row-f}\{\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right) , \left( {\begin{matrix} \downarrow&\leftarrow s\end{matrix}}\right) ,\ \left( {\begin{matrix} s\rightarrow&\uparrow \end{matrix}}\right) ,\ \left( {\begin{matrix} \uparrow&\leftarrow s\end{matrix}}\right) \}$ and the four other column-first modes in $\mathcal {D}_{col-f}$. Again, we have deterministic variants, and we can consider the union of all languages pertaining to these GRFA-variants.

Example 1

The set of all arrays over $\{\mathtt {a},\mathtt {b}\}$ such that each array in the set has exactly one row completely filled with $\mathtt {b}$’s and $\mathtt {a}$’s everywhere else is accepted by a GRFA M as shown in Fig. 1.

Fig. 1.

GRFA M that accepts the language in Example 1.

As with GBFAs, we can alternatively describe the work of a GRFA working in mode D by first performing a unary operation on the image and then processing the image in the mode $D_{RFA}=\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right)$ that corresponds to RFAs as introduced in [2]. Therefore, we obtain the following characterizations for these modes.

Theorem 4

The class $\mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$ coincides with the following classes: $T\left( \mathcal {L}_{\left( {\begin{matrix}s\downarrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA})\right)$, $R_v\left( \mathcal {L}_{\left( {\begin{matrix}\downarrow&\leftarrow s \end{matrix}}\right) }(\mathrm {GRFA})\right)$, $Q^{-1}\left( \mathcal {L}_{\left( {\begin{matrix}\leftarrow&\downarrow s \end{matrix}}\right) }(\mathrm {GRFA})\right)$, $Q\left( \mathcal {L}_{\left( {\begin{matrix}s \uparrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA})\right)$, $R_b\left( \mathcal {L}_{\left( {\begin{matrix}s \rightarrow&\uparrow \end{matrix}}\right) }(\mathrm {GRFA})\right)$, $T'\left( \mathcal {L}_{\left( {\begin{matrix}\leftarrow&\uparrow s \end{matrix}}\right) }(\mathrm {GRFA})\right)$, and $H\left( \mathcal {L}_{\left( {\begin{matrix}\uparrow&\leftarrow s \end{matrix}}\right) }(\mathrm {GRFA})\right)$.

Likewise, we can use the previous theorem to obtain a representation like Theorem 2 also for GRFAs.

Theorem 5

We obtain the following list of characterizations.

• $\mathcal {L}_{\left( {\begin{matrix}s\downarrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA})=T\left( \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\downarrow&\leftarrow s \end{matrix}}\right) }(\mathrm {GRFA})=(Q\circ T)\left( \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}s \uparrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA})=(Q\circ (Q\circ Q))\left( \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\leftarrow&\downarrow s \end{matrix}}\right) }(\mathrm {GRFA})=Q\left( \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}s \rightarrow&\uparrow \end{matrix}}\right) }(\mathrm {GRFA})=(T\circ Q)\left( \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\leftarrow&\uparrow s \end{matrix}}\right) }(\mathrm {GRFA})=(Q\circ (Q\circ T))\left( \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})\right)$.

• $\mathcal {L}_{\left( {\begin{matrix}\uparrow&\leftarrow s \end{matrix}}\right) }(\mathrm {GRFA})=(Q\circ Q)\left( \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})\right)$.

We can conclude from [2, Theorem 2]:

Corollary 1

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) = \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$.

With Theorems 2 and 5, we also get a complete list of correspondences between GBFAs and GRFAs as in Table 3. For instance, $Q(\mathcal {L}_{D_{RFA}}(\mathrm {GRFA}))=\mathcal {L}_{D}(\mathrm {GRFA})$ for $D=({\begin{matrix}\leftarrow&\downarrow s\end{matrix}})$ can be read off as the table entry. The previous result means that determinism does not restrict the power of GRFA in all processing modes.

5 Language Families Under the Unary Operators

We are now going to collect and prove several results relating the different language families that we introduced so far by means of the unary operators that we discussed above. The proofs will also show that it is quite valuable to have the different processing modes available.

Lemma 1

$\mathcal {L}_D(\mathrm {GRFA}) = R_v(\mathcal {L}_D(\mathrm {GRFA}))$ for $D\in \mathcal {D}_{row-f}$.

Table 3. Operators/directions for $\mathrm {GBFA}$s and $\mathrm {GRFA}$s

Proof

Let be some GRFA and let $L = L(M)$. Let us construct some GRFA $M^v$ that accepts $R_v(L)$. is defined by $Q^v = Q \times Q \times Q$, $Q_{\#} = \{q \mid p a \rightarrow q \wedge q \# \rightarrow r \in R \}$, $Q_I = \{(s,q,r)\}$ where $q \in Q$, $r \in Q_{\#}$,

$$\begin{aligned} R^v= & {} \{(\ell ,q,r) a \rightarrow (\ell ,p,r) \mid pa\rightarrow q\in R, \ell \in Q, r \in Q_{\#}, a \in \varSigma \}\\&\cup \{(\ell ,\ell ,p) \# \rightarrow (q,t,r) \mid p \# \rightarrow q \in R, \ell ,r,t \in Q, a \in \varSigma \}\\&\cup \{(\ell ,q,f) a \rightarrow (\ell ,p,f) \mid pa\rightarrow q\in R, \ell \in Q, f \in F, a \in \varSigma \}\, \end{aligned}$$

and $Q_F \subseteq Q^v$, $Q_F=\{(\ell ,\ell ,f)\mid f\in F, \ell \in Q\}$. The idea of the construction is that the mirror-image construction well-known from classical formal language theory. Here, the first component $\ell$ of some triple $(\ell ,q,r)\in Q\times Q\times Q$ memorizes the state associated to the left-most symbol of that row, q is the current state and r is associated to the right-most symbol in the row. Reading $\#$ switches to the next row until the final state is reached. We only considered the first processing mode here, the other three modes can be shown using similar constructions.    $\square$

As the vertical reflection can be likewise seen as a change in the processing mode, we can immediately conclude:

Corollary 2

$\mathcal {L}_{\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right) }(\mathrm {GRFA}) = \mathcal {L}_{\left( {\begin{matrix} \downarrow&\leftarrow s\end{matrix}}\right) }(\mathrm {GRFA})$;

$\mathcal {L}_{\left( {\begin{matrix} s\rightarrow&\uparrow \end{matrix}}\right) }(\mathrm {GRFA}) = \mathcal {L}_{\left( {\begin{matrix} \uparrow&\leftarrow s\end{matrix}}\right) }(\mathrm {GRFA})$.

Due to Theorem 5, we can also conclude:

Lemma 2

$\mathcal {L}_D(\mathrm {GRFA}) = R_b(\mathcal {L}_D(\mathrm {GRFA}))$ for $D\in \mathcal {D}_{row-f}$.

Lemma 3

$\mathcal {L}_D(\mathrm {GRFA}) = R_b(\mathcal {L}_D(\mathrm {GRFA}))$ for $D\in \mathcal {D}_{col-f}$.

Hence, we can immediately conclude the following characterizations.

Corollary 3

$\mathcal {L}_{\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right) }(\mathrm {GRFA}) = \mathcal {L}_{\left( {\begin{matrix} s\rightarrow&\uparrow \end{matrix}}\right) }(\mathrm {GRFA})$;

$\mathcal {L}_{\left( {\begin{matrix} \downarrow&\leftarrow s\end{matrix}}\right) }(\mathrm {GRFA}) = \mathcal {L}_{\left( {\begin{matrix} \uparrow&\leftarrow s\end{matrix}}\right) }(\mathrm {GRFA})$.

Corollary 4

$\mathcal {L}_{\left( {\begin{matrix}s\downarrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA}) = \mathcal {L}_{\left( {\begin{matrix}s \uparrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA})$;

$\mathcal {L}_{\left( {\begin{matrix}\leftarrow&\downarrow s \end{matrix}}\right) }(\mathrm {GRFA}) = \mathcal {L}_{\left( {\begin{matrix}\leftarrow&\uparrow s \end{matrix}}\right) }(\mathrm {GRFA})$.

Lemma 1 and Corollary 1 give the following corollary; also refer to Theorem 1; similar closure properties for the other processing modes can be easily derived by combining what we have shown so far.

Corollary 5

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) = R_v(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))$.

By Corollary 1 and Theorem 4, we obtain:

Theorem 6

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) = Q(\mathcal {L}_{\left( {\begin{matrix} s\uparrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA}))$.

Theorem 6 immediately yields the following result, as $Q\circ Q^{-1}$ is the identity.

Corollary 6

$Q^{-1}(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))=\mathcal {L}_{\left( {\begin{matrix} s\uparrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA})$.

Using Theorems 2, 5, Corollary 1, and Table 3, we can conclude the following:

Corollary 7

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) = Q^{-1}(\mathcal {L}_{\left( {\begin{matrix}s \uparrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA}))$.

As $Q^{-1} \circ Q^{-1} = H$, Corollaries 6 and 7 yield:

Corollary 8

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) = H(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))$.

As $R_b = H \circ R_v$, Corollaries 5 and 8 give:

Corollary 9

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) = R_b(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))$.

We can now summarize our characterization of picture language classes:

Theorem 7

The picture language family $\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})$ equals

• $\mathcal {L}_{D}(\mathrm {GBFA})$ for $D\in \left\{ D_{BFA},\left( {\begin{matrix}\downarrow &{} \leftarrow s\\ \rightarrow &{} \downarrow \end{matrix}}\right) ,\left( {\begin{matrix}\uparrow &{} \leftarrow \\ s \rightarrow &{} \uparrow \end{matrix}}\right) ,\left( {\begin{matrix}\rightarrow &{} \uparrow \\ \uparrow &{} \leftarrow s \end{matrix}}\right) \right\}$;

• $\mathcal {L}_{D}(\mathrm {GRFA})$ for $D\in \mathcal {D}_{row-f}$.

The picture language family $T(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))$ equals

• $\mathcal {L}_{D}(\mathrm {GBFA})$ for $D\in \left\{ \left( {\begin{matrix}s\downarrow &{} \rightarrow \\ \rightarrow &{} \uparrow \end{matrix}}\right) , \left( {\begin{matrix}\leftarrow &{} \downarrow s\\ \uparrow &{} \leftarrow \end{matrix}}\right) , \left( {\begin{matrix}\rightarrow &{} \downarrow \\ s \uparrow &{} \rightarrow \end{matrix}}\right) , \left( {\begin{matrix}\downarrow &{} \leftarrow \\ \leftarrow &{} \uparrow s \end{matrix}}\right) \right\}$;

• $\mathcal {L}_{D}(\mathrm {GRFA})$ for $D\in \mathcal {D}_{col-f}$.

Proof

(a) $\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) =_{\text {Cor.~5}} R_v(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})) =_{\text {Thm.~2}} \mathcal {L}_{\left( {\begin{matrix}\downarrow &{} \leftarrow s\\ \rightarrow &{} \downarrow \end{matrix}}\right) }(\mathrm {GBFA})$.

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) =_{\text {Cor.~9}} R_b(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})) =_{\text {Thm.~2}} \mathcal {L}_{\left( {\begin{matrix}\uparrow &{} \leftarrow \\ s \rightarrow &{} \uparrow \end{matrix}}\right) }(\mathrm {GBFA})$.

$\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) =_{\text {Cor.~8}} H(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})) =_{\text {Thm.~2}} \mathcal {L}_{\left( {\begin{matrix}\rightarrow &{} \uparrow \\ \uparrow &{} \leftarrow s \end{matrix}}\right) }(\mathrm {GBFA})$.

(b) $\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}) =_{\text {Cor.1}} \mathcal {L}_{\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right) }(\mathrm {GRFA}) =_{\text {Cor.2}} \mathcal {L}_{\left( {\begin{matrix} \downarrow&\leftarrow s\end{matrix}}\right) }(\mathrm {GRFA})$

$=_{\text {Cor.3}} \mathcal {L}_{\left( {\begin{matrix} \uparrow&\leftarrow s\end{matrix}}\right) }(\mathrm {GRFA}) =_{\text {Cor.2}} \mathcal {L}_{\left( {\begin{matrix} s\rightarrow&\uparrow \end{matrix}}\right) }(\mathrm {GRFA})$.

(c) $T(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})) =_{\text {Thm.2}} \mathcal {L}_{\left( {\begin{matrix}s\downarrow &{} \rightarrow \\ \rightarrow &{} \uparrow \end{matrix}}\right) }(\mathrm {GBFA})$.

$T(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))=_{\text {Table\,1a}}$ $Q(R_b(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})))$

$=_{\text {Cor.9}}$ $Q(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})) =_{\text {Thm.2}}$ $\mathcal {L}_{\left( {\begin{matrix}\leftarrow &{} \downarrow s\\ \uparrow &{} \leftarrow \end{matrix}}\right) }(\mathrm {GBFA})$.

$T(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))=_{\text {Table\,1a}}$ $Q^{-1}(R_v(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})))$

$=_{\text {Cor.5}}$ $Q^{-1}(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})) =_{\text {Thm.2}}$ $\mathcal {L}_{\left( {\begin{matrix}\rightarrow &{} \downarrow \\ s \uparrow &{} \rightarrow \end{matrix}}\right) }(\mathrm {GBFA})$.

$T(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))=_{\text {Table\,1a}}$ $T'(H(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})))$

$=_{\text {Cor.8}}$ $T'(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA})) =_{\text {Thm.2}}$ $\mathcal {L}_{\left( {\begin{matrix}\downarrow &{} \leftarrow \\ \leftarrow &{} \uparrow s \end{matrix}}\right) }(\mathrm {GBFA})$.

(d) Similarly, using Corollary 1 and Theorem 5.    $\square$

Corollary 10

$\mathcal {L}(\mathrm {GBFA})=\mathcal {L}_{\left( {\begin{matrix} s\downarrow&\rightarrow \end{matrix}}\right) }(\mathrm {GRFA}) \cup \mathcal {L}_{\left( {\begin{matrix} s\rightarrow&\downarrow \end{matrix}}\right) }(\mathrm {GRFA})$.

Remark 1

Unfortunately, as can be easily seen by applying the exchange property arguments derived in [2], the set of square-sized arrays of odd length whose middle row and middle column contain $\mathtt {b}$’s, while all other positions are filled with $\mathtt {a}$’s, is not in $\mathcal {L}(\mathrm {GBFA})$.

Let $\mathcal {O}$ collect the eight unary operators introduced in Sect. 2.2.

Corollary 11

$\forall O\in \mathcal {O}: O(\mathcal {L}(\mathrm {GBFA}))=\mathcal {L}(\mathrm {GBFA})$.

To underline the special importance of the two operations Q and T from $\mathcal {O}$, we can also state:

Corollary 12

$\mathcal {L}(\mathrm {GBFA})=Q(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))\cup \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})$ and

$\mathcal {L}(\mathrm {GBFA})=T(\mathcal {L}_{D_{BFA}}(\mathrm {GBFA}))\cup \mathcal {L}_{D_{BFA}}(\mathrm {GBFA})$.

Let us now turn to the two binary catenation operations.

Theorem 8

.

Proof

Let and $M_2 = (Q_2,\varSigma ,R_2,s_2,F_2,$ be two GRFAs, with $L_1 = L(M_1)$ and $L_2 = L(M_2)$. W.l.o.g., assume that $Q_1\cap Q_2=\emptyset$. Let us construct the GRFA that accepts (i.e., ). is defined by , . The idea of the construction is to first simulate $M_1$; whenever a final state is entered at the end of reading a line, then the simulation may switch to $M_2$.    $\square$

Corollary 13

.

Define a mapping $str: \varSigma ^{+}_2 \rightarrow (\varSigma \cup \{\#\})^+$, $W \mapsto w$ where $w = w_1 \# w_2$, $|W|_c = n$, $n \ge 2$ and $|W|_r = 2$; moreover, $|w_1| = |w_2| = n$, so that $|w| = 2n+1$.

Lemma 4

If $L\in \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$ with $L\subseteq \varSigma ^{+}_2$, then str(L) is context-free.

Proof

A pushdown automaton uses its finite control as the GRFA and simply checks if the two rows have equal length by using its pushdown store.    $\square$

Theorem 9

.

Proof

Let $L = \left\{ \begin{matrix} 1 &{} 0^{\ell } \\ 1 &{} 0^{\ell } \end{matrix} :\ell \ge 1 \right\}$. Clearly, $L\in \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$. However, is a variant of the crossing dependency language known to be not context-free. By Lemma 4, .    $\square$

Corollary 14

.

Theorem 10

$\mathcal {L}(\mathrm {GRFA})$ is closed neither under column catenation nor under row catenation.

Proof

Consider L from the proof of Theorem 9. By Theorem 8, . If , then also , contradicting the reasoning from Theorem 9. Exchange property arguments derived in [2] easily show that , intuitively because horizontal lines in arbitrary positions cannot be checked by finite automata working column by column.    $\square$

So far, we have used finite automata to only accept picture (languages). If such devices should be used in practice, at least two questions show up:

• Can we design such automata in a systematic, best modular fashion?

• Can we set up these automata so that they can tolerate certain input errors?

We will work towards this in the next section.

6 Picture Transforming Automata

To answer both questions, we introduce a generalization of Mealy machines to picture processing. A Mealy Picture Machine, or MPM for short, can be specified as a 9-tuple $M=(Q,\varSigma ,\Gamma ,\delta ,\lambda ,q_0,\#,F,D)$, where Q is a finite set of states, $\varSigma$ is a finite set of input symbols, $\Gamma$ is a finite set of output symbols, $\delta : Q\times (\varSigma \cup \{\#\})\rightarrow Q$ is the transition function, $\lambda : Q\times \varSigma \rightarrow \Gamma$ is the output function, $q_0\in Q$ is the initial state, $\# \notin \varSigma$, $\# \notin \Gamma$, is the special symbol that indicates the borders of the picture that is processed, F is the set of final states, and $D\in \mathcal {D}$ indicates the move directions of the GRFAs. Let us now give the notion of configurations to formalize the working of MPMs, based on the snapshots of their work. Here, we assume (w.l.o.g.) that $\varSigma \cap \Gamma = \emptyset$. Let $\Upsilon := \varSigma \cup \Gamma$ and $\Upsilon _{\#}:=\Upsilon \cup \{\#\}$. Then is the set of configurations of M. Hence, the first and last columns and the first and last rows are completely filled with $\#$, and these are the only positions that contain $\#$. The initial configuration is determined by the input array $A\in \varSigma ^{++}$. More precisely, if A has m rows and n columns, then

shows the according initial configuration $C_{init}(A)$. Similarly, a final configuration $C_{fin}(A')$ is then given by

for some $A' \in \Gamma ^{++}$ with m rows and n columns and for some $q_f\in F$. The processing of the machine depends on $D\in \mathcal {D}$ of the GRFAs. Let us now formalize the description with direction $D=\left( \begin{matrix}s\rightarrow&\downarrow \end{matrix}\right)$; the remaining seven directions can be formalized similarly but we did not explicitly write those here.¹

• If $(p,A,\mu )$ and $(q,B,\mu )$ are two configurations such that A and B are identical but for one position (i, j), $1\le i\le m+2$, $1\le j\le n+2$, where $B[i,j] \in \Gamma$, while $A[i,j]\in \varSigma$, then $(p,A,\mu )\vdash _M (q,B,\mu )$ if $\delta (p,A[i,j])=q$ and $\lambda (p,A[i,j])=B[i,j]$.

• If $(p,A,\mu )$ and $(q,A,\mu +1)$ are two configurations, then $(p,A,\mu )\vdash _M (q,A,\mu +1)$ if $\delta (p,\#)=q$.

The reflexive transitive closure of the relation $\vdash _M$ is denoted by $\vdash _M^*$. Notice that for each $A\in \varSigma ^{++}$ there is at most one $A'\in \Gamma ^{++}$ such that $C_{init}(A)\vdash _M^* C_{fin}(A')$. We can hence view M as a partial function $M:\varSigma ^{++}\nrightarrow \Gamma ^{++}$.

Theorem 11

Given $L \in \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$, with $L \subseteq \varSigma ^{++}$, and an MPM M: $\varSigma ^{++} \nrightarrow \Gamma ^{++}$, then $M(L) \in \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$.

The proof is very similar to the construction in the following theorem.

Theorem 12

Given $L \in \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$, with $L \subseteq \Gamma ^{++}$, and an MPM M: $\varSigma ^{++} \nrightarrow \Gamma ^{++}$, then $M^-(L) \in \mathcal {L}_{D_{RFA}}(\mathrm {GRFA})$ with $M^-(L)\subseteq \varSigma ^{++}$.

Proof

Let be some GRFA with $L(R) \subseteq \Gamma ^{++}$. Let $M:\varSigma ^{++}\nrightarrow \Gamma ^{++}$ be an MPM, $M=(Q_M,\varSigma ,\Gamma ,\delta ,\lambda ,s_M,\#,F_M,(\begin{matrix} s\rightarrow&\downarrow \end{matrix}))$ lifted to $M:2^{\varSigma ^{++}} \rightarrow 2^{\Gamma ^{++}}$ and $M^-:2^{\varSigma ^{++}} \rightarrow 2^{\Gamma ^{++}}$, i.e., $M^-(\Gamma ^{++}) \subseteq \varSigma ^{++}$. Now our aim is to define a GRFA $R'$ such that $L(R')= M^-(L(R))$. is defined by $Q' = Q \times Q_M$, $P' = \{(p,q)a \rightarrow (p',q') \mid \delta (q,a) = q' \wedge p \lambda (q,a) \rightarrow p' \in P\}\cup \{(p,q)\# \rightarrow (p',q') \mid \delta (q,\#) = q' \wedge p \# \rightarrow p' \in P\}$, $s' = (s,s_M) \in Q'$ is the start state and $F' = F \times F_M$.    $\square$

Let us now explain the extended power of GRFA by adding MPM with it. Also, we want to describe how to use MPMs and further closure properties to design more complicated GRFAs, starting off from very simple automata. Suppose the task is to design a GRFA $R'$ with $L(R') = L'$, see Table 4. How can we obtain such an automaton? We see that the pictures in $L'$ can be decomposed into two subsequent rows of $\mathtt {x}$ and one first column of $\mathtt {x}$. It looks easier to design automata for both tasks separately and then combine them later ‘somehow’. This is what MPMs can do. Assume that we have designed an MPM M that takes inputs from $\varSigma ^{++}$, with $\varSigma =\{\mathtt {x},\cdot \}$, and does the following:

• It converts all $\cdot$ symbols into $\mathtt {a}$ if they do not appear in the first column.

• It converts all $\mathtt {x}$ symbols into $\mathtt {a}$ if they appear in the first column, but into $\mathtt {b}$ if they appear in any other column.

Assume we have designed M. $M(L')$ can be found in Table 4. How can we further process this? Maybe, it would be an idea to design another MPM $M'$ that takes inputs from $\{\mathtt {a},\mathtt {b}\}^{++}$ and converts them according to the following:

• It converts $\mathtt {a}$ symbols into $\cdot$ but one special to be mentioned later.

• When it first reads a $\mathtt {b}$, this must be in the second column, and this $\mathtt {b}$ must be followed by $\mathtt {b}$’s only in the current row, which is then converted into a row of $\cdot$’s only. (If the first row where $M'$ encounters any $\mathtt {b}$’s is not of the form $\mathtt {a}\mathtt {b}^+$, $M'$ will not enter an accepting state when further processing the input array, as it has observed an input error.)

• In the next row, it is checked if the first symbol is a $\mathtt {a}$. This will then be converted to $\mathtt {x}$.

• Any further occurrences of $\mathtt {b}$ will be converted into $\mathtt {x}$.

Table 4. Simplifying array languages with MPMs

As a consequence, we find:

$$\begin{aligned} M'(M(L')) = \left\{ \begin{matrix} (\cdot &{} (\cdot )^n)_{\ell -1} \\ \cdot &{} (\cdot )^n_{} \\ \mathtt {x}&{} (\mathtt {x})^n_{} \\ (\cdot &{} (\cdot )^n)_{m-1} \end{matrix}\, :n,m,\ell \ge 1 \right\} = \left\{ \begin{matrix} (\cdot &{} (\cdot )^n)_{\ell } \\ \mathtt {x}&{} (\mathtt {x})^n_{} \\ (\cdot &{} (\cdot )^n)_{m-1} \end{matrix}\, :n,m,\ell \ge 1 \right\} . \end{aligned}$$

We can devise a quite similar machine R that then accepts all arrays over $\{\cdot ,\mathtt {x}\}$ that start with at least one row filled with $\cdot$, followed by one row of length at least two completely filled with $\mathtt {x}$ and then followed by an arbitrary number (possibly zero many) of rows filled up with $\cdot$. Now, the GRFA $R'$ we are looking for can be obtained by first constructing $R^\dagger$ for accepting $M^{\prime -}(L(R))$, based on R and $M'$, and then constructing $R'$ accepting $M^-(L(R^\dagger ))$, based on $R^\dagger$ and M. The drawback of this design approach could be a certain state explosion. For instance, already the smallest GRFA implementing L(R) has five states and the smallest MPM for $M'$ has six states, which gives us 30 states for $R^\dagger$, unless we interleave steps that delete useless states or even implement some state minimization procedures.

Notice that MPMs naturally generalize array homomorphisms as considered in [6]. Also, with the possibility to implement multiple passes (by applying the described form of transductions), it is easy to also implement set operations like union or intersection, as the first reading automaton could communicate with the automaton doing the second read by changing one particular part of the picture, for instance, by printing a special character at the very end, signaling acceptance. However, we also have negative results like the following one.

Theorem 13

$T(\mathcal {L}(\mathrm {GRFA}))$ is not closed under MPM nor under inverse MPM mappings.

Proof

By an exchange property argument [2], one can see that the language . However, it is easy to design an MPM M that only translates arrays from L into some arrays over the alphabet $\{0\}$. Also, $\{0\}^+_+\in T(\mathcal {L}(\mathrm {GRFA}))$. This shows non-closure under inverse MPM mappings, as $L=M^-(\{0\}^+_+)$. Conversely, let the MPM $M'$ work on arrays over the alphabet $\{0,1\}$ as follows: If a row starts with 0, then $M'$ will translate the whole row into a row of $\mathtt {a}$’s. If a row starts with 1, then $M'$ will translate the whole row into a row of $\mathtt {b}$’s. Now, it is easy to see that , while $M'(L')=L$ is not.    $\square$

It would therefore be interesting to study hierarchies of array languages defined by combining array processing devices that alternate between row-wise or column-wise processing, working on multiple passes over given images. This would also enable us to implement error-correction features, like thinning out blurred lines. This also shows that combining different processing modes in subsequent passes could be useful in practice.

7 Conclusions

We hope that the simple finite automata models that we presented in this paper can form a starting point to bring these syntactic ideas back into the practice of image processing. This is also why we studied seemingly simplistic working modes of such automata, including their use in image transformations. We also like to mention that currently a student of ours is implementing the machine transformation algorithms that we showed to prove closure properties of GBFA languages. We hope to report on this in the near future. This software will be available on request.

A possible further direction of research could be to integrate these models into pattern recognition algorithms. As exhibited by Flasiński in [3], this would necessitate the development of Grammatical Inference algorithms. In this context, it looks that we have to overcome the following technical problem: mostly, learners converge to canonical hypotheses like minimum-state automata and hence, efficient learners also comprise efficient state minimization algorithms. However, we can show the following result:

Theorem 14

It is NP-hard to decide, given some BDFA on some binary input alphabet $\varSigma =\{a,b\}$, whether or not there is an equivalent BDFA $M'$ with only one state.

The proof is not given due to space restrictions. This result seems to pose some difficulties in applying Grammatical Inference techniques to Syntactic Pattern Recognition in the context of picture scanning automata. It also indicates some limitations to the modular design approach of picture processing automata.