Finite Automata and Randomness

Abstract

The lecture surveys approaches using finite automata to define several notions of (automata-theoretic) randomness.

It focuses on the one hand on automata-theoretic randomness of infinite sequences in connection with automata-independent notions like disjunctivity and Borel normality.

On the other hand it considers the scale of relaxations of randomness (Borel normality and disjunctivity), that is, finite-state dimension and subword complexity and their interrelations.

1 Introduction

The (algorithmic) randomness of infinite sequences can be defined by means of computability. There have been three main approaches to the definition of algorithmically random sequences, namely

• 1. the measure-theoretic approach,

• 2. the unpredictability approach, and

• 3. the incompressibility (or complexity-theoretic) approach.

All these approaches are based on Turing machines and were shown to be equivalent in the case of Martin-Löf random sequences. We refer the reader to the textbooks [5, 9, 10, 13] for a complete history of Martin-Löf randomness and related topics.

After Martin-Löf’s measure-theoretic approach [11] and Schnorr’s unpredictability approach [16] already in the 1970s sequences random with respect to finite automata were considered. It turned out that two approaches equivalent in the algorithmic case yield different characterisations of sequences which might be called “random” in the automata case. The first approach is an adaptation of the betting or martingale approach of [16] to finite automata whereas the second – in an analogy to Martin-Löf’s measure theoretic approach – uses a randomness definition via null sets definable by finite automata.

Here we present a brief survey on both randomness approaches for finite automata and their relaxations which result in the finite-state dimension on the one hand and in a connection to subword complexity on the other hand.

2 Notation

We introduce the notation used throughout the paper. By we denote the set of natural numbers. Let $X=\{0,\dots ,r-1\}$ be a finite alphabet of cardinality $|X|= r\ge 2$, and $X^*$ be the set (monoid) of words on X, including the empty word e, and $X^{\omega }$ be the set of infinite sequences ($\omega$-words) over X. As usual we refer to subsets $W\subseteq X^{*}$ as languages and to subsets $F\subseteq X^{\omega }$ as $\omega$-languages.

For $w\in X^*$ and $\eta \in X^*\cup X^{\omega }$ let $w \cdot {}\eta$ be their concatenation. This concatenation product extends in an obvious way to subsets $W \subseteq X^*$ and $P\subseteq X^*\cup X^{\omega }$. For a language W let be the submonoid of $X^*$ generated by W, and by $W^\omega :=\{w_1\cdots w_i\cdots : w_i\in W\setminus \{e\}\}$ we denote the set of infinite strings formed by concatenating words in W. Furthermore |w| is the length of the word $w\in X^*$ and ${\mathbf {pref}(P)}$ (${\mathbf {infix}(P)}$) is the set of all finite prefixes (infixes) of strings in $P\subseteq X^*\cup X^\omega$, in particular, ${\mathbf {pref}(P)}\subseteq {\mathbf {infix}(P)}$. We shall abbreviate $w\in {\mathbf {pref}(\eta )}\ (\eta \in X^*\cup X^\omega )$ by $w\sqsubseteq \eta$. If $n\le |p|$ then p[0..n] is the n-length prefix of $p\in X^{*}\cup X^{\omega }$.

A (deterministic) finite automaton over X is a quintuple $\mathcal {A}=(X,Q,q_{0},\delta ,Q')$ where Q is a finite set of states, $q_{0}\in Q$ the initial state, $\delta : Q\times X\rightarrow Q$ is the transition function, and $Q'\subseteq Q$ is the set of final states. As usual $\delta$ also denotes the continuation of $\delta$ to $Q\times X^{*}$ defined by $\delta (q,e):= q$ and $\delta (q,wx):= \delta (\delta (q,w),x)$.

A language $W\subseteq X^{*}$ is called regular if there is a finite automaton $\mathcal {A}$ such that $W=\{w: \delta (q_{0},w)\in Q'\}$.

3 Randomness by Martingales

If one is asked why a certain event is random then often will be the answer that the event be “unpredictable”. In particular, an $\omega$-word $\xi = x_{1}x_{2}\cdots$ should be random if one cannot win by betting on its digits given other (previous) digits. For automata this yields the following.

Definition 1

(Betting automaton). is a finite betting automaton $:\iff$

• 1. $(X,Q,q_{0},\delta )$ is a finite automaton (without final states) and

• 2. and $\sum _{x\in X}\nu (q,x)\le 1$, for all $q\in Q$.

The automaton starts with capital $\mathcal {V}_{\mathcal {A}}(e)= 1$. After the history $w\in X^{*}$ its capital is $\mathcal {V}_{\mathcal {A}}(w)$ and the automaton bets $\nu (\delta (q_{0},w),x)\cdot \mathcal {V}_{\mathcal {A}}(w)$ on every x as the outcome of the next digit. Its reward is $r\cdot \nu (\delta (q_{0},w),x)\cdot \mathcal {V}_{\mathcal {A}}(w)\ (r=|X|)$ for the next digit x. This results in the following capital function (or martingale).

$$\begin{aligned} \begin{array}{rcl} \mathcal {V}_{\mathcal {A}}(e)&{}:=&{} 1, \text { and}\\ \mathcal {V}_{\mathcal {A}}(wx)&{}:=&{}r\cdot \nu (\delta (q_{0},w),x)\cdot \mathcal {V}_{\mathcal {A}}(w) \end{array} \end{aligned}$$

(1)

In order to formulate the main result we need still the following notion.

Definition 2

(Borel normal $\omega$-word). An $\omega$-word $\xi \in {X}^{\omega }$ is Borel normal iff every subword (infix) $w\in {X}^{*}$ appears with the same frequency:

$$\forall w\left( \lim _{n\rightarrow \infty }\frac{|\{i: i\le n\wedge \xi [0..i]\in {X}^{*}\cdot w\}|}{n}\right) =r^{-|w|}$$

Then Schnorr and Stimm [17] proved the following characterisation of $\omega$-words random w.r.t. finite betting automata.

Theorem 1

([17]). If $\xi \in {X}^{\omega }$ is Borel normal then for every finite automaton $\mathcal {A}$ it holds

• 1. , or

• 2. .

If $\xi \in {X}^{\omega }$ is not Borel normal then there are a finite automaton $\mathcal {A}$ and $\gamma >0$ such that

• 3.

Other recent approaches to relate Borel normality to finite automata can be found e.g. in [2, 3] or [20].

4 Finite-State Dimension

Next we turn to aspects of partial randomness via automaton definable martingales $\mathcal {V}_{\mathcal {A}}$. Finite-state dimension may be seen as the estimate of the maximally achievable exponent $\gamma$ in Theorem 1.3. To this end we define for a betting automaton $\mathcal {A}$ and a non-empty subset $F\subseteq X^{\omega }$

$$\begin{aligned} \alpha _{\mathcal {A}}(F):= \inf \left\{ \alpha : \forall \xi (\xi \in F\rightarrow \limsup _{n\rightarrow \infty } \frac{\mathcal {V}_{\mathcal {A}}(\xi [0..n])}{r^{(1-\alpha )\cdot n}}> 0) \right\} \end{aligned}$$

(2)

Observe that $1-\alpha$ corresponds to the exponent $\gamma$.

Then the finite-state dimension of F is obtained as

$$\begin{aligned} \dim _{\mathrm {FS}}(F) := \sup \left\{ \alpha _{\mathcal {A}}(F): \mathcal {A}\text { is a finite automaton}\right\} \end{aligned}$$

(3)

In this definition we followed Schnorr’s approach via martingales and order functions (cf. [26]) rather than the one by s-gales in [6]. If we replace $\limsup$ in Eq. (2) by $\liminf$ we obtain the so called strong finite-state dimension which has similar properties [7].

As an immediate consequence of Theorem 1 we obtain that $\dim _{\mathrm {FS}}(\xi )=1$ if and only if $\xi$ is Borel normal. One possibility to obtain $\omega$-words of smaller finite-state dimension is by dilution (inserting blocks of zeros) of Borel normal ones. In this way one proves

Lemma 1

([6, Lemma 6.5]). For every rational number there is an $\omega$-word $\xi$ such that $\dim _{\mathrm {FS}}(\xi )=t$.

The papers [4, 6, 7] give several equivalent definitions of finite-state dimension in terms of information-lossless compression by finite-state machines, by log-loss rates of continuous measures¹ on $X^{*}$, or by block-entropy rates.

Combining the results of [6] with the ones of [18, 19] in [8] it was observed that finite-state dimension has also a characterisation via decompression by transducers.

Definition 3

(Finite transducer). $\mathcal {M}= (X,Y, Q,q_{0},\delta ,\lambda )$ is a generalised sequential machine (or finite transducer) if and only if $(X,Q,q_{0},\delta )$ is a finite automaton without final states, Y is an alphabet and $\lambda : Q\times X\rightarrow Y^{*}$

The transducer realises a prefix monotone mapping $\varphi :X^{*}\rightarrow Y^{*}$ in the following way:

$$ \varphi (e) := e,\text { and } \varphi (wx):= \varphi (w)\cdot \lambda (\delta (q_{0},w),x)$$

This mapping can be extended to $\omega$-words via ${\mathbf {pref}(\overline{\varphi }(\eta ))}= {\mathbf {pref}(\varphi ({\mathbf {pref}(\eta )}))}$, that is, $\overline{\varphi }(\eta ):= \lim \limits _{v\rightarrow \eta }\varphi (v)$.

We define the decompression rate $\vartheta _\mathcal {M}(\eta )$ along an input $\eta$ as follows.

Definition 4

(Decompression along an input).

$$\vartheta _\mathcal {M}(\eta ):= \liminf _{n\rightarrow \infty }\frac{n}{|\varphi (\eta [0..n])|},$$

where $\mathcal {M}$ is a finite transducer and $\varphi$ its related mapping.

As the difference $|\varphi (wx)-\varphi (w)|$ is bounded, this quantity measures in some sense the asymptotic amount of digits necessary to obtain the first $\ell$ digits of the output.

Then the finite-state dimension of $\xi \in X^{\omega }$ turns out to be the simultaneous best choice of a transducer $\mathcal {M}$ with a suitable best input $\eta$ generating $\xi =\overline{\varphi }(\eta )$ (cf. [6, 8, 18, 19]).

Theorem 2

Consider the class $\mathcal {K}_{X}$ of transducers $\mathcal {M}$ having output alphabet $Y=X$. Then for all $\xi \in X^{\omega }$ we have

$\dim _{\mathrm {FS}}(\xi )= \inf \bigl \{\vartheta _{\mathcal {M}}(\eta ): \mathcal {M}\in \mathcal {K}_{X} \wedge \eta \in X^{\omega }\wedge \xi =\overline{\varphi }(\eta )\bigr \}$.

We conclude this section by presenting a connection between the finite-state dimension of some set $F\subseteq X^{\omega }$ and the entropy of regular languages W containing ${\mathbf {pref}(F)}$ [4, Theorem 3.5].

The entropy (or entropy rate) $H_{W}$ of a language $W\subseteq X^{*}$ is defined as [4, 22]

$$\begin{aligned} H_{W}:= \limsup _{n\rightarrow \infty }\frac{\log _{r}(1+|W\cap X^{n}|)}{n}. \end{aligned}$$

(4)

The entropy is monotone and stable, that is, $H_{W\cup V}= \max \{H_{W},H_{V}\}$. It should be mentioned that $H_{W}=H_{{\mathbf {pref}(W)}}=H_{{\mathbf {infix}(W)}}$, for regular languages.

Theorem 3

([4]). $\dim _{\mathrm {FS}}(F)\le \inf \{H_{W}: {\mathbf {pref}(F)}\subseteq W\wedge W\text { is regular}\}$

5 Automaton Definable Null Sets

We start this section with introducing $\omega$-languages definable by finite automata. For more background see the books [15, 28] or the surveys [23, 27].

Let $\mathcal {B}=(X,Q,\varDelta , q_0,Q')$ be a non-deterministic (Büchi-)automaton. Then the sequence is a run of $\mathcal {B}$ on the $\omega$-word $\xi = \xi (1)\cdot \xi (2)\cdots$ provided $(q_i,\xi (i+1),q_{i+1})\in \varDelta$ for all . A run is called successful if infinitely many of the $q_{i}$ are in the set of final states $Q'$.

The $\omega$-language $L_{\omega }(\mathcal {B})$ defined by $\mathcal {B}$ is then

$L_{\omega }(\mathcal {B})=\{\xi : \xi \in X^{\omega }\wedge \text { there is a successful run of }\mathcal {B}\text { on }\xi \}$.

Definition 5

(Regular $\omega$-language). An $\omega$-language $F\subseteq X^{\omega }$ is called regular if and only if F is accepted by a finite automaton

The following properties of the class of regular (automaton definable) $\omega$-languages are well-known.

Theorem 4

• 1. An $\omega$-language $F\subseteq X^{\omega }$ is regular if and only if there are an and regular languages $W_{i},V_{i}\subseteq X^*, i\le n,$ such that $F= \bigcup _{i=1}^{n}W_{i}\cdot V_{i}^{\omega }$.

• 2. The set of regular $\omega$-languages over X is closed under Boolean operations.

• 3. If $F\subseteq X^{\omega }$ is regular then ${\mathbf {pref}(F)}$ and ${\mathbf {infix}(F)}$ are regular languages.

Theorem 5

Let $\mathcal {DB}$ be the class of $\omega$-languages accepted by deterministic Büchi automata. Then

• 1. $\mathcal {DB}$ is a proper subclass of the class of regular $\omega$-languages, and

• 2. $\mathcal {DB}$ is closed under union and intersection but not under complementation.

• 3. If $W\subseteq X^{*}$ is a regular language then $\{\xi :\xi \in X^{\omega }\wedge {\mathbf {pref}(\xi )}\subseteq W\}\in \mathcal {DB}$ and $\{\xi :\xi \in X^{\omega }\wedge |{\mathbf {pref}(\xi )}\cap W|=\infty \}\in \mathcal {DB}$.

As measure on the space $X^{\omega }$ we use the usual product measure $\mu$ defined by its values on the cylinder sets $\mu (w\cdot X^{\omega }):= r^{-|w|}$. Then in [21, 24] the following characterisation of regular null sets via “forbidden subwords” is proved.

Theorem 6

Let F be a regular $\omega$-language.

• 1. If $F\in \mathcal {DB}$ then $\mu (F)=0$ if and only if there is word $w\in X^{*}$ such that $F\subseteq X^{\omega }\setminus X^{*}\cdot w\cdot X^{\omega }.$

• 2. $\mu (F)=0$ if and only if $F\subseteq \bigcup _{w\in X^{*}}X^{\omega }\setminus X^{*}\cdot w\cdot X^{\omega }.$

Remark 1

Theorem 6 holds for a much larger class of finite measures on $X^{\omega }$ including all non-degenerated product measures on $X^{\omega }$ (cf. [21, 24, 29, 30]).

Now we can characterise those $\omega$-words which are not contained in a regular $\omega$-language of measure zero.

Definition 6

(Disjunctivity). An $\omega$-word $\xi \in X^{\omega }$ is called disjunctive (or rich or saturated) if and only if it contains every word $w\in X^{*}$ as subword (infix).

Consequently, $\omega$-words random w.r.t. finite automata in the sense of the measure theoretic approach are exactly the disjunctive ones. This allows us to compare both of the presented approaches of randomness.

Proposition 1

Every Borel normal $\omega$-word is disjunctive, but there are disjunctive $\omega$-words which are not Borel normal, e.g. the $\omega$-word $\zeta := \prod _{w\in X^{*}}0^{|w|}\cdot w$.

6 Subword Complexity

The characterisation via “forbidden subwords” enables us to derive a notion of partial randomness similar to the finite-state dimension. To this end we use the entropy of languages defined in Eq. (4) and define for arbitrary $P\subseteq X^{*}\cup X^{\omega }$

Definition 7

(Subword complexity).

$$\begin{aligned} \tau (P):= H_{{\mathbf {infix}(P)}} \end{aligned}$$

In view of the inequality ${\mathbf {infix}(P)}\cap X^{n+m}\subseteq ({\mathbf {infix}(P)}\cap X^{n})\cdot ({\mathbf {infix}(P)}\cap X^{m})$ which holds for ${\mathbf {infix}(P)}$ the limit in Eq. (4) exists and equals

This value is also known as factor complexity in automata theory and topological entropy in symbolic dynamics.

The following is clear.

Proposition 2

$0\le \tau (\xi )\le 1$ and an $\omega$-word $\xi \in X^{\omega }$ is disjunctive if and only if $\tau (\xi )=1$.

For subword complexity one has for every possible value an $\omega$-word of exactly this complexity [12].

Theorem 7

For every $t, 0\le t\le 1,$ there is a $\xi \in X^{\omega }$ such that $\tau (\xi )=t$.

Similar to Eq. (5.1.2) of [22] one can derive the following identity.

$$\begin{aligned} \tau (P)= \inf \{H_{W}: W\subseteq X^{*}\wedge {\mathbf {infix}(P)}\subseteq W\wedge W\text { is regular}\} \end{aligned}$$

(5)

Now Theorem 3 yields the following relation to finite-state dimension.

$$\begin{aligned} \dim _{\mathrm {FS}}F\le \tau (F) \end{aligned}$$

(6)

For certain regular $\omega$-languages $F\subseteq X^{\omega }$ we have identity in Eq. (6).

Proposition 3

Let $F\subseteq X^{\omega }$ be non-empty and regular.

• 1. Then $\max \limits _{\xi \in F}\tau (\xi )$ exists and $\displaystyle \max _{\xi \in F}\tau (\xi )=\max _{\xi \in F}\dim _{\mathrm {FS}}\{\xi \}$.

• 2. If, moreover, $F=\{\xi : {\mathbf {pref}(\xi )}\subseteq {\mathbf {pref}(F)}\}$ then $\tau (F)= \dim _{\mathrm {FS}}F$.

7 Predicting Finite Automata

A further feature of randomness of an $\omega$-word $\xi$, similar to the one mentioned for betting automata, is the impossibility of the exact prediction of the next symbol. Here Tadaki [25] proposed the following.

Definition 8

(Predicting automaton). A transducer $\mathcal {A}= (X,X, Q,q_{0},\delta ,\lambda )$ is referred to as a predicting automaton if $\lambda : Q\rightarrow \{e\}\cup X$ is a labelling of states.

Definition 9

(Prediction). A predicting automaton $\mathcal {A}= (X,X, Q,q_{0},\delta ,\lambda )$ strongly predicts $\xi \in X^{\omega }$ if and only if

• 1. $\lambda (\delta (q_{0},\xi [0..n-1]))= \xi (n)$ for infinitely many , and

• 2. if $\lambda (\delta (q_{0},\xi [0..n-1]))\ne \xi (n)$ then $\lambda (\delta (q_{0},\xi [0..n-1]))=e$.

Definition 9 is a strong requirement, it forces the automaton to make on input $\xi$ infinitely many correct predictions and no incorrect ones. Here using the label $\lambda (q)=e$ the automaton may skip. Nevertheless, in the binary case $X=\{0,1\}$ we have the following.

Theorem 8

• 1. Let $\mathcal {A}= (\{0,1\},\{0,1\}, Q,q_{0},\delta ,\lambda )$ be a binary predicting automaton. If $\mathcal {A}$ strongly predicts $\xi \in \{0,1\}^{\omega }$ then $\xi$ is not disjunctive.

• 2. If $\xi \in \{0,1\}^{\omega }$ is disjunctive then no predicting automaton predicts $\xi$.

This theorem does not hold in the other cases when $|X|\ge 3$. Here we have to turn to “negative” prediction. We say that $\mathcal {A}$ weakly predicts $\xi$ provided $\lambda (\delta (q_{0},\xi [0..n-1]))\ne \xi (n)$ for infinitely many and $\lambda (\delta (q_{0},\xi [0..n-1]))=e$ otherwise. Then we have.

Theorem 9

• 1. Let $\mathcal {A}= (X,X, Q,q_{0},\delta ,\lambda )$ be a binary predicting automaton. If $\mathcal {A}$ weakly predicts $\xi \in X^{\omega }$ then $\xi$ is not disjunctive.

• 2. If $\xi \in X^{\omega }$ is disjunctive then no predicting automaton weakly predicts $\xi$.

8 Finite-State Genericity

This section reviews some connections between disjunctivity and finite-state genericity. As in [1] we define the following.

Definition 10

Let $\xi \in X^{\omega }$.

• 1. $\xi$ meets a function $\psi : X^{*}\rightarrow X^{*}$ if $w\cdot \psi (w)\sqsubset \xi$.

• 2. $\xi$ is finite-state generic if $\xi$ meets every function $\varphi$ realised by a finite transducer.

This can be interpreted in terms of the usual product topology on $X^{\omega }$ which can be defined by the metric $\varrho (\xi ,\eta ):= \sup \{r^{-n}: \xi (n)\ne \eta (n)\}$ where we agree on $\sup \emptyset =0$. The cylinder sets $w\cdot X^{\omega }$ are simultaneously open and closed balls of diameter $r^{-|w|}$. The closure $\mathcal {C}(F)$ of (smallest closed set containing) a set $F\subseteq X^{\omega }$ obtains as $\mathcal {C}(F)=\{\xi : {\mathbf {pref}(\xi )}\subseteq {\mathbf {pref}(F)}\}$.

A subset F is nowhere dense if its closure does not contain a non-empty open subset, that is, for every $w\in {\mathbf {pref}(F)}$ there is a continuation $v\in X^{*}$ such that $w\cdot v\cdot X^{\omega }\cap F=\emptyset$, that is “v leads w to a hole” in F.

Then Definition 10.2 gives an indication that finite-state generic $\omega$-words avoid “finite-state nowhere dense” subsets of $X^{\omega }$. This is shown by Theorem 4.4 of [1].

Theorem 10

([1]). An $\omega$-word $\xi$ is disjunctive if and only if it is finite-state generic.

Theorem 10 fits into the more general coincidence of measure and category for regular $\omega$-languages depicted in Fig. 1 (see [21, 24, 29, 30]). In the general case, however, the monograph [14] shows that measure and category (topological density) are two concepts which do no coincide.

Fig. 1.

Coincidence of measure and category for regular $\omega$-languages

As usual a subset F is meagre or of first Baire category if it is an at most countable union of nowhere dense sets, a set is of second Baire category if it is not meagre, and it is residual if its complement is meagre. The first column of Fig. 1 presents a comparison of the sizes of $F\subseteq X^{\omega }$, and the rows indicate that for regular $\omega$-languages $F\subseteq X^{\omega }$ properties of the same row coincide, e.g. $\mu (F)=0$ iff F is meagre.