Quality assessment of commercial bread samples based on breadcrumb features and freshness analysis using an ultrasonic machine vision (UVS) system

Abstract

This paper presents the use of a in situ developed ultrasonic machine vision system for quality parameter extraction of breadcrumb features and freshness. An image processing technique has been used for breadcrumb analysis on collected digital images of various bread samples while an ultrasonic assessment technique has been used for quantification of the freshness of various bread samples. Various threshold methods (isodata, Otsu, minimum error, moment preserving and fuzzy method) have been implemented and compared with the proposed method to segment breadcrumbs from collected digital bread images. Threshold performance was assessed by two important criteria such as uniformity and busyness (arrangement of a pixel to its neighborhood pixels) of the binary versions of input breadcrumb sample images. Quality parameters were computed for each optimal threshold on 500 digital images of bread slices. Other important quality parameter of bread is the outline of its brown color section, which corresponds to the appropriate baking stage. Slight variations in threshold lead to substantial variations in crumb feature values, with cell uniformity, void fraction, intensity and entropy calculation showing more sensitivity than others. Propagation delay and attenuation in the received acoustic signal have been calculated for stiffness and firmness evaluation. A second order relationship has been observed between the storage time and stiffness of the various bread samples. The proposed method is very efficient in the sense of quality parameter calculations. Although some of the previously reported methods showed a relatively higher amount of busyness than other methods, the reported method performs well on images with large void areas.

Introduction

Bread is one of the most consumed bakery items by people all over the world, therefore, the quality and freshness of bread are important factors for the customer as well the baker. The quality of any bread depends on various features such as crumb structure, crumb cell density, cell uniformity average area of the cells, void fraction, energy, texture quality and energy, which relates to the brightness of a chosen sample of bread. Freshness is also an important parameter from the consumer point of view which can be further related to the storage time of bread samples. Bread baking is a traditional process in which water, enzyme activity, starch, protein properties and heat are the principal factors that ultimately determine bread quality [1]. The shelf life and the freshness decide the distribution distance. Wholesale distribution of bread requires a shelf life of 5 days or more and freshness [2]. It has been reported that the state of freshness of bread is evaluated in terms of its taste, softness and water absorptive capacity [3]. Various analytical methods for determining bread freshness and staleness are used. Some of these methods include: (I) differential scanning calorimetry; (II) texture analysis and (III) near infrared spectroscopy [4]. These methods can extract different parameters such as starch, protein and packaging from bread samples, but all are still very costly and too complex for regular use. Bread texture is related to its geometric and mechanical properties, which also depend on its cellular structure. Bread tastes best when freshly baked. As the number of days after baking increases biological and chemical changes take place in the bread, some internally and some caused by external environmental factors. As a result of these unpredictable changes, the quality of the bread samples is reduced due to various effects such as microbiological effects, various contaminants, etc. [5]. Smolarz, van Hecke, and Bouvier reported a mathematical morphology-based model to define the structural elements of extruded biscuits and to determine various biscuit types [6]. More advanced visual techniques including digital image analysis with various kind of microscopy has been reported by Crowley, Graul, and Arendt [7]. Scanlon and Zghal reported the extraction of mechanical properties such as cell structure calculation, crumb analysis, etc. [8]. Crumb grain, defined by size, the distribution and the shape of cells and cell wall thickness parameter extraction based on algorithms has been reported by Kamman [9]. Gonzales-Barron and Butler reported a comparison of seven thresholding techniques with the k-means clustering algorithm for measurement of breadcrumb features in digital image analysis [10]. Lagrain, Boeckx, Wilderjans and Lauriks developed a non-contact ultrasound characterization model for breadcrumb analysis using a Biot-Allard model [11]. Aouzale, Chitnalah and Jakjoud reported the development of acoustical parameters based on the SPICE model for milk quality parameter extraction. In [12] the authors discuss various quality parameters based on the propagation delay and attenuation of the received acoustic signal. Aboonajmi, Akram, Nishizu, Kondo, Setarehdan and Raja Bipour developed an ultrasound-based technique for the determination of poultry egg quality parameter extraction based on various parameters of the reflected ultrasonic acoustic signal such as attenuation and propagation delay [13]. Valente, Lauxand Prades published a contribution about changes in the ultrasound characteristics of mango juice during ripening. The authors used a 25 MHz transmitter/receiver pair for development of an experimental setup for received ultrasonic acoustic signal features [14]. Gan, Pallav and Hutchins reported an application concerning non-contact ultrasonic quality measurements of food products based on various acoustical features such as propagation delay velocity and attenuation. This paper presents the design and development of a robust machine vision algorithm for bread-crumb feature extraction with an ultrasound-based system for stiffness extraction for different bread sample brands and experiments carried out along with the results obtained [15]. Furferi, Governi and Volpe [16] presented an artificial neural network method for predicting olive ripening based on the extracted RGB color index.

Materials and methods

Bread quality can be divided into two broad categories according to consumer preferences all over the world. Stiffness of the various bread samples indicates the freshness of the corresponding bread samples while breadcrumb features depict the quality of the bread samples at the time of baking.

Sample preparation and vision sensing database generation

The bread used for the various experiments covered a variety of brands such as Amul, Mercury and Britannia obtained from local markets. Twenty bread loves were sliced transversely using a mechanical slicer to obtain 20-mm thick slices. Ten central slices of each loaf were then subjected to the data acquisition and image generation processes for vision and ultrasonic assessment.

A 10 megapixel USB-based camera has been used with a monochromatic light source for image acquisition. An image-based dataset of a total of 15 images has been prepared for breadcrumb analysis. NT scanning software allowed the brightness and contrast to be adjusted to −1 and 40 for each color plane (RGB), respectively. Bread samples were preserved under four different environmental conditions: (a) controlled; (b) frozen; (c) open and (d) packaged (Fig. 1). The centre of each image (slice) was cropped to 500 × 500 pixels (representing 40 × 40 mm² of the slice area) and converted into 8 bit gray level images (Fig. 2).

Fig. 1

Bread samples preserved under different environmental conditions a controlled (T = 40 °C and RH = 80 %); b frozen (T = 8 °C and RH = 60–65 %); c open (T = 27 °C and RH = 45–55 %); d packaged

Fig. 2

Typical breadcrumb RGB images of Amul, Mercury and Britannia brand samples. Grey level converted images of 40 × 40 mm field of view

Experimental setup and data collection

The experimental setup (Fig. 3) used for breadcrumb and stiffness feature extraction was composed of two major sections, namely a vision sensing setup and an ultrasonic sensing setup, together with an ARM-11 processor-based processing and control unit. The vision sampling chamber comprises a cylindrical chamber with a 10.0 megapixel USB-based camera mounted on a stepper motor. The stepper motor can change the angle from −180° to +180° according to the control and processing unit for image acquisition from various angles. The processing and control unit communicates with the stepper motor and USB-based camera using a serial port and USB protocol, respectively. The vision sensing chamber also has a monochromatic light source to provide light at the time of image acquisition. The light intensity can also be controlled using cylindrical slits according to the contrast variation using the processing and control unit. The ARM-11-based processor is interfaced with the signal conditioning circuitry, a touch screen, and a Zigbee module for wireless data transfer. A qtopia 2.2-based graphical user interface was also developed to control the various operations such as image acquisition, data storage and wireless data transfer. The ultrasonic sampling chamber is composed of a black color aluminum chamber with an acoustic sensing module (ultrasonic transmitter and receiver couple) having 40 kHz as a center frequency with 2.0 kHz bandwidth, plus the required data acquisition circuits for peak detection in the received acoustic signals.

Fig. 3

a Beam angle variation (narrow beam width and wide beam width); b transmission pressure level with frequency variation; c developed ultra-vision system for food quality inspection

The ultrasonic and vision sensing datasets have been collected using the developed ultrasonic vision sensing (UVS) experimental setup. Three major acoustic signals represent the region of interest for various kinds of stiffness and firmness analysis of the various bread brand samples: transmitted, received and converted acoustic signals. The transmitted and received acoustic signals have been modeled using standard linear chirp signals:

$$ X\left( t \right)\,\, = \,\,\sin \left\{ {\emptyset_{0} + 2\pi (f_{0} t + 0.5kt^{2} )} \right\} $$

(1)

$\emptyset_{0}$, f ₀, k and t correspond to the initial phase difference, initial fundamental frequency, chirp rate and time, respectively:

$$ k\,\, = \,\,\left( {f_{1} - f_{0} } \right)/t_{1} $$

(2)

The transmitted acoustic signal incides on the bread sample and reflects back to the ultrasonic receiver after some energy loss due to absorbance by the bread sample. The received acoustic signal is subjected to measurement of two physical parameters, namely attenuation and propagation delay, which correspond to stiffness and firmness feature extraction. Monitoring parameters (Fig. 4) for stiffness and firmness analysis of the various bread samples are as follows:

Fig. 4

Collected acoustic dataset for stiffness and firmness analysis

$$ {{Tpd}}\,\, = \,\,T_{{start}} \,\, - \,\,T_{{stop}} $$

$$ {{Attenuation}}\,\, = \,\,A_{{transmitted}} \,\, - \,\,A_{{recieved}} $$

$$ {{Attenuation}}\left( {d.b.} \right) = 20\log {{Attenuation}} $$

(3)

Algorithm development and data analysis

A robust machine vision algorithm has been developed for breadcrumb feature analysis. A sharpening filter has been implemented for sharpening the breadcrumb images to extract the appropriate features. Six segmentation techniques have been implemented for segmentation and compared with the proposed manual threshold method. Various segmentation techniques have been implemented such as the manual, Ostu’s, isodata, moment preserving, minimum error and fuzzy c mean thresholding techniques. Segmented images have been processed to obtain the various crumb physical parameters such as number of cells, crumb cell density, cell area, etc. Breadcrumb features have been analyzed for bread samples preserved under different environmental conditions such as packed, frozen and controlled and open (Fig. 1). Various color models such as RGB, HSI and YCbCr have been implemented for the extraction of the brightness, texture properties, etc. Various pattern recognition techniques also have been implemented for classification of the various classes of bread samples such as rotten, fresh and frozen. Data analysis has also been implemented using the collected dataset from the ultrasonic module of the developed system. Attenuation and propagation delay have been calculated for stiffness and firmness extraction, which relates with the storage time of the various brands of bread samples.

Crumb sharpening for feature extraction

It has been observed that sharpening the breadcrumb images provides more accurate results compared to input images without sharpening. The technique basically used to enhance the breadcrumb structure features is given by:

$$ g\left( {m,n} \right)\,\, = \,\,\mathop \sum \limits_{l = - L}^{L} \mathop \sum \limits_{k = - L}^{L} h\left( {k, l} \right)\,\, \times \,\,s(m - k, n - l) $$

(4)

where g(m, n), h(k, l) and s(m − k, n − l) correspond to the sharpened image, kernel of the sharpening filter and input image, respectively. The kernel of the sharpening filter can be expressed as follows:

$$ h\left( {3,3} \right) = \left| {\begin{array}{*{20}c} 0 & { - 1} & 0 \\ { - 1} & 5 & { - 1} \\ 0 & { - 1} & 0 \\ \end{array} } \right| $$

(5)

Implemented thresholding techniques for segmentation

Six standard thresholding techniques (Fig. 5) for segmentation purposes have been implemented and compared. The exact calculated value of the threshold has a considerable effect on the boundary position and therefore on the overall size of the extracted objects. Ostu’s method [17] has been used to automatically perform clustering-based image thresholding, or there duction of a gray level image to a binary image. This technique assumes that the image contains two classes of pixels following a bimodal histogram (foreground and background pixels). The system then calculates the optimum threshold separating the two classes, so that their combined spread (intra-class variance) is minimal. In this method, algorithm exhaustively extracts the threshold that minimizes the intra-class variance (within the class), which can be defined as a weighted sum of variances of two classes, such as:

Fig. 5

Binary images of a randomly selected bread sample digital image using: a Ostu’s thresholding (thresh = 0.5614); b isodata thresholding (thresh = 0.8134); c fuzzy-based thresholding (thresh = 0.47843); d moment preserving thresholding method (thresh = 0.5342); e minimum error thresholding (thresh = 0.9961); f the proposed manual threshold method

$$ \sigma_{\omega }^{2} \left( t \right) = q_{1} \left( t \right)\sigma_{1}^{2} \left( t \right) + q_{2} \left( t \right)\sigma_{2}^{2} \left( t \right) $$

(6)

${\text{where }}q_{i} \left( t \right)$ represents the probability of the two classes separated by a threshold t and σ ² _i variance of these classes:

$$ q_{1} \left( t \right) = \mathop \sum \limits_{i = 1}^{n} P(i) $$

$$ q_{2} \left( t \right) = \mathop \sum \limits_{i = n + 1}^{I} P(i) $$

(7)

Ostu showed that minimizing the intra-class variance is the same as maximizing the inter-class variance:

$$ \sigma_{b}^{2} \left( t \right) = \sigma^{2} - \sigma_{\omega }^{2} \left( t \right) $$

(8)

$$ \sigma_{b}^{2} \left( t \right) = \omega_{1} (t) \omega_{2} (t)\left[ {\mu_{1} \left( t \right) - \mu_{2} \left( t \right)} \right]^{2} $$

(9)

ω _i , μ _i correspond to the class probabilities and means, respectively. Class probabilities ω _i have been computed from the extracted histogram as t while class means μ _i is:

$$ \mu_{1} \left( t \right) = \mathop \sum \limits_{i = 1}^{n} \frac{iP\left( i \right)}{{q_{1} \left( t \right)}} $$

$$ \mu_{2} \left( t \right) = \mathop \sum \limits_{i = n + 1}^{I} \frac{iP(i)}{{q_{2} \left( t \right)}} $$

(10)

In the minimum error thresholding algorithm developed by Kittler and Illingworth, the gray level histogram is viewed as an estimate of the probability density function of the mixture population comprising of the grey levels of the objects and background (j = 1, 2, 3). Each element of the GL image is normally distributed with mean m (j) and standard deviation $\sigma (j)$ and probability of p(j). These parameters are not usually known and a creation function J(t) is used instead:

$$ J\left( t \right) = 1\,\, + \,2\,\, \times \,\,\left[ {P_{1 } \left( t \right)\ln \sigma_{1 } \left( t \right) + P_{2 } \left( t \right)\ln \sigma_{2 } \left( t \right)} \right] - 2\left[ {P_{1 } \left( t \right)\ln P_{1} \left( t \right) + P_{2 } \left( t \right)\ln P_{2} \left( t \right)} \right] $$

(11)

where

$$ P_{1} \left( t \right) = \mathop \sum \limits_{i = 1}^{I} P(i) $$

$$ P_{1} \left( t \right) = \mathop \sum \limits_{i = I + 1}^{L - 1} P(i) = 1 - P_{1 } (t) $$

$$ m_{1} \left( t \right) = \,\frac{1}{{P_{1} \left( t \right)}}\mathop \sum \limits_{i = 0}^{I} i\,\, \times \,P(i) $$

$$ m_{2} \left( t \right) = \,\,\frac{1}{{P_{2} \left( t \right)}}\mathop \sum \limits_{i = I + 1}^{L - 1} i\,\, \times \,P(i) $$

$$ \sigma_{1} \left( t \right)\,\, = \,\,\frac{1}{{P_{1} \left( t \right)}}\mathop {\,\sum }\limits_{i = 0}^{I} [1 - m_{1} \left( t \right)]^{2} \, \times \,P(i) $$

$$ \sigma_{2} \left( t \right)\,\, = \,\,\frac{1}{{P_{2} \left( t \right)}}\mathop \sum \limits_{i = I + 1}^{L - 1} [1 - m_{2} \left( t \right)]^{2} \,\, \times \,P(i) $$

(12)

The optimal threshold t* is obtained by:

$$ J\left( {t^{*} } \right) = \hbox{min} \,\,J(t) $$

Fuzzy set theory has been used to partition an image space by minimizing the measure of fuzziness of the image. This measurement can be expressed as entropy, which is an index of fuzziness and non-fuzziness. In fuzzy set notation, the image set I = f(x, y) of size M × N can be expressed as:

$$ {\text{I }} = \, \left\{ {\left( {{\text{f}}\left( {{\text{x}},{\text{y}}} \right), \, \mu_{ 1} \left( {{\text{f}}\left( {{\text{x}},{\text{y}}} \right)} \right)} \right)} \right\} $$

(13)

The membership function levels µ_I(f(x, y)) can be viewed as a characteristic equation that can extract the fuzziness of a(x, y) pixel in given image I:

$$ \frac{1}{{1 + |f\left( {x,y} \right) - \mu_{0 (t)|/c} }}{\text{if }}\,\,{\text{f}}\left( {{\text{x}},{\text{y}}} \right) \, \le {\text{ t}} $$

(14)

$$ \mu_{ 1} \left( {{\text{f}}\left( {{\text{x}},{\text{y}}} \right)} \right) \, = \frac{1}{{1 + |f\left( {x,y} \right) - \mu_{1 (t)|/c} }}{\text{if f}}\left( {{\text{x}},{\text{y}}} \right) \, > {\text{ t}} $$

(15)

Average grey levels µ₀(t) and µ₁(t) can be regarded as target values of the background and the object for a given threshold value t; and C is a constant value such that 0.5 ≤ µ₁(f(x, y)) ≤ 1:

$$ \mu_{0} \left( t \right) = \,\,\frac{{\sum\limits_{i = 0}^{I} {i h(i)} }}{{\sum\limits_{I = 0}^{I} {h(i)} }} $$

$$ \mu_{1} \left( t \right) = \frac{{\sum\limits_{i = t + 1}^{L - 1} {i h(i)} }}{{\sum\limits_{i = I + 1}^{IL - 1} {h(i)} }} $$

(16)

The measure of fuzziness that was used in this work was the entropy E(I) obtained by using Shannon’s function S(µ(x_i)). The optimal threshold can be extracted by minimizing the measure of fuzziness E(X).

Morphological analysis has been implemented to extract the segmentation of bread samples as input images. This approach is especially designed to extract breadcrumb features perfectly. The processing starts with a gray-level thinning, followed by a gray-level opening, so that region segmentation is obtained. A morphological erosion operation has been used for extracting the breadcrumbs from binary images of the various bread samples. In order to perform erosion, a 3 × 3 structuring element has been used over the binary segmented image. Boundary extraction also has been performed for area extraction of various cells existing in breadcrumbs as well as also some uneven cells which influence bread quality:

$$ Eroded\left( {x,y} \right) = { \hbox{max} }\left( {\mathop \sum \limits_{i = 0}^{n - 1} \mathop \sum \limits_{j = 0}^{n - 1} I\left( {i,j} \right) \times w\left( {3x3} \right)} \right) $$

(17)

Brightness extraction using traditional color models

Brightness of the various bread samples is one of the important quality parameters according to consumers around the world. The HSV color model has been implemented (Table 1) for calculation of the amount of brightness. The HSV color model is designed in the way that humans perceive color in various objects. A particular color feature such as brightness can be easily recognized by HSV color model segmentation. Brightness can be easily calculated by segmenting the value coordinates of hue and saturation. Every RGB input bread sample digital image has been converted to a HSV color model image and segmented into value coordinates, which indicate the brightness of the various bread sample images:

Table 1 HSV color extraction of various bread samples preserved under different environmental conditions

$$ R^{\prime} = R/ 2 5 5G^{\prime} = G/ 2 5 5B^{\prime} = B/ 2 5 5 $$

$$ C_{ \hbox{max} } = { \hbox{max} }\left( {R^{\prime},G^{\prime}, B^{\prime}} \right){\text{ C}}_{ \hbox{min} } = { \hbox{min} }\left( {R^{\prime},G^{\prime}, B^{\prime}} \right) $$

$$ \Delta = C_{ \hbox{max} } - C_{ \hbox{min} } $$

(18)

$$ S = \begin{array}{*{20}c} {0 \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,if\,\Delta = 0} \\ {\frac{\Delta }{{C_{ \hbox{max} } }} \,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,{\text{otherwise}}} \\ \end{array} $$

$$ V = C_{ \hbox{max} } $$

$$ H = \begin{array}{*{20}c} {60^{ \circ } \times \,\,\left( {\frac{{\left( {(G^{\prime} - B^{\prime}} \right)}}{\Delta }} \right)\bmod 6, \,C_{\hbox{max} } = R^{\prime}} \\ {60^{ \circ } \times \,\left( {\frac{{\left( {(B^{\prime} - R^{\prime}} \right)}}{\Delta }} \right) + 2, \,C_{\hbox{max} } = G^{\prime}} \\ {60^{ \circ } \times \,\,\left( {\frac{{\left( {(R^{\prime} - G^{\prime}} \right)}}{\Delta }} \right) + 4,\, C_{\hbox{max} } = B^{\prime}} \\ \end{array} $$

(19)

It has been observed that the value channel varies in the various bread samples preserved under different environmental conditions. The value channel indicates the amount of brightness of various bread samples, which is one of the important quality parameter according to consumers. Brightness of the various bread samples varies with changes in the different environmental conditions such as open, packed, controlled, frozen and frozen.

Physical breadcrumb feature calculation

Automated robust algorithms have been developed for breadcrumb segmentation. Traditional and proposed thresholding have been programmed in Matlab 6.12 (The Math Works Inc., Natick, MA, USA) and tested on bread samples of various brands. The accuracy and efficiency of every segmentation technique depend on the selection of threshold within the given gray level of the digital image of a bread sample. The objective of this algorithm is to convert the input image to a gray level bread sample digital image and extract an exact binary image from it, which can be further processed for breadcrumb parameter extraction. The various crumb grain features extracted from the converted binary images by the various traditional and proposed thresholding algorithms are: (I) cell density (number of cells/cm²); (II) mean cell area in mm² (total area of the bread cells/number of total cells);(III) cell uniformity (uniform distribution of cells over the uniform area of various bread cells); (IV) void fraction (the fraction of the area corresponding to the bread pores); (V) number of cells (total number of cells over the total area of bread sample).

The system processing time for overall analysis of a single crumb of bread varies approximately between 3 s to 23 s using an ARM-11 processor-based system with 64 MB RAM operating at a maximum frequency of 533 MHz. Texture parameters (Table 2) of the various bread have been extracted using entropy calculation of the various breadcrumbs. Entropy of an input bread sample has been calculated using the following relation:

Table 2 Physical quality parameter calculation of various breadcrumb structures

$$ H\left( {x,\,y,\, z} \right) = \mathop \sum \limits_{i = 1}^{i = n} \mathop \sum \limits_{j = 1}^{j = n} \mathop \sum \limits_{k = 1}^{3} P \left( {i, \,j,\, k} \right)\,\, \times \,\,\log_{2} P(i, j, k) $$

$$ P\left( {x,\,y,\, z} \right) = \mathop \sum \limits_{i = 1}^{i = n} \mathop \sum \limits_{j = 1}^{j = n} \mathop \sum \limits_{k = 1}^{3} \frac{n(i, \,j,\, k)}{{m^{2} }} $$

(20)

Entropy is used to characterize the various texture parameters of the input image by a statistical measure of randomness. The area of the various cells present in selected breadcrumbs has been calculated and finally converted into a mean cell area using the abovementioned definition.

Freshness analysis using stiffness and firmness analysis

Freshness analysis of the various bread samples has also been performed using our ultrasonic stiffness and firmness monitoring system. All collected bread samples have been preserved in four different environments as mentioned. All samples went through the process of data collection over the time period. Two parameters have been analyzed for freshness parameter extraction, namely propagation delay between the transmitted and received ultrasonic signal and attenuation introduced in transmitted signal due to a significant loss that occurs in the transmitted signal energy. Propagation delays between transmitted and received acoustic signal have been derived from time of flight calculations while attenuation has been derived from the changes in the transmitted and received acoustic signals.

A non-contact ultrasonic system already has been developed for tomato quality inspection [18]. The same experimental setup has been used with some of the needed updates for firmness and stiffness extraction (Fig. 6) for the various bread brand samples, which was finally converted into a degree of freshness of the bread samples. The developed vision sensing system can discriminate between the various classes of the bread samples such as rotten (i.e., infected with various fungal infections) and fresh bread samples. It has been observed that the acoustic technique is much more effective compared to the vision sensing technique in the case of freshness analysis. It has been observed that the attenuation in the transmitted ultrasonic signal decreases and the propagation delay between transmitted and received ultrasonic signal also decreases over the time period of the analysis. A second order non-linear relationship has been observed between attenuation and freshness of the various bread samples.

Fig. 6

Acoustic responses for various bread samples (open environment) at the sample distance from sensing module = 20 cm; a day 1 to day 2 (attenuation = 6.8 db and propagation delay = 380 ms); b day 3 to day 5 (attenuation = 7.4 db and propagation delay = 430 ms); c day 6 to day 7 (attenuation = 8.6 db and propagation delay = 580 ms; d Variation of attenuation with different environmental conditions (frozen, controlled and open) (R ² = 0.97). transmitted acoustic signal (20 Vpp). received ultrasonic signal. received ultrasonic signal converted into a square wave pulse form (Color figure online)

Various vision sensing algorithms were also developed for discrimination between various classes of the bread samples such as fresh, less fresh and rotten. An RGB model and various pattern recognition algorithms such as PCA, LDA, QDA and KNN (Fig. 7) have been implemented for discrimination between various classes of bread samples. It has been observed that the disease that normally attacks the various bread samples is a regular fungus. RGB model extraction is able to extract the change in color caused by the fungus attack. An infection index has been defined using the extracted percentage change introduced in the green color index when a fungus attacks the various bread samples.

Fig. 7

Classification between various classes of bread samples using various pattern recognition algorithms a fresh bread sample; b less fresh bread sample; c rotten bread sample; d implemented principle component analysis (PC₁ = 92.8 % and PC₂ = 7.2 %); e linear discrimination analysis (Y = X + 10 and 4X − 5Y = 20); f quadratic discrimination algorithm (4(X − 10)² + (Y − 1)² = 16 and Y = X² – 2 × sqrt(3)); g K-nearest neighbors algorithm. fresh bread sample (day 1–day 2). less fresh bread sample (day 3–day 4). rotten bread sample (day 5–day 6) (Color figure online)

It has been observed that the developed image processing technique is able to discriminate between two broad bread classes such as fresh and infected. An RGB color model has been implemented on various classes of the bread samples and it has been observed that green color percentage varies in a uniform manner with the variation of stages of the bread samples such as fresh, less fresh and rotten. Principle component analysis (PCA) is basically a statistical procedure that uses an orthogonal transformation to convert a set of observations of possibly correlated variables into a set of values of linearly uncorrelated variables called principle components. The number of principle components is less than or equal to the number of original variables. The transformation is defined in such a way that the first principle component has a large possible variance (that is, it accounts for as much of the variability in the data as possible), and each succeeding component in turn has the highest variance possible under the constraint that it is orthogonal to (i.e., uncorrelated with) the preceding components. The developed system performs PCA on segmented bread sample images and generates different principle components for classification:

$$ C\left( {x,y} \right) = \mathop \sum \limits_{q = 1}^{N} X\left( {q, x} \right)\, \times \,X(q, y) $$

$$ Det \left( {C\left( {x,y} \right) - \lambda } \right) = 0 $$

(21)

Generating Eigen values $\lambda_{1, } \ldots e\lambda_{n, }$ and eigenvectors $V_{1, } \ldots dV_{n}$.

$$ V = \left[ { V_{1, } \ldots V_{n } } \right] $$

$$ Y = XV $$

Different analyses were also implemented in the developed algorithm to discriminate various patterns. LDA helps to predict the group membership for a number of subjects from a set of predictor variables. The creation variable (grouping variable) is the object of classification. The group variable also acts as a categorical variable, which extracts different datasets collected from segmented bread sample images. Linear discriminate analysis constructs one or more than on discriminate analysis D _i (linear combinations of the predictor variables X _k ) such that the different groups differ as much as possible on D:

$$ D_{i} = b_{0} + \mathop \sum \limits_{k = 1}^{p} b_{k} \,\, \times \,X_{k} $$

(22)

More precisely, the weights of the discriminant function are calculated in such a way that the ratio (between groups of sample space and within groups of sample space) is as large as possible. The first discriminant function D₁ distinguishes the first group from groups 2, 3, …, N and in the same way a second discriminant function D₂ distinguishes the first group from groups 3, …, N and the same for others. The system calculates the optimal weights using a training set consisting of the correct classification for a group of datasets of the different bread samples.

After this process, the system classifies different datasets using the previously calculated discriminant weights to obtain their discriminant scores. Some datasets are too much complex due to the close stages of the bread samples, so sometimes LDA is not able to discriminate between such kinds of data sets. Quadratic discriminant analysis has been implemented to discriminate very complex data sets of the bread samples. All the steps in LDA and QDA are the same except the classifiers. In LDA a linear classifier is used while in QDA a second order classifier is used. QDA provides more discrimination capability to the developed system, but at the same time it also requires more memory as well as more processing time. LDA needs to estimate K X P + P X P parameters while QDA needs to estimate K X P + P X P X P parameters. The general structure of classifier used in QDA implementation is presented as follows:

$$ Yqda = { \arg } { \hbox{min} }_{k = 1, \ldots .K } \left\{ {\left( {x - \mu_{k} } \right)^{T} \mathop \sum \limits_{k = 1}^{n} \left( {x - \mu_{k} } \right) - 2 \log \left( {\pi_{k} } \right) + \log Z} \right\} $$

$$ Z = \frac{1}{{n_{k} }}\mathop \sum \limits_{k = 1}^{n} \left( {x - \mu_{k} } \right)\, \times \,\left( {x - \mu_{k} } \right)^{T} $$

(23)

It has been observed that the implemented pattern recognition techniques such as PCA, LDA, QDA and KNN are also able to discriminate between the various classes of the bread samples such as fresh, less fresh and rotten.

Defects detection using extra-large pore segmentation

Pore distribution is one of the more important quality parameters, which is indicative of the fermentation process. Pores are the air packets found in various kinds of bread samples, where carbon dioxide from the fermentation process creates a network of primarily interconnected void structures. Pore size varies between varieties of bread. Sourdough bread is a variety with larger pores while rye bread has smaller and denser crumbs, so basically the pore structure distribution is a result of the fermentation processes of the various bread samples. According to the various bread manufacturing industries, there is a threshold area for various pores. Some of the bread samples consist of larger pores due to errors introduced at the time of bread manufacturing and those samples get mixed with the other bread samples. Bread samples consisting of pores having a larger area have a fast fermentation process due to easy formation of carbon dioxide, so there is an essential need to segment such pores having larger area and also classify normal bread samples and defective bread samples.

Bread samples with defects like larger pores have been segmented using an algorithm specially designed for pore segmentation. The developed algorithm consists of various image processing operations such as gray level conversion, binary image conversion with appropriate threshold, etc. Input images are first converted into gray level images in order to calculate the entropy in different local areas of the converted gray level images of randomly selected bread samples. The entropy extraction process is basically composed of various steps such as first it returns the array J, where each output pixel contains the entropy value of the 9-by-9 neighborhood around the corresponding pixel in the converted gray level image. For pixels on the border of a gray level image, the developed algorithm uses symmetric padding. In symmetric padding, the values of pixels to be padded are a mirror reflection of the border pixels in the converted gray level image. The converted entropy image (Fig. 8) is finally converted into a binary image with appropriate threshold calculation using the proposed threshold method based on various morphological operations. The converted entropy binary image is further transferred to trace uneven pore region boundaries using hole segmentation. All regular pores having a certain circular kind of holes are segmented using the difference between the boundaries of a regular hole and an uneven hole. The algorithm basically traces an exterior boundary of objects, as well as boundaries of holes inside these objects. It also descends into the outermost objects (parents) and traces their children (objects completely enclosed by the parents). BW must be a binary image where non-zero pixels belong to an object and 0-pixels constitute the background. B is a P-by-1 cell array, where P is the number of objects and holes. Each cell contains a Q-by-2 matrix, where Q is the number of boundary pixels for the corresponding region. Each row of these Q-by-2 matrices contains the row and column coordinates of a boundary pixel. The coordinates are ordered in a clockwise direction.

Fig. 8

Local entropy image conversion from a gray-level converted input bread sample image

It can be observed from Table 3 that the developed algorithm can detect and segment the extra-large pores from bread sample digital images. Histogram analysis also has been also implemented to classify bread samples having extra-large pores due to an increased fermentation time period. If the system can distinguish the bread samples having extra-large pores from regular bread samples then consumers can preserve a package of bread longer compared to a regular time period. It has been observed that the developed novel robust algorithm can discriminate between bread samples which have more storage time compared to regular bread samples having extra-large pores.

Table 3 Segmented pores having extra-large area compared to regular pores

A significant change has been observed in an implementation of histogram analysis on three different bread samples such as samples not having large pores, samples having large pores with much less area and samples having unavoidable large area pores. An approximate symmetry has been observed in all three histograms, except for some disturbances due to the existence of extra-large pores. Histogram analysis (Fig. 9) can distinguish between samples having long storage time and samples having much less storage time and high fermentation rate. Three typical breadcrumb samples were collected having different pore areas like regular-sized pores, less number of pores having a larger area and more number of pores having a larger area and processed through the histogram analysis. A similarity has been observed in all three histograms.

Fig. 9

Histogram analysis on various breadcrumb samples: a bread sample having regular-sized pores; b bread sample having a lesser number of large area pores; c bread sample having a high number of large area pores; d Histogram corresponds to bread sample having regular-sized pores; e Histogram corresponds to bread sample having a lesser number of large area pores; f Histogram corresponds to bread sample having a high number of large area pores

Every histogram has been modeled in the form of a downward opening parabolic curve with a single peak. The histogram of breadcrumbs having regular area pores has no disturbance at its peak. As the large pore area increases in selected bread crumbs, a valley (disturbance) appears in the peak of the histogram (Fig. 9). Histogram analysis is thus able to discriminate between all three breadcrumb samples.

Results and discussion

An UVS system has been developed and tested on 500 commercial bread samples regarding various quality parameters such as crumb structure, freshness, etc. A robust vision sensing algorithm has been developed and tested for crumb structure parameter extraction such as cell density, area of the cells, brightness of the bread samples, etc.

Relative comparison of thresholding techniques based on error calculation

The freshness of the bread samples also has been extracted with the help of an ultrasonic technique using attenuation and propagation delay calculations. Traditional thresholding with various proposed thresholding techniques has been implemented for binary image conversion. The efficiency of the various implemented thresholding techniques (Table 4) has been evaluated based on the appropriate breadcrumb feature calculations. Because threshold selection can strongly affect breadcrumb feature values even with slight variations in the optimal threshold, the performance of the traditional thresholding methods was evaluated in comparison with an implemented proposed manual thresholding technique.

Table 4 Performance calculation of various implemented segmentation techniques based on error index calculation

The implemented manual thresholding method was found reliable for quantification of cell size uniformity and particularly for void fraction (Figs. 10, 11). The freshness index extracted using the ultrasonic technique was cross-verified using the color index calculation obtained using the developed vision sensing algorithm. It has been observed that the attenuation introduced in the acoustic signal for bread samples preserved in an open environment increases monotonically and decreases monotonically in a controlled or frozen environment.

Fig. 10

Discrimination of various tested breadcrumb samples compared to reference breadcrumbs based of cell count calculation

Fig. 11

Performance analysis calculation based on cell detection in the breadcrumb structures (R ² = 0.94) (1) Ostu’s thresholding (thresh = 0.5614) (2) isodata thresholding (thresh = 0.8134); (3) fuzzy based thresholding (thresh = 0.47843); (4) moment preserving thresholding method (thresh = 0.5342); (5) minimum error thresholding (thresh = 0.9961); (6) proposed manual threshold method

The reason for a significant decrease in an open environment might be due to weakening of the breadcrumbs means the stiffness of bread crumb decreases over the time period of preservation, so most of the energy of the transmitted ultrasonic signal gets absorbed by the breadcrumb and the amplitude of the received ultrasonic signal decreases while it is the reverse for the other two environments. It has been observed that cell count of the bread samples varies over the time period of preservation. Cell count of the fresh bread sample varies from 290 to 310 whereas stale bread sample cell count varies from 390 to 420 (Fig. 10). So bread sample freshness can be determined on the basis of cell count variation.

It has been observed that proposed thresholding method performs much better compared to traditional thresholding techniques. The error index has been calculated based on the detection of the number of cells in a given breadcrumb structure. The error Index has been calculated as follows:

$$ {\text{Error }} = {\text{ Number of cells }}\left( {\text{automatic}} \right) - {\text{Number of cells }}\left( {\text{manual}} \right) $$

(24)

Relative fermentation rate and freshness calculation based on the detection of cells and calculated ultrasonic parameters

It has been observed that relative fermentation quantification information can be extracted based on the calculated cell count variation with the time (days). A non-linear second order relationship (Fig. 12) has been observed between the fermentation rate and cell count of the breadcrumb structure. Freshness of the bread sample has also been quantified based on the calculated attenuation and propagation delay.

Fig. 12

Freshness variation with attenuation and propagation delay in an open environment

It has been observed that attenuation increases monotonically over the storage time period whereas propagation delay also increases over the time period, but the rate of change in the propagation delay is less compared to the variation in attenuation. A second order non-linear relationship has been observed between calculated attenuation, propagation delay and freshness of the bread sample. The non-linear relationships between freshness and propagation delay, attenuation are as follows:

$$ {\text{Freshness}} = - 0.47\,\, \times \,\,{\text{attenuation}}^{2} + 9.2\,\, \times \,{\text{attenuation}} - 38 $$

(25)

$$ {\text{Freshness}} = 1.9\,\, \times \,\,10^{ - 5} \,\, \times \,\,{\text{propagation delay}}^{2} + 0.0025\,\, \times \,\,{\text{propagation delay}} - 2.7 $$

(26)

Conclusions

The goal of this study, design and implementation is to investigate specific parameters related to the quality of various commercial bread samples. An ultrasonic machine vision sensing system has been developed and tested for various bread samples. The developed system can be an interesting method for the characterization of textures, freshness, and physical parameters in various baked food products. Detailed analysis on various bread samples regarding freshness and different quality parameters has been performed preserved in different environmental conditions such as open, controlled and frozen. It was discovered that the developed ultrasonic machine vision system resulted in higher correctness classifications, averaging 97 %, in the sense of prediction of various quality parameters and freshness detection. Therefore, this artificial technique and developed system has potential for use in routine inspection of bread and other bakery products It can also be useful in food industries and disease attack detection and prediction in various agriculture products [19]. It has been observed that amalgamation of vision and ultrasonic sensing techniques could be a useful tool to extract almost all important quality parameters (Table 5) of various bakery items.

Table 5 Comparative analysis of vision and ultrasonic sensing techniques based on their performance on extracted quality parameters