Keywords

1 Introduction

High throughput analysis has been an essential tool in cell biology research [1,2,3]. Conventional intensity based flow cytometer has been widely used in high throughput cell analysis [4]. In intensity based flow cytometer, cells are suspended and flow through laser illumination region at high speed, then fluorescence intensity and scattering intensity of large amount of cells are measured by photodetectors and stored for statistical analysis [5]. Since the invention of the first imaging flow cytometer (IFC) in 2000s [6], imaging capability are integrated with flow cytometer, and researchers are able to acquire images for each single cell using IFC. Thanks to the development of IFC, large amount of cell images can be recorded in a short time period for phenotype study that benefits drug discovery [7], immunology research [8], cancer research [9], etc. However, the data size of high throughput cell analysis also increased dramatically due to the imaging capability of IFC [10]. In conventional cytometer, only light intensity is recorded, and there are only a few parameters for each cell. However, in IFC based cell analysis, images are recorded for each cell. Furthermore, due to the multi-modal and multi-fluorescence-color capability of IFC, usually 3–4 images are recorded for each single cell in real applications. The transfer and storage of large cell image datasets remains to be the bottleneck of IFC cell analysis, and efficient data compression method needs to be developed.

Compressive sensing [11] is a technique for data compression. In compressive sensing process, sensing matrix are designed to project high dimensional data to low dimensional space. To realize successful data reconstruction, the original data needs to meet sparsity requirement. IFC datasets consist of large number of cell images, and each image is a single cell image. Due to the single cell image characteristics, either the cell image itself is sparse or the total variation of cell image is sparse, which makes compressive sensing suitable for IFC image compression [12]. The data compression can be realized using simple matrix multiplication, which makes the compression process time efficient. In addition, the compressive sensing algorithm is compatible with other existing image compression techniques, meaning that the cell images can be first compressed by other image compression method, then further compressed using compressive sensing method.

However, the image reconstruction process of compressive sensing is time consuming. Typically, reconstructing one image takes several seconds to several minutes [13]. There are usually hundreds of thousands of single cell images in a IFC dataset, which makes the time consuming reconstruction process unacceptable in real IFC applications. Another disadvantage of compressive sensing reconstruction is that predefined constraints such as image sparsity is required for successful image reconstruction. And designing the predefined constraints requires manual experience of cell property.

Convolution neural network [14] is a powerful technology for image reconstruction. Since convolution neural network is a forward generative model, the image reconstruction speed is much improved. Here, we combine compressive sensing and convolution neural network to propose a novel workflow for IFC cell image compression. The proposed workflow consists of two major steps. The first step is image compression. In the image compression step, random generated sensing matrix are generated for compressive sensing. The second step is image reconstruction. In the reconstruction step, we design a neural network to bypass the compressive sensing based image reconstruction. Using the neural network generative model, not only the image reconstruction speed is significantly improved, but also the requirement for predefined constraints is released. To demonstrate the proposed method, we compressed the Jurkat cell image using compressive sensing method. And for the image reconstruction step, we compared the proposed neural network based method to the compressive sensing method. From the experimental results, we can conclude that the proposed method is significantly faster, and reconstructed image quality is improved.

This paper will be arranged as follows. First, related works are introduced. Second, the proposed method is described in detail. Third, experiments and experimental results are discussed.

2 Related Works

2.1 Compressive Sensing

Compressed sensing, also known as compressed sampling or sparse sampling, is a technique for data compression based on the assumption that the original data is sparse in certain space [15]. As shown in Fig. 1, the concept of compressive sensing can be expressed as Eq. (1). Where s is the original data(signal/image) to be compressed, \(\Psi \) is the sparse basis matrix, and \(\Phi \) is the sensing matrix. Assuming \(\mathrm{s}\) is a one dimensional \(n\times 1\) matrix, then \(\Psi \) is a \(n\times n\) matrix. \(\Psi \) projects \(\mathrm{s}\) to a sparse basis using \(x=\mathrm{\Psi s}\). Then the sensing matrix \(\Phi \) is introduced for data compression. The size of \(\Phi \) is \(k\times n\), where \(n\gg k\). \(\Phi \) projects a higher dimensional matrix \(x\) onto a lower dimensional space, and \(y=\Phi x\) is the compressed data of length \(k\). Let \(A=\mathrm{\Phi \Psi }\), we can simplify Eq. (1) as \(y=\mathrm{As}\).

$$y=\mathrm{\Phi \Psi s}$$
(1)

In the reconstruction process, \(y\) and \(\mathrm{A}\) are known, the purpose of reconstruction is to solve \(\mathrm{s}\). However, as discussed above, \(n\gg k\), the problem is an ill-posed problem. Based on the sparsity assumption, a regularized term \(\gamma \left(s\right)\) is introduced. So, \(\mathrm{s}\) is reconstructed by solving Eq. (2), where \(\gamma \left(s\right)\) is the regularization term designed based on the sparsity assumption, and \(\lambda \) is the coefficient of the regularization term.

$$\widehat{s}={\mathit{argmin}\Vert y-As\Vert }_{2}^{2}+\gamma \left(s\right)$$
(2)

Compressive sensing solves the reconstruction problem by iteratively optimizing Eq. (2). There are several compressive sensing algorithms. L1-Magic [16] and TwIST [17] are representative algorithms, and we compare the performance of both algorithms to our proposed method. L1-Maigic provides high quality reconstruction but is time consuming. TwIST provides relatively high reconstruction speed but sacrifice image quality. As shown by our results, the proposed method outperforms L1-Magic and TwIST in both speed and reconstruction quality.

Fig. 1.
figure 1

Principle of compressed sensing

Full size image

2.2 Convolutional Neural Network

Convolution neural network [18] is a feedforward neural network, whose artificial neurons can respond to part of the surrounding elements within the coverage area and perform well in large-scale image processing. Through the cooperation of convolution, pooling, activation and other operations, the convolution neural network can better learn the features of spatial association. Convolution neural network can be used as a forward generative model for image generation. And in this paper, we designed a network for IFC compressive sensing image reconstruction to significantly improve the image reconstruction speed.

3 Image Compression Based on Compressive Sensing and CNN

The proposed method consists of two major steps. The first step is data compression step, and the second is image reconstruction steps. We will describe both steps in details in this session.

Fig. 2.
figure 2

(a) Data compression process. (b) Image reconstruction process

Full size image

3.1 Data Compression

The data compression step is shown in Fig. 2(a), where cell image is projected to a low dimensional space using compressive sensing method. Assume the image size is \(n\times n\). And \(k\) sensing matrix are generated for data compression, and in this paper, we use Gaussian random matrix as the sensing matrix since it satisfies the incoherence requirement. For each sensing matrix, we can calculate the correlation between image and the sensing matrix as shown by Eq. (3),

$$S\left(i\right)={\sum }_{x=1}^{N}{\sum }_{y=1}^{N}I\left(x,y\right){M}^{i}\left(x,y\right)$$
(3)

where \(I\) is the image, \({M}^{i}\) is the ith sensing matrix. And \(S\) is a \(k\) dimensional vector. So, when \(n\times n\gg k\), \(S\) is significantly compressed. In this paper, \(n=55\) and 2 different values of \(k\) are tested.

In this paper, we propose convolution neural network for image reconstruction. To compare performance, we also employed the conventional compressive sensing algorithm for image reconstruction. For compressive sensing, we use Total variation (TV) as the regularization term in Eq. (2). And the TV regularization term can be described as Eq. (4),

$$\psi \left(I\right)={\sum }_{x}{\sum }_{y}\sqrt{{D}_{x}{\left(I\right)}^{2}+{D}_{y}{\left(I\right)}^{2}}$$
(4)

where \(\psi \left(I\right)\) is the TV regularization term, \({D}_{x}\left(I\right)\) and \({D}_{y}\left(I\right)\) are gradients of \(I\) in \(x\) and \(y\) axis.

3.2 Image Reconstruction

The image reconstruction step is shown in Fig. 2(b). The purpose of image reconstruction step is to generate cell images using the compressed low dimensional data. We deigned a CNN structure convolution neural network as the generator. The designed network is a forward generative model which takes the compressed data as input and outputs the reconstructed cell image, thus significantly improves the reconstruction speed. Also, shown by our experimental results, the convolutional layers efficiently extract the data features and improves the reconstructed image quality. The designed convolution neural network structure is shown in Fig. 3, which consists of six convolutional layers, four maximum pooling layers and one full connection layer. The input dimension is determined by the compression ratio. In this paper, we test 2 different compression ratio 20% and 5%, corresponding to the input dimension as 606 and 151. The input signal is stretched to 4096 × 1 through a fully connected layer and then reshaped to 64 × 64. The features are extracted by the sub-sampling network, which is composed of six sub-conversion layers, which are composed of activation layer and convolution layer. The activation function uses ReLU function. The convolution layer is composed of 3 × 3 convolution blocks, whose step size is 1. We set four maximum pooling layers at the back of the first four convolution layers with a size of 2 × 2 and step size of 1. The pooling layer selects features and filters information from the output feature maps. Finally a 55 × 55 output image is obtained through the last convolutional layer. The generated model is trained in the way of supervised learning. The loss function is the absolute value of the difference between the generated image and the original image, and can be described as Eq. (5), where \(I\) is the groundtruth image and \(\widehat{I}\) is the predicted image. Once the generative model is trained, it takes the compressed data as input and directly outputs the reconstructed image.

$$Loss=\iint \left|I-\widehat{I}\right|dxdy$$
(5)
Fig. 3.
figure 3

CNN structure diagram

Full size image

4 Experimental Results

The flow chart of experiment is shown in Fig. 4. First, sensing matrix are generated and cell images are compressed. Following is the reconstruction part. In the reconstruction part, the compressed data is used as convolution neural network input to generate reconstructed image. First, the reconstruction model is trained using training set. Then we optimize the model until the best performance is achieved. The trained and optimized model is used for image reconstruction. Finally, the performance is evaluated. The details of the experiment and the final results will be explained in a later section.

Fig. 4.
figure 4

The experimental process

Full size image

4.1 Dataset

In our experiment, we used the Jurkat cell image dataset in the paper published by T Blasi et al. [19]. The dataset is generated using an imaging flow cytometer platform named ImageStreamX. Before image acquisition, Jurkat cells are stained with PI antibody and MPM2 antibody. Then the stained cells are flow through the imaging flow cytometer. Bright field, dark field and both fluorescence channels are imaged. For demonstration purpose, we choose the PI stained fluorescence imaging channel. These cell images are 55 by 55 pixels in size, with a pixel size of 0.33 um. Among them, 80% of the images in the data set are used as our training set and 20% of the samples are used as our test set.

4.2 Experimental Setup

Our compressed sensing image reconstruction based on supervised learning is completed using TensorFlow [20] on an NVIDIA GeForce RTX 3060 Ti GPU. We used Adam optimizer [21] to train our model, set the epoch to 150, and the batch size to 20, and the learning rate to be fixed at 5 × 10–5. As mentioned in the previous content, we will compare with TwIST algorithm and L1-Magic algorithm in the same data set and experimental environment to test the effect of our image reconstruction.

In order to test the reconstruction effect of our experiment, we tested and evaluated the supervised learning method, TwIST algorithm, L1-Magic algorithm, and the performance are compared. For TwIST and L1-Magic, we followed the same experiment environment as [16, 17].

4.3 Performance Evaluation Index

In order to evaluate the quality of image reconstruction, PSNR [22] and SSIM [23] are introduced. Given a clean image I and a noisy image K of size m × n, mean square error is defined as follows:

$$MSE=\frac{1}{mn}{\sum }_{x=0}^{m-1}{\sum }_{y=0}^{n-1}{\left[I\left(x,y\right)-\widehat{I}\left(x,y\right)\right]}^{2}$$
(6)

PSNR (Peak Signal-to-Noise Ratio) is defined as follows:

$$PSNR=10{\mathit{log}}_{10}\left(\frac{MA{X}_{I}^{2}}{MSE}\right)$$
(7)

where \(MA{X}_{I}^{2}\) is the maximum possible pixel value of the picture. If each pixel is represented by an 8-bit binary, 255.

SSIM (Structure Similarity) formula is based on three comparative measures between samples x and y, brightness, contrast and structure.

$$l\left(x,y\right)=\frac{2{\mu }_{x}{\mu }_{y}+{c}_{1}}{{\mu }_{x}^{2}+{\mu }_{y}^{2}+{c}_{1}}$$
(8)
$$c\left(x,y\right)=\frac{2{\sigma }_{x}{\sigma }_{y}+{c}_{2}}{{\sigma }_{x}^{2}+{\sigma }_{y}^{2}+{c}_{2}}$$
(9)
$$s\left(x,y\right)=\frac{{\sigma }_{xy}+{c}_{3}}{{\sigma }_{x}{\sigma }_{y}+{c}_{3}}$$
(10)

where x is the original image, y is the reconstructed image; \({\mu }_{x}\), \({\mu }_{y}\) is the mean value of x, y; \({\sigma }_{x}^{2}\), \({\sigma }_{y}^{2}\) is the variance of x, y; \({\sigma }_{xy}\) is the covariance of x, y; \({c}_{1}={\left({k}_{1}L\right)}^{2}\), \({c}_{2}={\left({k}_{2}L\right)}^{2}\), in general we take \({c}_{3}={c}_{2}/2\); \(L\) is the range of pixel values, we set \({k}_{1}\) = 0.01, \({k}_{2}\) = 0.02 as the default value.

SSIM formula is defined as follows:

$$SSIM\left(x,y\right)=l{\left(x,y\right)}^{\alpha }c{\left(x,y\right)}^{\beta }s{\left(x,y\right)}^{\gamma }$$
(11)

Set \(\alpha =\beta =\gamma =1\), and new SSIM formula is shown in Eq. (12):

$$SSIM\left(x,y\right)=\frac{\left(2{\mu }_{x}{\mu }_{y}+{c}_{1}\right)\left(2{\sigma }_{xy}+{c}_{2}\right)}{\left({\mu }_{x}^{2}+{\mu }_{y}^{2}+{c}_{1}\right)\left({\sigma }_{x}^{2}+{\sigma }_{y}^{2}+{c}_{2}\right)}$$
(12)

4.4 Image Reconstruction Result

In this section, we discuss the performance of image reconstruction. To quantitatively evaluate the reconstruction performance, three evaluation indexes are used, including PSNR, SSIM and image reconstruction speed. To compare performance, we also employ the TwIST compressive sensing algorithm and L1-Magic algorithm. First, we set the compression ratio to 20%. The example reconstructed images are shown in Fig. 5, the first column are the original images, the second column are the images reconstructed using our proposed method, the third column are the images reconstructed using L1-Magic method, the fourth column are the image reconstructed using the TwIST algorithm, and the fifth column is the compressed low dimensional data corresponding to the original images. As can be seen from the Fig. 5, the reconstruction performance of the proposed method and L1-Magic method is relatively good, while the images reconstructed using the TwIST algorithm have obviously worse quality. Then we compared the SSIM, PSNR and image reconstruction speed of the three methods, and the results are shown in Table 1, Our proposed method and L1-Magic method have better reconstruction quality, SSIM is 0.9932 and 0.9812, PSNR is 45.18 dB and 40.10 dB, respectively. As we can see, the reconstruction quality of TwIST algorithm is not as good as the two methods. SSIM and PSNR were only 0.8592 and 31.57 dB. From Table 1, we can conclude that our proposed method is much faster in reconstruction speed.

Fig. 5.
figure 5

Example reconstructed images (compression ratio = 20%)

Full size image
Table 1. Performance evaluation of image reconstruction (compression ratio = 20%)
Full size table

Then we reduced the compression ratio to 5%. When the compression ratio is 5%, the image reconstruction of TwIST algorithm is unsuccessful, so we will not show the reconstruction images. As can be seen from the Fig. 6, where the first column are the original images, the second column are the images reconstructed using our proposed method, the third column are the images reconstructed using L1-Magic method, and the fourth column are the compressed low dimensional data corresponding to the original images. The reconstruction quality of our proposed method is relatively good, while the images reconstructed using the L1-Magic algorithm are obviously worse. Then we compared the SSIM, PSNR and image reconstruction time of the two methods. And the results are shown in Table 2, Our proposed method has better reconstruction performance, SSIM is 0.9918, PSNR is 43.05 dB. In contrast, L1-Magic algorithm does not have such good performance. Its SSIM and PSNR are 0.9135 and 29.61 dB respectively. And the proposed method provides much faster image reconstruction. In conclusion, compared with the other methods, our proposed method not only ensures the image reconstruction quality, but also significantly improves the reconstruction speed, showing the superiority of our proposed method.

Fig. 6.
figure 6

Example reconstructed images (compression ratio = 5%)

Full size image
Table 2. Performance evaluation of image reconstruction (compression ratio = 5%)
Full size table

5 Conclusion

In this paper, we proposed a data compression method for imaging flow cytometer data combining compressive sensing and convolution neural network. In the proposed method, random generated sensing matrix are used for image compression and convolution neural network is used for image reconstruction. We use the Jurkat cell dataset to demonstrate our proposed method. The experimental results show that the proposed method is time efficient and provides good image reconstruction quality. We anticipate that the proposed method would improve the data compression process for massive imaging flow cytometer dataset and have broad application in cell biology research.