Abstract
Digital rock physics (DRP) is a rapidly evolving technology targeting fast turnaround times for repeatable core analysis and multi-physics simulation of rock properties. We develop and validate a rapid and scalable distributed-parallel single-phase pore-scale flow simulator for permeability estimation on real 3D pore-scale micro-CT images using a novel variant of the lattice Boltzmann method (LBM). The LBM code implementation is designed to take maximum advantage of distributed computing on multiple general-purpose graphics processing units (GPGPUs). We describe and extensively test the distributed parallel implementation of an innovative LBM algorithm for simulating flow in pore-scale media based on the multiple-relaxation-time (MRT) model that utilizes a precise treatment of body force. While the individual components of the resulting simulator can be separately found in various references, our novel contributions are (1) the integration of all of the mathematical and high-performance computing components together with a highly optimized code implementation and (2) the delivery of quantitative results with the simulator in terms of robustness, accuracy, and computational efficiency for a variety of flow geometries including various types of real rock images. We report on extensive validations of the simulator in terms of accuracy and provide near-ideal distributed parallel scalability results on large pore-scale image volumes that were largely computationally inaccessible prior to our implementation. We validate the accuracy of the MRT-LBM simulator on model geometries with analytical solutions. Permeability estimation results are then provided on large 3D binary microstructures including a sphere pack and rocks from various sandstone and carbonate formations. We quantify the scalability behavior of the distributed parallel implementation of MRT-LBM as a function of model type/size and the number of utilized GPGPUs for a panoply of permeability estimation problems.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
Adler, P.M., Jacquin, C.G., Quiblier, J.A.: Flow in simulated porous media. Intern. J. Multiphase Flow 16, 691–712 (1990)
Andrä, H., Combaret, N., Dvorkin, J., Glatt, E., Han, J., Kabel, M., Keehm, Y., Krzikalla, F., Lee, M., Madonna, C., Marsh, M., Mukerji, T., Saenger, E.H., Sain, R., Saxena, N., Ricker, S., Wiegmann, A., Zhan, X.: Digital rock physics benchmarks—part I: imaging and segmentation. Comput. Geosci. 50, 25–32 (2013)
Andrä, H., Combaret, N., Dvorkin, J., Glatt, E., Han, J., Kabel, M., Keehm, Y., Krzikalla, F., Lee, M., Madonna, C., Marsh, M., Mukerji, T., Saenger, E.H., Sain, R., Saxena, N., Ricker, S., Wiegmann, A., Zhan, X.: Digital rock physics benchmarks—part II: computing effective properties. Comput. Geosci. 50, 33–43 (2013)
ANSYS Inc: ANSYS FLUENT Theory Guide (2010)
Arns, C.H., Bauget, F., Limaye, A., Sakellariou, A., Senden, T., Sheppard, A., Sok, R.M., Pinczewski, V., Bakke, S., Berge, L.I., Oren, P., Knackstedt, M.: Pore-scale characterization of carbonates using x-ray microtomography. SPE J. 10, 26–29 (2005)
Arns, C.H., Knackstedt, M.A., Pinczewski, W.V., Garboczi, E.J.: Computation of linear elastic properties from microtomographic images: methodology and agreement between theory and experiment. Geophysics 67, 1396–1405 (2002)
Bear, J.: Dynamics of fluids in porous media. American Elsevier Publishing Company Inc., New York (1972)
Benzi, R., Succi, S., Vergassola, M.: The lattice Boltzmann-equation—theory and applications. Phys. Rep. 222, 145–197 (1992)
Berg, S., Ott, H., Klapp, S.A., Schwing, A., Neiteler, R., Brussee, N., Makurat, A., Leu, L., Enzmann, F., Schwarz, J. -O., Kersten, M., Irvine, S., Stampanoni, M.: Real-time 3D imaging of Haines jumps in porous media flow. Proc. Nat. Acad. Sci. (PNAS) 110, 3755–3759 (2013)
Berg, S., Rücker, M., Ott, H., Georgiadis, A., van der Linde, H., Enzmann, F., Kersten, M., Armstrong, R.T., de With, S., Becker, J., Wiegmann, A.: Connected pathway relative permeability from pore-scale imaging of imbibition. Adv. Water Res. 90, 24–35 (2016)
Bernaschi, M., Fatica, M., Melchionna, S., Succi, S., Kaxiras, E.: A flexible high-performance lattice Boltzmann GPU code for the simulations of fluid flows in complex geometries. Concurr. Comput.: Prac. Exp. 22, 1–14 (2010)
Berryman, J.G., Blair, S.C.: Use of digital image analysis to estimate fluid permeability of porous materials: application of two-point correlation functions. J. Appl. Phys. 60, 1930–1938 (1986)
Bhatnagar, P.L., Gross, E.P., Krook, M.: A model for collision processes in gases. I. Small amplitude processes in charged and neutral one-component systems. Phys. Rev. 94, 511–525 (1954)
Bisson, M., Bernaschi, M., Melchionna, S., Succi, S., Kaxiras, E.: Multiscale hemodynamics using GPU clusters. Commun. Comput. Phys. 11, 48–64 (2012)
Blunt, M.J., Bijeljic, B., Dong, H., Gharbi, O., Iglauer, S., Mostaghimi, P., Paluszny, A., Pentland, C.: Pore-scale imaging and modelling. Adv. Water Res. 51, 197–216 (2013)
Bouzidi, M.H., Firdaouss, M., Lallemand, P.: Momentum transfer of a Boltzmann-lattice fluid with boundaries. Phys. Fluids 13, 3452–3459 (2001)
Cancelliere, A., Chang, C., Foti, E., Rothman, D.H., Succi, S.: The permeability of a random medium: comparison of simulation with theory. Phys. Fluids A 2, 2085–2088 (1990)
Chen, H., Teixeira, C., Molvig, K.: Digital physics approach to computational fluid dynamics: some basic theoretical features. Intern. J. Modern Phys. C 8, 675–684 (1997)
Chen, C., Wang, Z., Majeti, D., Vrvilo, N., Warburton, T., Sarkar, V., Li, G.: Optimization of lattice Boltzmann simulation with graphics-processing-unit parallel computing and the application in reservoir characterization. SPE J. 21(3), 1425–1435 (2016)
COMSOL Multiphysics Documentation: COMSOL Multiphysics, version 5.3, Pore-scale flow. https://www.comsol.com/model/download/407771/models.ssf.porescaleflow.pdf. Accessed: 4 June 2017 (2017)
d’Humières, D., Ginzburg, I., Krafczyk, M., Lallemand, P., Luo, L.: Multiple-relaxation-time lattice Boltzmann models in three dimensions. Philos. Trans. Royal Soc. A 360, 437–451 (2002)
Dvorkin, J., Armbruster, M., Baldwin, C., Fang, Q., Derzhi, N., Gomez, C., Nur, B., Nur, A., Mu, Y.: The future of rock physics: computational methods vs. lab testing. First Break 26, 63–68 (2008)
Dvorkin, J., Derzhi, N., Diaz, E., Fang, Q.: Relevance of computational rock physics. Geophysics 76, E141–E153 (2011)
Ferrèol, B., Rothman, D.H.: Lattice-Boltzmann simulations of flow through Fontainebleau sandstone. Transp. Porous Media 20, 3–20 (1995)
Ginzburg, I., d’Humières, D.: Multireflection boundary conditions for lattice Boltzmann models. Phys. Rev. E 68, 066614 (2003)
Gray, F., Boek, E.: Enhancing computational precision for lattice Boltzmann schemes in porous media flows. Computation 4, 11 (2016)
Guo, Z., Zheng, C.: Analysis of lattice Boltzmann equation for microscale gas flows: relaxation times, boundary conditions and the Knudsen layer. Intern. J. Comput. Fluid Dyn. 22, 465–473 (2008)
Guo, Z., Zheng, C., Shi, B.: Discrete lattice effects on the forcing term in the lattice Boltzmann method. Phys. Rev. E 65, 046308 (2002)
Hassanizadeh, S.M., Gray, W.: High velocity flow in porous media. Transp. Porous Media 2, 521–531 (1987)
He, X., Luo, L.S.: Theory of the lattice Boltzmann method: from the Boltzmann equation to the lattice Boltzmann equation. Phys. Rev. E 56, 6811–6817 (1997)
He, X., Zou, Q., Luo, L., Dembo, M.: Analytic solutions of simple flows and analysis of nonslip boundary conditions for the lattice Boltzmann BGK model. J. Stat. Phys. 87, 115–136 (1997)
Hinch, E.J.: An averaged-equation approach to particle interactions in a fluid suspension. J. Fluid Mech. 83, 695–720 (1977)
Howells, I.D.: Drag due to the motion of a Newtonian fluid through a sparse random array of small fixed rigid objects. J. Fluid Mech. 64, 449–476 (1974)
Keehm, Y.: Computational rock physics: Transport properties in porous media and applications. Ph.D. Dissertation, Stanford University, Stanford (2003)
Keehm, Y., Mukerji, T., Nur, A.: Permeability prediction from thin sections: 3D reconstruction and Lattice-Boltzmann flow simulation. Geophys. Res. Lett. 31, L04606 (2004)
Kendon, V.M., Cates, M.E., Desplat, J. -C., Pagonabarraga, I., Bladon, P.: Inertial effects in three-dimensional spinodal decomposition of a symmetric binary fluid mixture: a lattice Boltzmann study. J. Fluid Mech. 440, 147–203 (2000)
Knackstedt, M.A., Sheppard, A.P., Sahimi, M.: Pore network modelling of two-phase flow in porous rock: the effect of correlated heterogeneity. Adv. Water Res. 24, 257–277 (2001)
Knackstedt, M.A., Arns, C., Madadi, M., Sheppard, A.P., Latham, S., Sok, R., Bächle, G., Eberli, G.: Elastic and flow properties of carbonate core derived from 3D X ray-CT images. SEG Tech. Prog. Expanded Abstr. 27, 1804–1809 (2008)
Knackstedt, M.A., Latham, S., Madadi, M., Sheppard, A., Varslot, T., Arns, C.: Digital rock physics: 3D imaging of core material and correlations to acoustic and flow properties. Lead. Edge 28, 28–33 (2009)
Kuzmin, A., Guo, Z., Mohamad, A.: Simultaneous incorporation of mass and force terms in the multi-relaxation-time framework for lattice Boltzmann schemes. Philos. Transac. Royal Soc. A Math. Phys. Eng. Sci. 369, 2219–2227 (2011)
Ladd, A.J.C.: Numerical simulations of particulate suspensions via a discretized Boltzmann equation: part 1: theoretical foundation. J. Fluid Mech. 271, 285–309 (1994)
Ladd, A.J.C.: Numerical simulations of particulate suspensions via a discretized Boltzmann equation: part 2. Numerical results. J. Fluid Mech. 271, 311–339 (1994)
Lallemand, P., Luo, L.S.: Theory of the lattice Boltzmann method: dispersion, dissipation, isotropy, Galilean invariance, and stability. Phys. Rev. E 61, 6546–6562 (2000)
Leu, L., Berg, S., Enzmann, F., Armstrong, R.T., Kersten, M.: Fast x-ray micro-tomography of multiphase flow in Berea Sandstone: a sensitivity study on image processing. Transp. Porous Media 105, 451–469 (2014)
Llwellin, E.W.: LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part I: theory and implementation. Comput. Geosci. 36, 115–122 (2010)
Llwellin, E.W.: LBflow: An extensible lattice Boltzmann framework for the simulation of geophysical flows. Part II: usage and validation. Comput. Geosci. 36, 123–132 (2010)
Martys, N.S., Chen, H.: Simulation of multicomponent fluids in complex three dimensional geometries by the lattice Boltzmann method. Phys. Rev. E 53, 743–750 (1996)
Mavko, G., Mukerji, T., Dvorkin, J.: The rock physics handbook: Tools for seismic analysis of porous media, 2nd edn. Cambridge University Press, Cambridge (2009)
Meakin, P., Tartakovsky, A.M.: Modeling and simulation of pore-scale multiphase fluid flow and reactive transport in fractured and porous media. Rev. Geophys. RG3002, 1–47 (2009)
Otomo, H., Fan, H., Li, Y., Dressler, M., Staroselsky, I., Zhang, R., Chen, H.: Studies of accurate multi-component lattice Boltzmann models on benchmark cases required for engineering applications. J. Comput. Sci. 17, 334–339 (2016)
Øren, P., Bakke, S., Rueslåtten, H.: Digital core laboratory: rock and flow properties derived from computer generated rocks. In: International Symposium of the Society of Core Analysts held in Trondheim, pp. 1–12, Norway (2006)
Palabos Documentation. http://www.palabos.org/documentation/userguide/. Accessed: 4 June 2017 (2017)
Pan, C., Luo, L.-S., Miller, C.T.: An evaluation of lattice Boltzmann schemes for porous medium flow simulation. Comput. Fluids 35, 898–909 (2006)
Piller, M., Casagrande, D., Schena, G., Santini, M.: Pore-scale simulation of laminar flow through porous media. J. Phys.: Conf. Ser. 501, 012010 (2014)
Premnath, K.N., Abraham, J.: Three-dimensional multi-relaxation-time (MRT) lattice-Boltzmann models for multiphase flow. J. Comput. Phys. 224, 539–559 (2007)
Premnath, K.N., Pattison, M.J., Banerjee, S.: An investigation of the Lattice Boltzmann method for large eddy simulation of complex turbulent separated flow. J. Fluids Eng. 135, 051401–1-12 (2013)
Rumpf, H.C.H., Gupte, A.R.: Einflüsse der Porosität und Korngröarticleßenverteilung im Widerstandsgesetz der Porenströmung. Chem. Ing. Tech. 43, 367–375 (1971)
Saenger, E.H., Enzmann, F., Keehm, Y., Steeb, H.: Digital rock physics: effect of fluid viscosity on effective elastic properties. J. Appl. Geophys. 74, 236–241 (2011)
Sain, R., Mukerji, T., Mavko, G.: How computational rock-physics tools can be used to simulate geologic processes, understand pore-scale heterogeneity, and refine theoretical models. Lead. Edge 33, 324–334 (2014)
Saxena, N., Mavko, G.: Estimating elastic moduli of rocks from thin sections: digital rock study of 3D properties from 2D images. Comput. Geosci. 88, 9–21 (2016)
Saxena, N., Hofmann, R., Alpak, F.O., Berg, S., Dietderich, J., Agarwal, U., Tandon, K., Hunter, S., Freeman, J., Wilson, O.B.: References and benchmarks for pore-scale flow simulated using micro-CT images of porous media and digital rocks. Adv. Water Res. 109, 211–235 (2017)
Saxena, N., Hofmann, R., Alpak, F.O., Dietderich, J., Hunter, S., Day-Stirrat, R.: Effect of image segmentation andamp; voxel size on micro-CT computed effective transport andamp; elastic properties. Mar. Petrol. Geol. 86, 972–990 (2017)
Saxena, N., Hofmann, R., Hows, A., Alpak, F.O., Freeman, J., Hunter, S., Appel, M.: Imaging and computational considerations for digital rock physics. Advances in Water Resources in review (2018)
Saxena, N., Mavko, G., Hofmann, R., Srisutthiyakorn, N.: Estimating permeability from thin sections without reconstruction: graphical study of 3D properties from 2D images. Comput. Geosci. 102, 79–99 (2017)
Skordos, P.A.: Initial and boundary conditions for the lattice Boltzmann method. Phys. Rev. 48, 4823–4842 (1993)
Shah, S.M., Gray, F., Crawshaw, J.P., Boek, E.S.: Micro-computed tomography pore-scale study of flow in porous media: effect of voxel resolution. Adv. Water Res. 95, 276–287 (2016)
Sisavath, S., Jing, X., Pain, C.C., Zimmerman, R.W.: Creeping flow through an axisymmetric sudden contraction or expansion. J. Fluids Eng. 124, 273–278 (2002)
Sisavath, S., Jing, X., Zimmerman, R.W.: Creeping flow through a pipe of varying radius. Phys. Fluids 13, 2762–2772 (2001)
Sochi, T.: Newtonian flow in converging-diverging capillaries. Intern. J. Model. Simul. Scientific Comput. 4, 1350011 (2013)
Succi, S.: The lattice boltzmann equation for fluid dynamics and beyond. numerical mathematics and scientific computation series. Clarendon Press, Oxford (2001)
Sukop, M.C., Thorne, D.T.: Lattice Boltzmann Modeling: An Introduction for Geoscientists and Engineers. Springer-Verlag, Berlin (2006)
Tölke, J.: Implementation of a lattice Boltzmann kernel using the compute unified device architecture developed by nVIDIA. Comput. Visual. Sci. 13(1), 29–39 (2010)
Tölke, J., Baldwin, C., Mu, Y., Derzhi, N., Fang, Q., Grader, A., Dvorkin, J.: Computer simulations of fluid flow in sediment: from images to permeability. Lead. Edge 29, 68–74 (2010)
Torquato, S: Random Heterogeneous Materials: Microstructure and Macroscopic Properties. Springer Science & Business Media, New York (2002)
Wiegmann, A.: Computation of the permeability of porous materials from their microstructure by FFF-Stokes. Berichte des Fraunhofer ITWM, Nr. 129. Kaiserslautern, Germany (2007)
Zhang, D., Zhang, R., Chen, S., Soll, W.E.: Pore scale study of flow in porous media: scale dependency, REV, and statistical REV. Geophys. Res. Lett. 27, 1195–1198 (2000)
Acknowledgments
We thank Shell International Exploration and Production Inc. for permission to publish this paper. Farrel Gray’s doctoral studentship was provided by the Qatar Carbonates and Carbon Storage Research Centre jointly funded by Qatar Petroleum, Shell and the Qatar Science and Technology Park.
Author information
Authors and Affiliations
Corresponding author
Rights and permissions
About this article
Cite this article
Alpak, F.O., Gray, F., Saxena, N. et al. A distributed parallel multiple-relaxation-time lattice Boltzmann method on general-purpose graphics processing units for the rapid and scalable computation of absolute permeability from high-resolution 3D micro-CT images. Comput Geosci 22, 815–832 (2018). https://doi.org/10.1007/s10596-018-9727-7
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s10596-018-9727-7