此語言尚未有已發佈之文章
文章發佈後將於此處顯示。
top of page
csx-home>vol-01>issue-04>A method for automated regression test in scientific computing libraries: illustration with SPHinXsys
PTL PREMIUM
Submitted 10 March 2023
Revised 12 April 2023
Accepted 23 April 2023
A method for automated regression test in scientific computing libraries: illustration with SPHinXsys
Bo Zhang
Chi Zhang
Xiangyu Hu
TUM School of Engineering and Design, Technical University of Munich,
85748 Garching, Germany
Canadian Science Letters X
2023 ° 13(05) ° 01-04
https://www.wikipt.org/csx-home
DOI: 10.1490/69802.105csx
Acknowledgments
The first author would like to acknowledge the financial support provided by the China Scholarship Council (No.202006230071). C. Zhang and
Abstract
Scientific computing libraries, either in-house or open-source, have experienced enormous progress in both engineering and scientific research. It is therefore essential to ensure that the modifications in the source code aroused by bug fixing or new feature development wouldn’t compromise the accuracy and functionality that has already been validated and verified. With this in mind, this paper introduces a method for developing and implementing an automatic regression test environment and takes the open-source multiphysics library SPHinXsys [1] as an example. Firstly, the reference database for each benchmark test is generated from monitored data by multiple executions. This database contains the maximum variation range of metrics for different types of strategies, i.e., time-averaged method, ensemble-averaged method as well as the dynamic time warping method, covering the uncertainty arising from parallel computing, particle relaxation, physical instabilities, etc. Then, new results obtained after source code modification will be tested with them according to a curve-similarity-based comparison. Whenever the source code is updated, the regression test will be carried out automatically for all test cases and used to report the validity of the current results. This regression test environment has already been implemented in all dynamics test cases released in SPHinXsys, including fluid dynamics, solid mechanics, fluid-structure interaction, thermal and mass diffusion, reaction-diffusion, and their multi-physics coupling, and shows good capability for testing various problems. It’s worth noting that while the present test environment is built and implemented for a specific scientific computing library, its underlying principle is generic and can be applied to many others. Keywords: Scientific computing, Open-source library, Verification and validation, Regression test, Automatic test environment, Curve similarity comparison, Smoothed particle hydrodynamics.
Introduction
The development of computers has pushed scientific computing to become an indispensable part of many technologies and industries, such as in assessing climate change [2], designing new energy conductors [3], and imposes an ever-widening effect on better predicting and understanding the phenomena of nature and engineered systems. Following the definition of validation and verification of scientific computing claimed by William [4], scientific computing should always represent the real world and the conceptual model accurately. However, this is a challenging task due to the complex mathematical models and calculations, which often require changes to separate parts of the application and definitely increase the possibility of errors. Moreover, the development of scientific applications is a lasting work, and changes occur frequently due to different requirements and new features introduced. The applications should always produce trustworthy results and assure the application qualities with the help of validation and verification conducted alongside both application development and usage [5]. Implementing testing, including unit test, integration test, regression test, system test, etc., can provide concrete validation and verification procedures. Notwithstanding the fact that it has already been adopted in many IT software, scientific applications, especially some open-source libraries, find it difficult to perform those testing directly with traditional techniques. One main challenge is due to the characteristics of the scientific application, as it’s hard to find a test oracle to check if the program can gain the expected output when executing test cases [6, 7]. Another challenge arises from cultural differences between scientists and the software engineering community [6]. Many scientific applications are usually developed by small-group scientists, who may not be very familiar with accepted software engineering practices, and haven’t studied the developing process of their software in much detail [8], and therefore may overlook the impact of changes. Regression test stands a great chance to ensure the output validity of scientific applications under development. It is a re-testing activity, which refers to executing the test suite with given inputs and comparing the output with previously stored reference results when modifications occur or new features are added. In this way, developers can make sure that their changes don’t cause any unexpected side effects and previous functionalities are still verified [7]. Since taking the regression test for all test cases is generally
Conclusion
This paper introduces a method for developing an automatic regression test environment for open-source scientific libraries and uses SPHinXsys as an illustration to demonstrate its functionality. For scientific libraries under centralized development, it’s essential to guarantee the accuracy of simulation results all the time, and the regression test provides this procedure. The reference database for each benchmark test is generated using different strategies, and the new result after code modifications can be automatically tested with them once the source code is updated. This regression test environment has been implemented in all test cases released in SPHinXsys, and it shows great functionality to check the validity of the new result obtained after code modifications. By doing such work, we also want to draw some attention from general scientific computing communities to emphasize the software performance during development. The principle of the regression test proposed here is universal and can be applied and extended in other libraries and applications. In the future, other regression test methods will be implemented, and with the number of test cases swelling due to adding new dynamics features, selection and reduction of test cases will also be adopted.
References
[1] C. Zhang, M. Rezavand, Y. Zhu, Y. Yu, D. Wu, W. Zhang, J. Wang, X. Hu, Sphinxsys: An open-source multi-physics and multi-resolution library based on smoothed particle hydrodynamics, Computer Physics Communications 267 (2021) 108066.
[2] J. B. Drake, P. W. Jones, G. R. Carr Jr, Overview of the software design of the community climate system model, The International Journal of High Performance Computing Applications 19 (3) (2005) 177–186.
[3] D. E. Post, R. P. Kendall, Software project management and quality engineering practices for complex, coupled multiphysics, massively parallel computational simulations: Lessons learned from asci, The International Journal of High Performance Computing Applications 18 (4) (2004) 399–416.
[4] W. L. Oberkampf, C. J. Roy, Verification and validation in scientific computing., Cambridge: Cambridge University Press, 2010.
[5] P. Farrell, M. Piggott, G. Gorman, D. Ham, C. Wilson, T. Bond, Automated continuous verification for numerical simulation, Geoscientific Model Development 4 (2) (2011) 435–449. 32
[6] U. Kanewala, J. M. Bieman, Testing scientific software: A systematic literature review, Information and software technology 56 (10) (2014) 1219–1232.
[7] H. Remmel, B. Paech, P. Bastian, C. Engwer, System testing a scientific framework using a regression-test environment, Computing in Science & Engineering 14 (2) (2011) 38–45.
[8] K. Bojana, Framework for developing scientific applications, Ph.D. thesis, UNIVERSITY SS CYRIL AND METHODIUS (2018).
[9] H. Ural, H. Yenig¨un, Regression test suite selection using dependence analysis, Journal of Software: Evolution and Process 25 (7) (2013) 681– 709.
[10] A. P. Agrawal, A. Choudhary, A. Kaur, An effective regression test case selection using hybrid whale optimization algorithm, International Journal of Distributed Systems and Technologies (IJDST) 11 (1) (2020) 53–67.
[11] D. Di Nardo, N. Alshahwan, L. Briand, Y. Labiche, Coverage-based regression test case selection, minimization and prioritization: A case study on an industrial system, Software Testing, Verification and Reliability 25 (4) (2015) 371–396.
[12] S. Prasad, M. Jain, S. Singh, C. Patvardhan, Regression optimizer a multi coverage criteria test suite minimization technique, International Journal of Applied Information Systems (IJAIS) 1 (8). 33
[13] S. Harikarthik, V. Palanisamy, P. Ramanathan, Optimal test suite selection in regression testing with testcase prioritization using modified ann and whale optimization algorithm, Cluster Computing 22 (5) (2019) 11425–11434.
[14] L. Tahat, B. Korel, M. Harman, H. Ural, Regression test suite prioritization using system models, Software Testing, Verification and Reliability 22 (7) (2012) 481–506.
[15] X. Lin, M. Simon, N. Niu, Exploratory metamorphic testing for scientific software, Computing in science & engineering 22 (2) (2018) 78–87.
[16] X. Lin, M. Simon, N. Niu, Hierarchical metamorphic relations for testing scientific software, in: 2018 IEEE/ACM 13th International Workshop on Software Engineering for Science (SE4Science), IEEE, 2018, pp. 1–8.
[17] Z. Peng, X. Lin, N. Niu, Unit tests of scientific software: A study on swmm, in: International Conference on Computational Science, Springer, 2020, pp. 413–427.
[18] Z. Peng, X. Lin, M. Simon, N. Niu, Unit and regression tests of scientific software: A study on swmm, Journal of Computational Science 53 (2021) 101347.
[19] Z. Liu, K. Brinckman, S. Dash, Automated validation of cfd codes for analysis of scramjet propulsive flows using crave, in: 45th AIAA/ASME/SAE/ASEE Joint Propulsion Conference & Exhibit, 2009, p. 4846. 34
[20] H. J. Happ, Linux c-shell regression testing for the shamrc cfd code, in: Users’˜ Group Conference, 2011, p. 41.
[21] Kitware Inc. et al., CMake. https://cmake.org/documentation/, accessed on 7thJuly 2021.
[22] R. Baxter, Software engineering is software engineering, in: 26th International Conference on Software Engineering, W36 Workshop Software Engineering for High Performance System (HPCS) Applications, IET, 2004, pp. 4–18.
[23] L. B. Lucy, A numerical approach to the testing of the fission hypothesis, The astronomical journal 82 (1977) 1013–1024.
[24] R. A. Gingold, J. J. Monaghan, Smoothed particle hydrodynamics: theory and application to non-spherical stars, Monthly notices of the royal astronomical society 181 (3) (1977) 375–389.
[25] C. Zhang, X. Hu, N. A. Adams, A weakly compressible sph method based on a low-dissipation riemann solver, Journal of Computational Physics 335 (2017) 605–620.
[26] C. Zhang, M. Rezavand, X. Hu, Dual-criteria time stepping for weakly compressible smoothed particle hydrodynamics, Journal of Computational Physics 404 (2020) 109135.
[27] C. Zhang, Y. Zhu, Y. Yu, M. Rezavand, X. Hu, A simple artificial damping method for total lagrangian smoothed particle hydrodynamics, arXiv preprint arXiv:2102.04898. 35
[28] C. Zhang, M. Rezavand, X. Hu, A multi-resolution sph method for fluidstructure interactions, Journal of Computational Physics 429 (2021) 110028.
[29] C. Zhang, J. Wang, M. Rezavand, D. Wu, X. Hu, An integrative smoothed particle hydrodynamics method for modeling cardiac function, Computer Methods in Applied Mechanics and Engineering 381 (2021) 113847.
[30] G. Bilotta, A. H´erault, A. Cappello, G. Ganci, C. Del Negro, Gpusph: a smoothed particle hydrodynamics model for the thermal and rheological evolution of lava flows, Geological Society, London, Special Publications 426 (1) (2016) 387–408.
[31] M. Gomez-Gesteira, B. Rogers, A. Crespo, R. Dalrymple, M. Narayanaswamy, J. Dominguez, Sphysics – development of a free-surface fluid solver – part 1: Theory and formulations, Computers and Geosciences 48 (2012) 289–299.
[32] A. J. Crespo, J. M. Dom´ınguez, B. D. Rogers, M. G´omez-Gesteira, S. Longshaw, R. Canelas, R. Vacondio, A. Barreiro, O. Garc´ıa-Feal, Dualsphysics: Open-source parallel cfd solver based on smoothed particle hydrodynamics (sph), Computer Physics Communications 187 (2015) 204–216.
[33] J. L. Cercos-Pita, Aquagpusph, a new free 3d sph solver accelerated with opencl, Computer Physics Communications 192 (2015) 295–312. 36
[34] V. Springel, The cosmological simulation code gadget-2, Monthly notices of the royal astronomical society 364 (4) (2005) 1105–1134.
[35] P. F. Hopkins, A new public release of the gizmo code, arXiv preprint arXiv:1712.01294.
[36] K. Pohl, G. B¨ockle, F. Van Der Linden, Software product line engineering: foundations, principles, and techniques, Vol. 1, Springer, 2005.
[37] C. Zhang, X. Y. Hu, N. A. Adams, A generalized transport-velocity formulation for smoothed particle hydrodynamics, Journal of Computational Physics 337 (2017) 216–232.
[38] C. Zhang, Y. Wei, F. Dias, X. Hu, An efficient fully lagrangian solver for modeling wave interaction with oscillating wave surge converter, Ocean Engineering 236 (2021) 109540.
[39] H. Sakoe, S. Chiba, Dynamic programming algorithm optimization for spoken word recognition, IEEE transactions on acoustics, speech, and signal processing 26 (1) (1978) 43–49.
[40] C. Myers, L. Rabiner, A. Rosenberg, Performance tradeoffs in dynamic time warping algorithms for isolated word recognition, IEEE Transactions on Acoustics, Speech, and Signal Processing 28 (6) (1980) 623–635.
[41] J. Cheng, F. Wei, Y. Liu, C. Li, Q. Chen, X. Chen, Chinese sign language recognition based on dtw-distance-mapping features, Mathematical Problems in Engineering 2020.
bottom of page