Feel++ is an open-source software project designed to create improved methods for solving partial differential equations on high performance computers. Professor Christophe Prud’homme of the University of Strasbourg has been leading the project and has been using a PRACE allocation to test to out some of their new methods.
A partial differential equation is a relation between a function of several variables and its (partial) derivatives. It is used in physics, mathematics and engineering to model most natural phenomena such as propagation of sound, electrodynamics, fluid flow, elasticity or more complex problems.
The Feel++ project has been developing an open-source library that provides a large range of numerical methods for solving partial differential equations. Led by Professor Christophe Prud’homme of the University of Strasbourg, it has been a large collaborative effort, working alongside other mathematicians as well as engineers, doctors, industry experts and others to develop scalable solutions for HPC and apply them to applications that are useful for researchers.
For instance, one of the main collaborations has been with the Laboratoire des Champs Magnétiques Intenses (LNCMI), part of the European Magnetic Field Lab, which is dedicated to generating extremely powerful magnetic fields that can be used for scientific research and making them available to the scientific community. “Many winners of the Nobel Prize have worked with LNCMI on the way to their accolades,” says Prud’homme. “We have helped LNCMI by providing them with the software to help them design their magnets.”
The team has been using their PRACE allocation to test out the methods they are developing on Tier-0 computers. “The design of our library allows us to quickly put our methods into applications,” Prud’homme explains. “There are a few which are still in development and will require further adaptations and improvements, but some researchers are already using them.”
The library has two types of interface. One of them is designed for mathematicians, using C++ to describe things in a way that is very close to the mathematical language of the methodologies. This is useful for those developing the methodologies and for deploying applications on to supercomputers. The other type of interface uses multi-physics toolboxes, providing a more user-friendly, application oriented way for researchers from a wide array of disciplines to access and work with the library.
The Eye2Brain system requires mathematical models parameterised with patient- specific data that can both quantitatively describe the fluid-dynamical and metabolic connections between the eye and the brain, and identify the main factors that influence these connections.
Over the course of the PRACE allocation, the Feel++ project has developed two different kinds of method. The first of these is based on domain decomposition and is called a mortar method. “For this one we can reach extremely good scalability and excellent effectiveness, meaning that there is both minimal communication between processing elements as well as good performance when larger numbers of cores are used,” says Prud’homme. The design of this numerical method and the results of scaling it has been published in M2AN.
The other type of method developed by Feel++ is a so-called “fictitious domain” method. One real-life application that this has been used to explore has been tracking the rheology and movement of red blood cells in blood flows. There are already a number of ways of doing this, but this particular method disconnects the discrete geometrical model of the fluid from the geometrical model of the cells. From a mathematical point of view, the method developed by Feel++ has been shown to work well and is able to simulate extremely large numbers of cells at once to provide a realistic picture.
Prud’homme and his colleagues used part of their allocation to compare their fictitious domain method with other methods such as level-set methods, and showed that there are advantages and disadvantages for both. “It’s not quite clear what the best compromise here is yet, but what we did show is that scaling the fictitious domain method is quite difficult compared to our mortar method,” he says. “However, this was expected because that method was designed from the ground up to scale well, whereas this method was more a case of trying to handle a specific physical problem in a flexible way.”
The mortar method has now been integrated into the Feel++ framework and is being used to help the team tackle large-scale problems. This has required a number of design changes and adaptations, but the rewards have been significant in terms of speedup, effectivity and scaling.
The PRACE allocation has also allowed the team to strengthen their core framework and start moving into new territories. They have just finished a European project called MSO4SC, in which they have been developing an HPC cloud framework that connects their software directly to supercomputers via the cloud. “As far as we know this
is the first type of framework like this,” says Prud’homme. “This allows users to access supercomputers in a seamless way, using our applications that have already been tuned for the specific machines being used. All we do is provide some guidelines about how to select the proper resources for the task at hand.”
One pilot application being tested out is Eye2Brain, which uses a code to simulate flows and biomechanical behaviour inside the eye and part of the brain. The application provides insight into some of the mechanisms behind the development of certain conditions such as glaucoma, and will work with more neurodegenerative diseases in the future. This provides an in-silico measurement tool alongside the physical equipment in clinics, and the team worked alongside medical professionals to develop the models and parts of the software. They hope to provide doctors and medical technicians access to this software soon so that it can be used in a clinical research setting.
This PRACE allocation has provided Feel++ with the platform to work on their codes and bring them up to a level of maturity so that they can scale seamlessly to larger machines. “What we want is to be able to provide really useful applications to a wide variety of people like doctors, physicists and engineers,” says Prud’homme. “That is what we are now achieving thanks to PRACE.”
For more information
Resources awarded by PRACE
This project was awarded 6 million core hours on Curie hosted by GENCI at CEA, France
Publications
Thibaut Metivet, Vincent Chabannes, Mourad Ismail, Christophe Prud ‘Homme. High-order finite-element framework for the efficient simulation of multifluid flows. 2018.
Cécile Daversin, Christophe Prudhomme, Christophe Trophime. Full 3D MultiPhysics Model of High Field PolyHelices Magnets. IEEE Transactions on Applied Superconductivity, Institute of Electrical and Electronics Engineers, 2016, 26 (4), pp.1-4.
Silvia Bertoluzza, Micol Pennacchio, Christophe Prud’Homme, Abdoulaye Samake. Substructuring Preconditioners for h-p Mortar FEM
ESAIM: Mathematical Modelling and Numerical Analysis, EDP Sciences, 2016, 50 (4), pp.1057-1082.