Software idioms for component-based and topology-aware simulation assembly and data exchange in high performance computing and visualisation environments

The emerging exascale systems pose many challenges to the development of HPC hardware and software solutions. Today, more and more scientific applications consist of multiple components distributed among different systems. The following questions arise: how to deal with the huge data generated by such simulations; how to transfer the data to the postprocessing units efficiently; how to deal with the growing complexity of the software? The recent research concerning access to the data generated by the simulation shows that, to achieve acceptable solutions for HPC, a combination of different techniques is required. Such techniques often need to be executed on separate clusters. Eligible methods are for example contract-based dataflow networks, in-situ processing or out-of-core streaming. HPC implementations of those algorithms are restricted due to the limited per-node memory and I/O resources. Thus, topology- and network-aware data movement becomes an important factor. It has to be considered to improve the overall system. As a result, the integration and development of such hardware-oriented applications exhibit greater software complexity. Methods widely used in the field of software engineering such as component-based software and integrated development environments provide one way to tackle that. In this thesis, we propose: first, a query-driven exploration paradigm suitable for accessing multi-scale simulation data, which has small memory overhead and good scalability with respect to the underlying parallel applications; second, an efficient method for demand-driven exchange of data in parallel environments, which takes into account the underlying topology of the network and the available hardware resources; third, a component-based method for the development of HPC applications. All this is embedded into an integrated development environment with a light layer for the interaction between software components running on distributed systems. To show the impact of our methods on the posed questions we examine a coupling scenario of the Lattice-Boltzmann and Navier-Stokes solvers, both realised as distributed parallel components through our architecture.