Huge data but small programs: visualization design via multiple embedded DSLs

Abstract

Although applications of functional programming are diverse, most examples deal with modest amounts of data -- no more than a few megabytes. This paper describes how Haskell has been used to address a challenging visualization problem, involving 200 time steps from an astrophysics simulation of early star formation. Each individual step involves nearly 40 million samples, and uncompressed the complete dataset is nearly a terabyte.

Our solution makes novel and extensive use of domain specific languages to specify data resources, rendering abstractions, and most significantly, the desired visualization. The result is a powerful framework for multi-field visualization, including the use of animation to explore the evolution of data over time. This approach represents a significant departure from standard practices in visualization, and has applications well beyond the original problem. That our solution consists of less than 4.5K lines of code is itself a notable result. This paper motivates and describes the overall architecture of our solution, and technical features of the DSLs that are used in place of the traditional visualization pipeline. We conclude with thoughts on how functional technologies may find even broader use within other branches of visualization.