Iteration aware prefetching for remote data access

TL;DR: It is shown that latency penalties can be dramatically reduced using explicit knowledge of a user's access pattern represented as an iterator, and planned additions to existing grid services to allow selection of datasets according to the user access pattern are described.

Abstract: Although processing speed, storage capacity and network bandwidth are steadily increasing, network latency remains a bottleneck for scientists accessing large remote data sets. This problem is most acute with n-dimensional data. Grid researchers have only recently begun to develop tools for efficient remote access to n-dimensional data sets. Within the context of the Granite Scientific Database system, we show that latency penalties can be dramatically reduced using explicit knowledge of a user's access pattern represented as an iterator. The iterator not only performs an n-dimensional iteration for the user, but also communicates the access pattern to Granite so that a prefetching cache can be constructed that is tuned to the user's access pattern. We experimentally evaluate a scenario for incorporating Granite's prefetching mechanism into the grid, demonstrating extraordinary performance gains. In light of these results, we describe planned additions to existing grid services to allow selection of datasets according to the user access pattern