Physical Data Flow

Abstract: We present a code-generation-based optimization approach to bringing performance and scalability to distributed stream processing applications. We express stream processing applications using an operator-based, stream-centric language called SPADE, which supports composing distributed data flow graphs out of toolkits of type-generic operators. A major challenge in building such applications is to find an effective and flexible way of mapping the logical graph of operators into a physical one that can be deployed on a set of distributed nodes. This involves finding how best operators map to processes and how best processes map to computing nodes. In this paper, we take a two-stage optimization approach, where an instrumented version of the application is first generated by the SPADE compiler to profile and collect statistics about the processing and communication characteristics of the operators within the application. In the second stage, the profiling information is fed to an optimizer to come up with a physical data flow graph that is deployable across nodes in a computing cluster. This approach not only creates highly optimized applications that are tailored to the underlying computing and networking infrastructure, but also makes it possible to re-target the application to a different hardware setup by simply repeating the optimization step and re-compiling the application to match the physical flow graph produced by the optimizer. Using real-world applications, from diverse domains such as finance and radio-astronomy, we demonstrate the effectiveness of our approach on System S -- a large-scale, distributed stream processing platform.

Abstract: A method, system and article of manufacture for accessing data in a database independent of the particular manner in which the data is physically represented and for modeling a physical representation of the data in the database for architecting a relationship between the physical representation of the data in the database and a logical representation of the data. One embodiment provides a method for architecting a relationship between a physical representation of data in a database and a logical representation of the data. The method comprises providing the logical representation defining logical fields; and then providing, on the basis of the logical representation, the physical representation for the data, whereby the logical fields defined by the logical representation provide an abstract view of the data and map logical fields of abstract queries to respective data in the database.

Abstract: An alignment data structure is used to map a logical data block start address to a physical data block start address dynamically, to service a client data access request. A separate alignment data structure can be provided for each data object managed by the storage system. Each such alignment data structure can be stored in, or referenced by a pointer in, the inode of the corresponding data object. A consequence of the mapping is that certain physical storage medium regions are not mapped to any logical data blocks. These unmapped regions may be visible only to the file system layer and layers that reside between the file system layer and the mass storage subsystem. They can be used, if desired, to store system information, i.e., information that is not visible to any storage client.

Abstract: The Object-Oriented Software Development Method (OOSD) includes object-oriented requirements analysis, as well as object-oriented design. OOSD is a practical method of developing a software system which focuses on the objects of a problem throughout development. OOSD's focus on objects early in the development, with attention to generating a useful model, creates a picture of the system that is modifiable, reusable, reliable, and understandable — the last perhaps most important because the picture of a software system created by a development method must be an effective device for communication among developers, customers, management, and quality-assurance personnel.Most object-oriented methods competing for the attention of the software developer actually apply traditional Structured Analysis (function-based), or variations of Structured Analysis, to requirements activity, and work through a transition process to an object-oriented design [1,2,7,10,11]. In these methods the developer begins with functionally-based requirements analysis, and only reaches an object-oriented design by the intermediary step of converting a traditional, functionally-decomposed data flow diagram (DFD) to an object-oriented DFD (or equivalent). In this conversion process, objects are identified through a set of heuristics which group “transformations” in the DFD generated during requirements analysis. These methods carry a number of interesting but unfortunate burdens. Lower-level objects, which directly relate to real-world objects, are easily identified, but higher-level objects are generally more arbitrary, so that developers do not consistently identify a hierarchy of objects which achieves significant improvement in software engineering goals (e.g., reliability, maintainability, reusability). The heuristics for identifying objects usually relate the DFD transforms to the object that controls execution of an operation, rather than the object which “owns” the operation. These methods generally ignore the need to convert behavior descriptions of the DFD transforms into behavior descriptions of the objects. Finally, the use of Structured Analysis in an otherwise object-oriented approach complicates the tracing of requirements by forcing the developer to look first to DFD transforms and their behavior descriptions, and then to the objects.