The notion of program behavior is related to the execution of a program. There have been relatively few studies on the behavior of parallel programs executing on shared-memory multiprocessor systems. Yet the understanding of parallel program behavior is crucial to the design of parallel systems. The goal of our study is to analyze the memory reference behavior of large parallel scientific application programs executing on a shared-memory multiprocessor. Several important aspects of memory reference behavior are studied, such as the characteristics of the memory reference pattern, the degree of data sharing among processors, and the temporal and spatial locality of memory references. We discuss the impact of memory reference behavior on memory hierarchy design in shared-memory multiprocessors, and trace reduction techniques.