Sifting through network data to cull activity patterns with HEAPs

Today's large campus and enterprise networks are characterized by their complexity, i.e. containing thousands of hosts, and diversity, i.e. with various applications and usage patterns. To effectively manage and secure such networks, network operators and system administrators are faced with the challenge of characterizing, profiling and tracking activity patterns passing through their networks. Because of the large number of IP addresses and the prevalence of dynamic IP addresses, profiling and tracking individual hosts may not be effective nor scalable. In this paper, we develop a hierarchical extraction of activity patterns (HEAPs), which is a method for characterizing and profiling activity patterns within subnets. By representing activities within a subnet in a host-port association matrix (HPAM) and applying pLSA, we obtain co-clusters that capture the significant and dominant activity patterns of the subnet. Using these co-clusters, we utilize hierarchical clustering to cluster activity patterns to assist network operators and security analysts gain a "big-picture" view of the network activity-patterns. We also develop a novel method to track and quantify changes in activity patterns within subnets over time and demonstrate how to utilize this method to identify major changes and anomalies within the network.