TY - GEN
T1 - ADMAD
T2 - 5th IEEE International Workshop on Storage Network Architecture and Parallel I/Os, SNAPI 2008
AU - Liu, Chuanyi
AU - Lu, Yingping
AU - Shi, Chunhui
AU - Lu, Guanlin
AU - Du, David H.C.
AU - Wang, Dong Sheng
PY - 2008
Y1 - 2008
N2 - There is a huge amount of duplicated or redundant data in current storage systems. So Data Deduplication, which uses lossless data compression schemes to minimize the duplicated data at the interfile level, has been receiving broad attention in recent years. But there are still research challenges in current approaches and storage systems, such as: how to chunking the files more efficiently and better leverage potential similarity and identity among dedicated applications; how to store the chunks effectively and reliably into secondary storage devices. In this paper, we propose ADMAD: an Application-Driven Metadata Aware De-duplication Archival Storage System, which makes use of certain meta-data information of different levels in the I/O path to direct the file partitioning into more Meaningful data Chunks (MC) to maximally reduce the inter-file level duplications. However, the chunks may be with different lengths and variable sizes, storing them into storage devices may result in a lot of fragments and involve a high percentage of random disk accesses, which is very inefficient. Therefore, in ADMAD, chunks are further packaged into fixed sized Objects as the storage units to speed up the I/O performance as well as to ease the data management. Preliminary experiments have demonstrated that the proposed system can further reduce the required storage space when compared with current methods (from 20% to near 50% according to several datasets), and largely improves the writing performance (about 50%-70% in average).
AB - There is a huge amount of duplicated or redundant data in current storage systems. So Data Deduplication, which uses lossless data compression schemes to minimize the duplicated data at the interfile level, has been receiving broad attention in recent years. But there are still research challenges in current approaches and storage systems, such as: how to chunking the files more efficiently and better leverage potential similarity and identity among dedicated applications; how to store the chunks effectively and reliably into secondary storage devices. In this paper, we propose ADMAD: an Application-Driven Metadata Aware De-duplication Archival Storage System, which makes use of certain meta-data information of different levels in the I/O path to direct the file partitioning into more Meaningful data Chunks (MC) to maximally reduce the inter-file level duplications. However, the chunks may be with different lengths and variable sizes, storing them into storage devices may result in a lot of fragments and involve a high percentage of random disk accesses, which is very inefficient. Therefore, in ADMAD, chunks are further packaged into fixed sized Objects as the storage units to speed up the I/O performance as well as to ease the data management. Preliminary experiments have demonstrated that the proposed system can further reduce the required storage space when compared with current methods (from 20% to near 50% according to several datasets), and largely improves the writing performance (about 50%-70% in average).
UR - http://www.scopus.com/inward/record.url?scp=58049182904&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=58049182904&partnerID=8YFLogxK
U2 - 10.1109/SNAPI.2008.11
DO - 10.1109/SNAPI.2008.11
M3 - Conference contribution
AN - SCOPUS:58049182904
SN - 9780769534084
T3 - Proceedings - 5th IEEE International Workshop on Storage Network Architecture and Parallel I/Os, SNAPI 2008
SP - 29
EP - 35
BT - Proceedings - 5th IEEE International Workshop on Storage Network Architecture and Parallel I/Os, SNAPI 2008
Y2 - 22 September 2008 through 22 September 2008
ER -