Project Group: Development of a flexible high-performance File System

News

  • The first meeting will be on Friday, October 22nd, from 5:30 pm, and will take place in the PC2 user center. The meeting includes a lectiure on storage and file systems.

Introduction

Research, companies, and governments require to store and process huge amounts of data. Today’s file systems are often spread to multiple parallel storage and metadata servers to cope with these demands. These file systems can store multiple petabytes of data and serve thousands of clients in parallel.

The novel pNFS (NFSv4.1) protocol allows a standardized access to a clustered file system and its server-side implementation yields a broad design space that affects the scalability, extendibility and overall performance of the file system.

Goals

Most file systems are tailored for specific workloads, while the goal of this project is to exploit standard protocols and modern technologies to create a very flexible and configurable distributed file system. Therefore, a standard compliant (distributed) pNFS server should be developed that could be used as a building block for scalable, efficient storage solutions. This includes

  • Implementation of a scalable, distributed pNFS Server
  • Exploitation of the given protocols to dynamically add new features
  • Addition of new storage layouts that support novel features like deduplication
  • Definition and implementation of Server-to-Server protocols
  • Performance evaluations

Requirements

This project group might be the right choice for you, if you are interested in having fun with parallel, distributed systems and file system internals.

Lecures

Seminar Topics

No. Topic Literature Student Advisor
1 Basic Concepts of Distributed File Systems E. Levy and A. Silberschatz: Distributed file systems- concepts and examples
P. Braam: File Systems for Clusters from a Protocol Perspective 		Serge Rittscher Ivan Popov
2 Ceph File System S. Weil, S. Brandt, E. Miller, D. Long, and C. Maltzahn: Ceph: A Scalable, High-Performance Distributed File System
A. Leung and E. L. Miller. Scalable security for large, high performance storage systems
S. Weil, K. Pollack, S. Brandt, and E. Miller: Dynamic metadata management for petabyte-scale file systems 		Christoph Kleineweber Matthias Grawinkel
3 Designing Reliable File Systems V. Henson, A. van den Ven, A. Gud, and Z. Brown: Chunkfs: Using divide–and–conquer to improve file system reliability and repair
K. Keeton, C. Santos, D. Beyer, J. Chase, J. Wilkes: Designing for disasters 		Karthik Neela Dirk Meister
4 Panasas File System G. Gibson, D. Nagle, K. Amiri, F. Chang, E. Feinberg, H. Gobioff, C. Lee, B. Ozceri, E. Riedel, D. Rochberg, J. Zelenka: File Server Scaling with Network-Attached Secure Disks
B. Welch, M. Unangst, Z. Abbasi, G. Gibson, B. Mueller, J. Small, J. Zelenka, and B. Zhou: Scalable Performance of the Panasas Parallel File System 		Gokul Krishnan  Ivan Popov
5 GPFS File System R. Fagin, J. Nievergelt, N. Pipenger, and H. Strong: Extendible hashing - a fast access method for dynamic files
F. Schmuck and R. Haskin: GPFS: A Shared-Disk File System for Large Computing Clusters 		Viktor Gottfried Ivan Popov
6 NFS B. Callaghan: NFS Illustrated
F. Pohlmann, K. Hess: NFSv4 delivers seamless network access 		Sebastian Moors Dirk Meister
7  Caching G. Yadgar, M. Factor, A. Schuster: Karma: Know-it-All Replacement for a Multilevel Cache
I. Ari: Design and Management of Globally Distributed Network Caches
T. Wong and J. Wilkes: My cache or yours? Making storage more exclusive 		Marie-Christine Jacobs Andre Brinkmann
8  PVFS  P. Carns, W. Ligon, R. Ross, R. Thakur: PVFS: A Parallel File System for Linux Clusters
A. Devulapalli, D. Dalessandro, P. Wyckoff, N. Ali, P. Sadayappan: Integrating Parallel File Systems with Object-based Storage Devices
A. Devulapalli, D. Dalessandro, P. Wyckoff: Data Structure Consistency Using Atomic Operations in Storage Devices 		Markus Mäsker Andre Brinkmann
9 Metadata Management   Y. Hua, Y. Zhu, H. Jiang, D. Feng, L. Tian: Scalable and Adaptive Metadata Management in Ultra Large-scale File Systems
Y. Hua, H. Jiang, Y. Zhu, D. Feng, L. Tian: SmartStore: A New Metadata Organization Paradigm with Semantic-Awareness for Next-Generation File Systems
J. Xing, J. Xiong, N. Sun, J. Ma: Adaptive and Scalable Metadata Management to Support A Trillion Files 		Denis Dridger Matthias Grawinkel
10 IO and API  S. Patil, G. Gibson, G. Ganger, J. Lopez, M. Polte, W. Tantisiroj, and L. Xiao: In search of an API for scalable file systems: Under the table or above it?
D. Hildebrand, A. Nisar, R. Haskin: pNFS, POSIX, and MPI-­‐IO: A Tale of Three Semantics
S. Lang, R. Latham, D. Kimpe, R. Ross: Interfaces for Coordinated Access in the File System
S. Lang, P. Carns, R. Latham, R. Ross, K. Harms, W. Allcock: I/O Performance Challenges at Leadership Scale 		Sandeep Korrapati Andre Brinkmann
11  Transaction Support C. Wright, R. Spillane, G. Sivathanu, E. Zadok: Extending ACID Semantis to the File System
R. Spillane, S. Gaikwad, M. Chinni, E. Zadok, C. P. Wright: Enabling transactional file access via lightweight kernel extensions
D. Porter, O. Hofmann, C. Rossbach, A. Benn, E. Witchel: Operating System Transactions 		Benjamin Raddatz Matthias Grawinkel
12  pNFS  D. Hildebrand and P. Honeyman: Direct-pNFS: Scalable, Transparent, and Versatile Access to Parallel File Systems
M. Factor, K. Meth, D. Naor, O. Rodeh, J. Satran: Object Storage: The Future Building Block for Storage Systems
pNFS Specification 		Ajay Kumar  Dirk Meister
Teaching