dCache: Inter-disciplinary storage system

The dCache project provides open-source software deployed internationally to satisfy ever more demanding storage requirements. Its multifaceted approach provides an integrated way of supporting different use-cases with the same storage, from high throughput data ingest, data sharing over wide area networks, efficient access from HPC clusters and long term data persistence on a tertiary storage. Though it was originally developed for the HEP experiments, today it is used by various scientific communities, including astrophysics, biomed, life science, which have their specific requirements. In this paper we describe some of the new requirements as well as demonstrate how dCache developers are addressing them.

Time Slicing Method

Tertiary storage devices provide huge storage capacity at low cost. Multimedia objects stored on the tertiary storage devices are accessed with high latency. Despite the high access latency, some tertiary storage devices are able to deliver data at high throughput. The time slicing method is designed to reduce the start-up latency in accessing multimedia objects from tertiary storage devices. The start-up latency is lowered by reducing the amount of data being migrated in stage one of the staging method being described in the last chapter. In order to support the time-slicing method, the tertiary storage devices should have the ability to deliver data at high throughput. The tertiary storage devices that cannot deliver data at sufficiently high throughput; the start-up latency cannot be reduced.

Tertiary Storage Devices

The main objective of the tertiary storage level is to provide huge storage capacity at low cost. Several types of storage devices are available to be used at the tertiary storage level in Hierarchical Storage Systems (HSS). They include: • Magnetic tapes • Optical disks • Optical tapes These storage devices are composed of fixed storage drives and removable media units. The storage drives are fixed to the computer system. The removable media unit can be removed from the drives so that the storage capacity can be expanded with more media units. When data on a media are accessed, the media unit is accessed from their normal location. One of the storage drives on the computer system is chosen. If there is a media unit in the storage drive, the old media unit is unloaded and ejected. The new media unit is then loaded to the drive. Each type of storage drive may handle the storage drives and media units differently. The magnetic tapes are described below in the next section. Then, the optical tapes are presented. Afterwards, the optical disks are briefly described before this chapter is summarized.

Staging Methods

When data are stored in the tertiary storage devices, the tape drives shall read them from the tapes using the input/output (I/O) operations. Due to the long delay in exchanging tapes, it is inconvenient to exchange a tape for each read/write access operation. Thus, the entire object or file is accessed from the tape drives well before they are being used (Federighi & Rowe, 1994; Kienzle, 1995; Pang, 1997). These accessed objects are temporarily stored in the magnetic hard disks as secondary storage level.

Normal Pipelining

Multimedia objects can be stored on tertiary storage devices to provide large storage capacity at low cost. The staging method retrieves the whole objects to the staging buffers prior to consumption. Thus, the start-up latency is high. The time slice method being described in the last chapter reduces the start-up latency only when the tertiary storage bandwidth is higher than double of the displaying data rate of the object. However, if the tertiary storage bandwidth is below double of the data consumption rate of the object, then we can only stage the object prior to using it. The pipelining methods aim at minimizing the start-up latency when the tertiary storage bandwidth is not higher than the data consumption rate of the objects. The pipelining methods are used to reduce the start-up latency and staging buffer size. In the normal pipelining method, the sizes of the slices are minimized to maximize the overlapping between the displaying time and the retrieval time of the slices. In the space efficient pipelining methods, the buffer size in accessing the slices is minimized. In the segmented pipelining method, the latency in serving interactive requests is reduced. The normal pipelining method is described in this chapter. The space efficient pipelining method and the segmented pipelining method are presented in the following two chapters. We shall describe the objective of the normal pipelining method. Then, the bounds on the sizes of the slices are shown. After that, the start-up latency and the minimum size of the first slice are shown. The reduction in the startup latency using the normal pipelining method is presented.

Contiguous Placement on Hierarchical Storage Systems

The contiguous placement is the most common method to place traditional data files on tertiary storage devices. The storage space in the media units is checked. The data file is stored on a media unit with enough space to store the data file. When tertiary storage devices are used to store multimedia objects, the objects are stored and retrieved similar to traditional data files. Since the main application of the tertiary storage devices is to back up multimedia objects from computers, the objectives of the contiguous method are: 1. supporting back up of multimedia objects efficiently, and 2. reducing the number of separate media units that are used to store an object. We will describe in the next sections the simple contiguous placement method. Afterwards, the log structured placement method is explained before we summarize this chapter.

Space Efficient Pipelining

Multimedia objects that are stored on tertiary storage devices enjoy the large storage capacity at low cost. These objects may be retrieved using staging, time slicing, or pipelining. The staging method retrieves the whole objects to the staging buffers prior to consumption at the cost of high start-up latency. The time slice method reduces the start-up latency at the cost of heavy switching overheads. The pipelining methods aim at minimizing the start-up latency when the tertiary storage bandwidth is not higher than the data consumption rate of the objects. Three pipelining methods are used to reduce the start-up latency and staging buffer size: 1. Normal pipelining 2. Space efficient pipelining 3. Segmented pipelining In the normal pipelining method, the sizes of the slices are minimized to maximize the overlapping between the displaying time and the retrieval time of the slices. In the space efficient pipelining (SEP) methods, the buffer size in accessing the slices is minimized. In the segmented pipelining method, the latency in serving interactive requests is reduced. We have described the normal pipelining method in the previous chapter. The space efficient pipelining method is explained in this chapter. The segmented pipelining method is presented in the next chapter. In this chapter, the basic space efficient pipelining algorithm is first described in the next section. Next, the buffer replacement policies are explained before this chapter is summarized.