Web Proxy Auto Discovery for Dynamically Created Web Proxies

The WLCG Web Proxy Auto Discovery (WPAD) service provides a convenient mechanism for jobs running anywhere on the WLCG to dynamically discover web proxy cache servers that are nearby. The web proxy caches are general purpose for a number of different http applications, but different applications have different usage characteristics and not all proxy caches are engineered to work with the heaviest loads. For this reason, the initial sources of information for WLCG WPAD were the static configurations that ATLAS and CMS maintain for the Conditions data that they read through the Frontier Distributed Database system, which is the most demanding popular WLCG application for web proxy caches. That works well for use at traditional statically defined WLCG sites, but now that usage of commercial clouds is increasing, there is also a need for web proxy caches to dynamically register themselves as they are created. A package called Shoal had already been created to manage dynamically created web proxy caches. This paper describes the integration of the Shoal package into the WLCG WPAD system, such that both statically and dynamically created web proxy caches can be located from a single source. It also describes other improvements to the WLCG WPAD system since the last CHEP publication.

The Open High Throughput Computing Content Delivery Network

LHC experiments make extensive use of web proxy caches, especially for software distribution via the CernVM File System and for conditions data via the Frontier Distributed Database Caching system. Since many jobs read the same data, cache hit rates are high and hence most of the traffic flows efficiently over Local Area Networks. However, it is not always possible to have local web caches, particularly for opportunistic cases where experiments have little control over site services. The Open High Throughput Computing (HTC) Content Delivery Network (CDN), openhtc.io, aims to address this by using web proxy caches from a commercial CDN provider. Cloudflare provides a simple interface for registering DNS aliases of any web server and does reverse proxy web caching on those aliases. The openhtc.io domain is hosted on Cloudflare's free tier CDN which has no bandwidth limit and makes use of data centers throughout the world, so the average performance for clients is much improved compared to reading from CERN or a Tier 1. The load on WLCG servers is also significantly reduced. WLCG Web Proxy Auto Discovery is used to select local web caches when they are available and otherwise select openhtc.io caching. This paper describes the Open HTC CDN in detail and provides initial results from its use for LHC@Home and USCMS opportunistic computing.

Client-Driven Joint Cache Management and Rate Adaptation for Dynamic Adaptive Streaming over HTTP

Due to the fact that proxy-driven proxy cache management and the client-driven streaming solution of Dynamic Adaptive Streaming over HTTP (DASH) are two independent processes, some difficulties and challenges arise in media data management at the proxy cache and rate adaptation at the DASH client. This paper presents a novel client-driven joint proxy cache management and DASH rate adaptation method, named CLICRA, which moves prefetching intelligence from the proxy cache to the client. Based on the philosophy of CLICRA, this paper proposes a rate adaptation algorithm, which selects bitrates for the next media segments to be requested by using the predicted buffered media time in the client. CLICRA is realized by conveying information on the segments that are likely to be fetched subsequently to the proxy cache so that it can use the information for prefetching. Simulation results show that the proposed method outperforms the conventional segment-fetch-time-based rate adaptation and the proxy-driven proxy cache management significantly not only in streaming quality at the client but also in bandwidth and storage usage in proxy caches.

Multimedia Proxy Servers

On the Internet, multimedia objects are stored in content servers. The clients behind some proxy servers are located over a wide area network (WAN) far from the content servers (Figure 1). When a client accesses multimedia objects from a content server, the content server must either allocate sufficient disk and network resources to multicast or unicast the objects to the client (Ma & Shin, 2002). Otherwise, it rejects the client. Thus, the popular content server becomes the bottleneck in delivering multimedia objects. Proxy servers have the disk cache space, network bandwidth, and availability to cache parts of the multimedia objects for clients, making them good candidates to solve the bottleneck problem. However, large multimedia objects are not cached or only partially cached in current proxy servers. When fast optical networks are widely deployed, this problem is becoming more severe. Therefore, proxy caches must be enhanced to alleviate the bottleneck in popular content servers with multimedia objects. Multimedia proxy servers perform several functions in accessing multimedia objects over the Internet. We first present the background in the next section. Next, the cache replacement policies being used in proxy servers are described. Then, the object partitioning methods are described. After that, the transcoding method that converts high-resolution objects into low-resolution objects is described. Afterward, we present the cooperative caching method that can be applied to cache objects on proxy servers. Lastly, we describe a method to distribute proxy-server load using a depot.

Efficient and Scalable Client-Clustering for Proxy Cache

The recent increase in popularity of the Web has led to a considerable increase in the amount of Internet traffic. As a result, the Web has now become one of the primary bottlenecks to network performance and web caching has become an increasingly important issue. Web caching aims to reduce network traffic, server load, and user-perceived retrieval delay by replicating popular content on caches that are strategically placed within the network. Browser caches reside in the clients’ desktop, and proxy caches are deployed on dedicated machines at the boundary of corporate network and Internet service providers.