High Performance Wide Area Data Transfers Over High Performance Networks 

 

 

Paul Woodward

Laboratory for Computational Science and Engineering

University of Minnesota 

Abstract 

This paper introduces a new user-level communication protocol designed to provide high-performance data transfers across high-bandwidth, high-delay, networks. The protocol incorporates the most important enhancements defined by the networking community to improve the performance to TCP for this environment, and also defines enhancements unique to this protocol. In terms of the so-called ?Large Window? extensions to TCP, this protocol implements a communication window that is essentially infinite, and provides a selective acknowledgement window that spans the entire data transfer. In terms of user-level extensions to TCP, it implements a user-defined acknowledgement frequency, a user-defined ?batch sending? window, and a simple framework within which the user can define the algorithm that determines the next data packet to be sent out across all eligible packets. We present experimental results demonstrating data throughput on the order of 85% - 92% of the maximum available bandwidth across both short haul and long haul high-performance networks. For comparison, we also tested an optimized TCP algorithm across the same networks. We found that a single optimized TCP stream also provided excellent performance across the short haul network (on the order of 90% of the maximum bandwidth), but found that it could not provide good performance over a long haul network (achieving only 10% of the maximum bandwidth). 

1 Introduction 

The Internet2 initiative [19] promises the development of high-performance networking applications in diverse areas such as distributed collaboration, visualization of scientific data, high performance grid-based computations, and Internet telephony. Abilene [20], the high-performance back-bone network associated with the Internet2 project, provides an OC-48 connection between the regional aggregation points (Gigapops) that it connects. Thus there is significant bandwidth available for such advanced networking applications, and the issue becomes one of actually realizing the available bandwidth.  

It has been well documented that, in practice, user-level distributed applications connected by Abilene achieve only a small percentage of the available bandwidth [1, 3,4,5,6,7,10]. The primary reason for this poor performance is that the Transmission Control Protocol (TCP) [11], the communication mechanism of choice for most applications, is not well suited for high-performance, wide area data transfers [3,4,5,6,13,14,15]. Thus one critical area of current research is the development of mechanisms to improve the performance of TCP in a high-bandwidth, high-delay environment, and another is to study alternative communication protocols that can circumvent the problems associated with TCP

      . 

In this paper, we present the results of our efforts to achieve high-performance wide area data transfers between selected sites connected by the Abilene backbone network. In particular, we developed and tested a very simple user-level communication protocol, utilizing a single UDP stream with a simple acknowledgement and retransmission mechanism, designed specifically for this type of environment. We tested our user-level protocol against (an optimized TCP stream) on data transfers between Argonne National Laboratory and the Laboratory of Computational Science and Engineering (LCSE) at the University of Minnesota, and between Argonne National Laboratory and the Center for Advanced Computing Research (CACR) at the California Institute of Technology.  

Our results are encouraging, We obtained over 90% of the maximum available bandwidth on data transfers between ANL and LCSE using both approaches: that is, using a single (optimized) TCP stream and our simple user-level protocol that utilizes a single UDP stream. On data transfers between ANL and CACR however, only the user-level protocol was able to obtain such a high percentage of this maximum bandwidth. In particular, the user-level protocol obtained up to 85% of the maximum available bandwidth while the optimized TCP stream obtained only on the order of 10% of this maximum.   

There are three primary contributions of this paper. First, it outlines a simple user-level protocol that was shown to provide excellent performance across both short and long haul high-performance networks. As noted, these results were obtained with a single UDP data stream in conjunction with a simple acknowledgement and retransmission scheme. This is in contrast to the (the very few) alternative approaches that utilize multiple TCP streams (up to eight streams per host) to improve performance in this setting [1,10,13]. A single data stream has the advantage that it does not require the kernel to multiplex multiple TCP streams, where such multiplexing may negatively impact the performance of applications executing on that particular host.  

Another contribution of this paper is that it provides a detailed study of the impact on performance due to the various parameters that can be controlled at the user level. This provides insight into the relationship between the acknowledgement frequency and the amount of network resources ?wasted? due to the greedy nature of the algorithm, and also helps to explain the performance of TCP in this environment. Thirdly, the end-points of the experiments conducted between ANL and LCSE were two Pentium3-based Windows 2000 boxes running the ?off-the-shelf? winsock2 API. Windows 2000 supports the ?Large Window? extensions to TCP [6], and it is important to look at the performance of the Winsock2 API, with all of its support for high-performance data transfers, given that the vast majority of published bandwidth studies employ either Unix or Linux TCP implementations.  

The rest of this paper is organized as follows. In Section 2 we discussed other approaches to providing high-performance data transfers in a high-delay, high-bandwidth environment. In Section 3, the experimental design is presented. In Section 4, the user-level communication protocol is presented. In Section 5 we present the results of our experimental studies, and provide conclusions and future research in Section 6. 

2  Related Work 

There is a significant amount of research relating to obtaining (a high percentage of) the available bandwidth in high-bandwidth, high-delay networks. This research is proceeding along two fronts: One approach is fundamental research into mechanisms to improve the performance of TCP itself. The other approach is to develop techniques at the application level that circumvent the performance problems associated with TCP. We discuss each approach in turn.  

As discussed in [13,14,15], the size of the TCP window is the single most important factor in achieving good performance over high-bandwidth, high-delay  networks. To keep such ?fat? pipes full, the TCP window size should be at least as large as the product of the bandwidth and the round-trip delay. This has led to research in automatically tuning the size of the TCP socket buffers at runtime [16]. Also, it has led to the development of commercial TCP implementations that allow the system administrator to significantly increase the size of the TCP window to achieve better performance [14]. Another area of active research is the use of a Selective Acknowledgement mechanism [8,14,18] rather than the standard cumulative acknowledgement scheme. In this approach, the receiving TCP sends to the sending TCP a Selective Acknowledgement (SACK) packet that specifies exactly those packets that have been received, allowing the sender to retransmit only those segments that are missing. Additionally, ?fast retransmit? and ?fast recovery? algorithms have been developed that allow a TCP sender to retransmit a packet before the retransmission timer expires, and allows the TCP sender to increase the size of its congestion control window, when three duplicate acknowledgement packets are received (without intervening acknowledgements) [8,18]. An excellent source of information, detailing which commercial and experimental versions of TCP support which particular TCP extensions, may be found in [14]. 

At the user level, the allocation of multiple TCP streams has been investigated. PSockets [13] employs multiple TCP sockets to increase the size of the TCP window. As discussed by the authors, the limitations on TCP window sizes  are on a per socket basis, and thus striping the data across multiple sockets provides an aggregate TCP buffer size that is closer to the (ideal size) of the bandwidth times round-trip delay. A similar approach has been investigated within the domain of satellite-based information systems [10]. The Globus project [17] developed a grid-FTP[1] tool that employs multiple TCP streams per host, with (perhaps) multiple hosts. This again significantly increases the size of the TCP window. It is also worth noting that using multiple TCP sockets increases the probability that, at any given time, there will be at least one TCP stream that is ready to fire.  

Experimental Design 

We investigated (reasonably) large-scale data transfers on two high-performance network connections: one between Argonne National Laboratory (ANL) and the Laboratory for Computational Science and Engineering (LCSE) at the University of Minnesota, and one between ANL and the Center for Advanced Computing Research (CACR) at the California Institute of Technology. ANL is connected to both of these sites across Abilene. The endpoints at both ANL and LCSE were Intel Pentium3-based PCs running Windows 2000 and using the Winsock2 API. We did not have access to a Windows 2000 machine at CACR at the time of this writing, and used instead an SGI Origin200 (with two 225Mhz MIPS R10000 processors) running IRIX 6.5. Similarly, we did not have access to an IRIX 6.5 machine at ANL, and thus were forced to run the experiments between a Windows 2000 machine at one end and an IRIX 6.5 machine at the other.  

As noted on the Pittsburgh Super Computing Website [14], IRIX 6.5 (like Windows 2000) supports the RFC 1323 [6] ?Large Window? extensions to TCP. Both machines also support MTU path discovery [9], where the segment size is determined by the maximum packet size that can be transmitted across the complete path without fragmentation, rather than simply using a pre-determined segment size that may be smaller (and thus less efficient) than this value. Also, both machines support TCP Selective Acknowledgements [8,18]. However, the default TCP window size on the SGI Origin200 is (approximately) 64KB, and we did not have system-level access to the machine that would have allowed us to increase this window size. The TCP window under Windows 2000 is (or can very easily) be extended to one Gigabyte [7,14]. The slowest link on either pair of connections was 100 Mb/sec, which was incurred between the desktop PC at ANL used in these experiments,  and the Math and Computer Science Division?s  external router. 

The round-trip delay between ANL and LCSE was measured (using traceroute) to be on the order of 26 milliseconds, and we (loosely) categorized this as a short haul network. The round-trip delay between ANL and CACR was on the order of 65 milliseconds, which we (again loosely) categorized as a long haul network. The transmitted data size for the experiments between ANL and LCSE was 40 MB, and was 20 MB between ANL and CACR. Similar to the results found in [13], the amount of data transferred did not have a significant impact on the throughput rate, so we opted for a smaller data size on the long haul connection to decrease the cost of experimentation. It is interesting to note that the bandwidth delay product for the ANL to LCSE connection was 1.04 Gigabytes, which was only fractionally larger than the TCP window size used by Windows 2000. The bandwidth delay product for the connection between ANL and CACR was 1.3 Gigabytes, which is orders of magnitude larger than the default (approximately) 64 KB buffer allocated by IRIX6.5. As will be seen, this significant difference in the size of the TCP window had a tremendous impact on the performance of the TCP algorithm.   

In our experiments, we replicated thirty trials of sending either 20 MBs of data (across the haul network),  or 40 MBs of data (across the short haul network). The metric of interest was the percentage of the maximum available bandwidth (which is 100 Mb/sec) that was obtained for each approach. A byte re-ordering cost was necessitated by the architectural differences between the SGI Origin 200 and the Windows 2000 machine, and this cost was included in the final bandwidth values reported.  

Communication Protocols 

We tested two communication protocols: an optimized (single) TCP stream and a user-level UDP-based protocol. We found that tuning the performance of the Windows-based TCP implementation was very simple. The primary optimization was to simply request a socket buffer size greater than 64 KB, which automatically enabled the ?Large Window? TCP extensions [7,14]. Additionally, we disabled the Nagel algorithm to avoid delays in placing packets on the network, and experimented with decomposing the data into smaller ?chunk sizes? (thus requiring multiple calls to the TPC send routine to complete the data transfer). We were very limited in optimizing TCP as implemented on the SGI Origin200 since we did not have system-level access to the TCP stack.  

4.1 User-Level UDP Protocol 

The user-level protocol we developed incorporates, at the user level, many of the important extensions defined for TCP in a high-bandwidth, high-delay network environment. One important assumption made by the user-level algorithm however is that both the sender and the receiver have pre-allocated buffers large enough to accommodate complete data transfer. This seems to be a very reasonable assumption, and certainly applies to the applications in which we are interested. In particular, it applies to wide-area MPI [21] (where for every send there is a matching receive), the File Transfer Protocol (FTP, where the disk is used  as a data buffer), and data visualization applications (where the generated data is consumed by the data receiver). Given this very important characteristic of the applications of interest, our user-level protocol pushes to the limit the idea of ?Large Window? extensions developed for TCP: that is, the window size is essentially infinite since it spans the entire data buffer (albeit at the user level). It also pushes to the limit the idea of selective acknowledgements. Given a pre-allocated receive buffer and constant packet sizes, each data packet in the entire buffer can be numbered. The data receiver can then maintain a very simple data structure with one byte (or even one bit) allocated per data packet, where this data structure tracks the received/not received status of every packet to be received. This information can then be sent to the data sending process at a user-defined acknowledgement frequency. Thus the selective acknowledgement window is also in a sense infinite. That is, the data sender is provided with enough information to determine exactly those packets, across the entire data transfer, that have not yet been received (or at least not received at the time the acknowledgement packet was created and sent).  

There are also features unique to our protocol that have been determined (experimentally) to have a very significant impact on performance. One such feature is the acknowledgement frequency, which, as discussed above, determines the frequency at which an acknowledgement packet (containing the complete history of the data transfer) is sent to the data sender. Another issue is the algorithm that determines the next data packet, across all eligible packets (to be defined below), to be sent next.  (To understand the importance of this issue consider that when the data sender receives an acknowledgment packet, it must determine whether to perhaps re-send a data packet that was lost, or to send a ?new? data packet that has not yet been sent for the first time.) The third user-level parameter has to do with the number of packets the data sender should transmit before checking for (and processing) an acknowledgement packet.  

Algorithm Executed by the Data Sender 

The total data buffer was divided into pre-determined, equal, and fixed-sized packets of 1024 bytes. This packet size was determined by executing the MTU path discovery algorithm along the paths from Argonne National Laboratory to both LCSE and CACR. Thus in the case of the 20 MB data buffer there were of 19,532 packets, and there were 39,063 packets with 40 MB data buffer. One UDP socket was used to transmit data from the sender to the receiver, and another UDP socket was used to send acknowledgement packets from the receiver to the sender.  

The data-sending algorithm iterates over three basic phases. In the first phase, the data sender employs some algorithm to determine the number of data packets to be placed onto the network before looking for, and processing if available, an acknowledgement packet. This is referred to as a ?batch-sending? operation since all such packets are placed onto the network without interruption (although the select system call is used to ensure adequate buffer space for the packet). It is very important to note that after a batch-send operation the data sender looks for, but does not block for, an acknowledgement packet.  

In the second phase of the algorithm, the data sender looks for, and if available, processes an acknowledgement packet. Processing of an acknowledgement packet entails updating the receive/not received status for each data packet acknowledged, and determining the number of packets that were received by the data receiver between the time it created the previous acknowledgement packet and the time it created the current acknowledgement packet. This information can then be used to determine the number of packets to send in the next batch-send operation. If no acknowledgement packet is available, this information can also be used to determine the number of packets to send in the next batch-send operation. Note that a repeated batch-sending operation with zero packets is logically equivalent to blocking on an acknowledgement.  

In the third phase of the algorithm, the data sender executes some user-defined algorithm to choose the next packet, out of all unacknowledged packets, to be placed onto the network.  

4.2.1 Parameters for the Send Algorithm 

The first parameter studied was the number of packets to be sent in the next batch-send operation. Intuitively, one would expect that the data sender should check for an acknowledgement packet on a very frequent basis, thus limiting the number of packets to be placed onto the network in a given batch-send operation.  Our experimental results supported this intuition, finding that two packets per batch-send operation provided the best performance. We therefore used this number in all subsequent experimentation 

We also performed extensive experimentation to determine which particular packet, out of all unacknowledged packets, should next be placed onto the network. We tried several algorithms, and, in the end, it became quite clear that the best approach (by far) was to treat the data as a circular buffer. That is, the algorithm never went back to re-transmit a packet that was not yet acknowledged, if there were any packets that had not yet been sent for the first time. Similarly, a given packet was re-transmitted for the n+1st time only if all other unacknowledged packets had been re-transmitted n times.  

As can be seen, the algorithm executed by the sender is very greedy, continuing to transmit (or re-transmit) packets (without blocking) until it receives an acknowledgement packet from the data receiver specifying that all data has been successfully received. Thus a reasonable question to ask is how wasteful of network resources is this approach. One measure of wasted resources applicable to this approach is the number of duplicate packets received by the data receiver over the course of the entire data transfer. We did track this information, where the data receiver maintained a simple counter that was incremented every time it received a packet that had already been marked as having been received. In hind-sight, it also would have also been useful to track the number of messages still in the pipe when the receiver determined it had obtained all of the data (and thus had stopped trying to read packets off of the network). In future research, we will track this information, and also attempt to track the number of packets lost in the network due to contention.  

4.3 Algorithm Executed by the Data Receiver 

The data receiver iterates over a loop with the select system call at the top of the loop. The select system call takes as parameters a set of socket descriptors (in the case of Windows 2000), or a set of file descriptors (in the case of Unix). The select call also takes as a parameter a timer. The select call returns when one of the sockets is available for reading or writing, or when the timer expires. We set the time out value for the data receiver at 1.5 seconds. The basic algorithm is as follows. 

Use the select system call to determine if a data packet is available. 
If the select call times out send an acknowledgement packet (that contains a complete history of the data transfer). 
If a packet is available, read it off of the network and determine if it is a duplicate (i.e. has already been received and acknowledged). If it is a duplicate packet, discard it and increment a counter tracking the number of duplicate packets received. If it is not a duplicate packet, place the packet into its proper position within the data buffer using the packet number as an offset. 
If the data packet is not a duplicate, increment a counter tracking the total number of packets received by the data receiver. If this value exceeds some user-defined threshold, send an acknowledgement packet and reset the counter to zero. 
If the packet is a duplicate, and the number of duplicate packets received exceeds some threshold value, then send an acknowledgement packet and reset the duplicate counter to zero. 

4.3.1 Parameters for the Data Receiver  

The most important parameter with respect to the data receiver is the number of new packets received before generating and sending an acknowledgement packet. The frequency with which the data receiver sends acknowledgement packets essentially determines the level of synchronization between the two processes. A small value (and thus a high level of synchronization) implies that the data receiver must frequently stop pulling packets off of the network to create and send acknowledgement packets. Given that the algorithm is UDP-based, those packets missed while creating and sending an acknowledgement will, in all likely-hood, be lost. A very high value, and thus a very low level of synchronization, results in both the data sender and data receiver spending virtually all of their time placing packets on, and reading packets off, of the network. As will be seen, this is actually a very good approach when the pipe is completely clear. 

The acknowledgement frequency, as described above, takes into consideration only the number of new data packets received. We found that it is also necessary to place a bound on the number of duplicate messages received before sending an acknowledgement packet. The data sender only sends duplicate packets when it cannot correctly determine (or anticipate) the packets that have not yet been received by the data receiver. Thus when the data sender if clearly ?off the mark? in terms of the packets it is selecting to send (or re-send), it is helpful for the data receiver to send an acknowledgement packet to provide an updated view of the state of the data transfer. Our experimentation suggested that sending an acknowledgement packet whenever the duplicate count reached 50 provided the best performance over both network connections.  

 

Experimental Results 

We compared the performance of the user-level UDP protocol against that of an optimized TCP implementation (when both end-points were executing the winsock2 API), and against a TCP implementation that we were unable to optimize in any meaningful way (i.e. the IRIX 6.0 TCP implementation). As noted, we were unable to significantly modify the window size on the SGI Origin200 since we did not have system-level access on the machine. First, consider the performance of the user-level UDP protocol shown in Figure 1. This figure depicts the performance of the approach as a function of the number of packets received before triggering an acknowledgement. As can be seen, this simple protocol, involving a single UDP stream (for data sending), provides excellent performance across both platforms and across both the short haul and the long haul connections.  In particular, the protocol achieved a throughput of over 90% of the maximum available bandwidth the on the connection between ANL and LCSE. Also, it obtained on the order of 85% of the available on the connection between ANL and CACR. It is important to remember that these results were obtained using a single (optimized) UDP stream.  

 

 

Figure 1. This figure depicts the percentage of the maximum available bandwidth obtained by the user-level UDP protocol as a function of the number of packets received before sending an acknowledgement packet.  

It is interesting to note the impact on performance due to the acknowledgement frequency. When this frequency was very high (e.g. 1/50), there was a significant detrimental impact on performance. This is due to the fact that when the data sender and data receiver were tightly synchronized, both processes were spending a non-trivial amount of time preparing, sending, receiving, and processing acknowledgement packets. Even though the cost of preparing/processing acknowledgements was not necessarily large, this extra time devoted to processing such packets (and thus not sending/receiving packets) turned out to have a significant impact (at least on the high-performance networks we tested). As the acknowledgement frequency decreased (down to a minimum value of one out of every 3000 packets), the performance began to improve. The throughput on the short haul network peaked out at a little over 90% of the available bandwidth. In the case of the long haul network, the throughput peaked out at approximately 85% of the available bandwidth. 

It is also interesting to look at the number of duplicate packets received by the data receiver across the complete data transfer (as note however, this value does not reflect packets in the pipe when the data receiver stopped looking for more packets). In the case of the short haul network, it was rare to observe more than one or two duplicate packets for all acknowledgement frequencies studied. Thus the pipe was very clear, and there were virtually no packets being lost in the network. This was somewhat surprising given that the trials were conducted during normal business hours, although the summer students at Argonne National Laboratory had departed (vastly reducing the load on the ANL networks). 

 

Figure 2. This figure depicts the number of duplicate packets received by the data receiver as a function of the acknowledgement frequency.  

As can be seen however, the number of duplicate packets did significantly increase when the data was transferred over the long haul network. This number was not large when the data sender and data receiver were tightly synchronized, but as the level of synchronization began to decrease, the number of duplicate packets began to increase (and rather dramatically after the acknowledgement frequency was less than 1/900). The number of duplicate packets reached a value of around 550 (when the acknowledgement frequency was reduced to 1/2500). It is interesting to note that even though the number of duplicate packets significantly increased with a decreased acknowledgement frequency, this did not have a significant negative impact on performance. This can be best understood by considering that even with 550 duplicate packets, this represented only 0.5 MB of additional data on the network. This, in turn, represented less than 3% of the total data transfer.  

TCP Performance 

Figure 3 depicts the performance of TCP across the short and long haul networks. As can be seen, the results obtained using the Windows 2000 TCP implementation (across the short haul network) were quite impressive, providing approximately the same performance as that of the user-level protocol. These results certainly help emphasize the fact that large TCP windows are imperative over high-bandwidth, high-delay networks. The other factor allowing TCP to obtain such good performance was most likely due to the absence of contention in the network. The fact that virtually no packets were duplicated in the UDP protocol (at least strongly) suggests that TCP experienced very little packet loss across this same network. Thus the TCP congestion control mechanisms were not triggered, allowing the TCP window to be advanced without (any significant) delay. Clearly the research in optimizing TCP for this type of network has produced dramatic improvements in performance.

Figure 3. This figure shows the performance of TCP across a long haul and a short haul high-performance network. The ?chunk? size reflects a decomposition of the total buffer size into smaller data units.  

As can be seen however, the performance of TCP drops dramatically over the long haul network. There were two reasons for this: First, the TCP window size was significantly (by several orders of magnitude) smaller than the bandwidth delay product. As noted, we were not able to modify this window size since we did not have system-level access. Secondly, judging by the packet loss incurred by the user-level protocol, it is likely that TCP also experienced packet loss during the data transfer. Thus the TCP congestion control algorithms were most likely triggered, resulting in a significant drop in performance. When we secure a high-performance Windows 2000 machine at the other end-point of this connection, we should be able to determine how much of the degradation was due to the window size and how much was due to packet loss and the subsequent triggering of the congestion control mechanisms.   

We were also interested in whether breaking the data into smaller data units (?chunks?), and invoking the TCP send algorithm multiple times, would have an impact on performance. As can be seen, this approach did result in some performance improvement over the short haul network, but did not appear to have any impact on performance across the long haul network.  

Conclusions and Future Research 

In this paper, we have reported on the design and implementation of a user-level UDP protocol designed for high-delay, high-bandwidth networks. The most important features of this algorithm include a (logically) infinite window size, a (logically) infinite selective acknowledgement window, and a user-define acknowledgement frequency. This algorithm was shown to achieve a throughput of over 90% of the maximum bandwidth when executed across a short haul, high-performance network. Over a long haul network, it was still able to achieve throughout on the order of 85% of this maximum. 

This research also clearly demonstrates the importance of the window size in the performance of TCP. When the ?Large Window? extensions to TCP were enabled, a single TCP stream was able to achieve on the order of 90% of the available bandwidth. This performance was dramatically reduced however when the TCP window was significantly smaller than the bandwidth delay product, and when packet loss was introduced into the transfer.  

Currently, we are investigating the significant performance decline suffered by TCP across the long haul network. We are interested in sorting out the impact on performance due to the TCP window size and due to the triggering of the TCP congestion control algorithms. We are also interested in providing a better definition and a more robust measurement of any waste in network resources due to the greedy nature of the user-level algorithm. Finally, we wish to study the impact on the performance of other applications (executing on the same processor) based on the communication protocol being employed (i.e. the user-level protocol defined herein, versus the use of multiple TCP streams. 

 

References

      [1] Allcock, B. Bester, Bresnahan, J., Chervenak, A., Foster, I.,  Kesselman, C. Meder, S., Nefedova, V., Quesnet, D., and S. Tuecke. Secure,  Efficient Data Transport and  Replica Management for  High-Performance  Data-Intensive Computing. Preprint   ANL/MCS-P871-0201, Feb.  2001.

      [2] Allman, M., Paxson, V., and W.Stevens. TCP Congestion Control. RFC 2581, April 1999.

[3] Feng, W. and P. Tinnakornsrisuphap.  The Failure of TCP in High-Performance Computational Grids. In the Proceedings of the Super Computing 2000 (SC2000).  

[4] Hobby, R. Internet2 End-to-End Performance Initiative (or Pat Pipes Are Not Enough). URL: http//www.internet2.org.

[5] Irwin, B. and M. Mathis. Web100: Facilitating High-Performance Network Use. White Paper for the Internet2 End-to-End Performance Initiative. URL:http://www.internet2.edu/e2epi/web02/p_web100.shtml

[6] Jacobson, V., Braden, R., and D. Borman. TCP Extensions for high performance. RFC 1323, May 1992.

[7] MacDonald and W. Barkley. Microsoft Windows 2000 TCP/IP Implementation Details. White Paper, May 2000.

[8] Mathis, M., Mahdavi, J., Floyd, S. and A. Romanow. TCP Selective Acknowledgement Options. RFC 2018

[9] Mogul, J. and S. Deering, "Path MTU Discovery", RFC 1191,

            November 1990.

[10] Ostermann, S., Allman, M., and H. Kruse. An Application-Level solution to TCP?s Satellite Inefficiencies. Workshop on Satellite-based Information Services (WOSBIS), November, 1996.

[11]  J. Postel, Transmission Control Protocol, RFC793, September 1981.

[12] Semke, J., Jamshid Mahdavi, J.,  and M. Mathis, Automatic TCP Buffer Tuning, Computer Communications Review, a publication of ACM SIGCOMM, volume 28, number 4, October 1998].

[13] Sivakumar, H., Bailey, S., and R. Grossman. PSockets: The Case for Application-level Network Striping for Data Intensive Applications using High Speed Wide Area Networks. In Proceedings of Super Computing 2000 (SC2000).

[14] URL: http://www.psc.edu/networking/perf_tune.html#intro

      Enabling High Performance Data Transfers on Hosts: (Notes for Users and System Administrators)

[15] URL: http://dast.nlanr.net/Articles/GettingStarted/TCP_window_size.html  

[16] URL: http://dast.nlanr.net/Projects/Autobuf_v1.0/autotcp.html. Automatic TCP Window Tuning and Applications

[17] URL: http://www.globus.org

[18] URL: http://www.psc.edu/networking/all_sack.html. List of sack implementations  

[19] URL: http://www.internet2.org

[20] URL: http://www.internet2.edu/abilene

[21] URL: http://www-unix.mcs.anl.gov/mpi/mpich/