TCP INCAST PROBLEM
Incast is a
many-to-one communication pattern commonly found in cloud data centers
implementing scale out distributed storage and computing frameworks such as
Hadoop, MapReduce, HDFS, Cassandra, etc. — powering applications such as web
search, maps, social networks, data warehousing and analytics. Incast can also
more specifically be referred to as TCP Incast, as the cloud applications
creating this communication pattern rely heavily on TCP.
The basic
pattern of Incast begins when a singular Parent server places a request for
data to a cluster of Nodes which all receive the request simultaneously. The
cluster of Nodes may in turn all synchronously respond to the singular Parent.
The result is a micro burst of many machines simultaneously sending TCP data
streams to one machine (many to one).
The Incast
traffic can be very short lived flows (micro burts), depending on the
application. For example, you could have a Parent server request 80KB of
data across 40 Nodes. Each Node simultaneously responds with 2KB of data.
That’s just two packets from each Node. In a real world scenario,
the Parent could be a database server requesting a 80KB photo of the newest
friend added to your social network.
“TCP Incast” may also be used in a context referring to the
detrimental effect on TCP throughput caused by network congestion the Incast
communication patterns create. This simultaneous many-to-one burst can cause egress congestion at
the network port attached to the Parent server, overwhelming the port egress
buffer. The resulting packet loss requires Nodes to detect the loss (missing
ACKs), re-send data (after RTO), and slowly ramp up throughput per standard TCP
behavior. The application issuing the original request might wait until all
data has been received before the job can complete, or just decide to return
partial results after a certain delay threshold is exceeded. Either way, the
speed, quality, and consistency of performance suffers.
The congestion caused by Incast has the effect of increasing the
latency observed by the application and its users. There’s a give and take with
application performance as Node cluster sizes grow. Larger clusters provide
more distributed processing capacity allowing more jobs to complete faster.
However larger clusters also mean a wider fan-in source for Incast traffic,
increasing network congestion and lowering TCP throughput per Node.
No comments:
Post a Comment