New improvements to IPFS Bitswap for faster container image distribution

Leveraging IPFS as a peer-to-peer CDN lets nodes inside Netflix’s infrastructure collaborate and seed common pieces to neighboring nodes, helping make container distribution faster.To speed this up even more, we added some useful new capabilities to the Bitswap protocol which cut transfer time in half for benchmarked use cases around container distribution. In this way the node tries to maintain a high download speed without too many duplicate blocks.In order to improve Bitswap performance and efficiency, the IPFS team made some changes to the way Bitswap fetches blocks.Initially a node wants to know which of its peers have the root block, but doesn’t actually want to receive the block itself (because it sends this “discovery” want to many peers). So we added a DONT_HAVE response.With these changes a node can cheaply work out how blocks are distributed amongst its peers, and can direct the requests for blocks more accurately, increasing overall download speed and reducing the number of duplicate blocks.The node can also quickly recognize when all the peers in a session don’t have a block it needs, and go out to the DHT to find out who has the block.For more details on how Bitswap worked before (“master”), and the changes we’ve made (“poc”), check out these slides, this presentation at the IPFS Weekly Call, and this recent performance comparison.