By Yicheng Qin, July 24, 2015
Originally published on the CoreOS blog (archived copy) . Re-published with the authors' permission.
After months of focused work, etcd 2.1 has been released. Since the etcd 2.0 release in January, the team has gathered a ton of valuable feedback from real-world environments. And based on that feedback, this release introduces: authentication/authorization APIs, new metric endpoints, improved transportation stability, increased performance between etcd servers, and enhanced cluster stability.
For a quick overview, etcd is an open source, distributed, consistent key value store for shared configuration, service discovery, and scheduler coordination. By using etcd, applications can ensure that even in the face of individual servers failing, the application will continue to work. etcd is a core component of CoreOS software that facilitates safe automatic updates, coordinating work being scheduled to hosts, and setting up overlay networking for containers.
If you want to skip the talk and get right to the code, you can find new binaries on GitHub. etcd 2.1.1 is available in CoreOS 752.1.0 (currently in the alpha channel), so feel free to take it for a spin.
Upgrading from etcd 2.0 to etcd 2.1 is a zero-downtime rolling upgrade. The basic approach is that you can upgrade a cluster running etcd 2.0 one-by one to etcd 2.1. For more details, please read the upgrade documentation. If you are running your cluster under etcd 0.4.x, please upgrade to etcd 2.0 first and then follow the rolling upgrade steps.
Also, with this release, etcd 2.1 is now the current stable etcd release; as such, all bug fixes will go into new etcd 2.1.x releases and won't be backported to etcd 2.0.x.
A major feature in this release is the /v2/auth endpoint, which adds auth to the etcd key/value API. This API lets you manage authorization of key prefixes with users and roles and authenticate those users using HTTP basic authentication, enabling users to have more control within teams. This includes support in the etcd HTTP server, the command-line etcdctl client, and the Go etcd/client package. You can find full details in the authentication documentation. Please note that this is an experimental feature and will be improved based on user feedback. We think we got the details right but may adjust the API in a subsequent release.
Many users of etcd have networks with inconsistent performance and latency. We can't make etcd work perfectly in all of these difficult environments but what we have done in this release is optimize the way etcd uses the network in a variety of ways to make it perform in an optimal manner.
First, to reduce the connection creation overhead and to make the consensus protocol (raft) communication more efficient and stable, etcd now maintains long running connections with other peers. Next, to reduce the raft command commit latency, each raft append message is now attached to a commit index. The commit latency is reduced from 100ms to 1ms under light load (<100 writes/second). And finally, etcd's raft implementation now provides better internal flow control, significantly reducing the possibility of raft message loss, and improving CPU and memory efficiency.
For four months we have been running etcd against a fault-injecting and functional testing framework we built. Our goal is to ensure etcd is failure-resistant while under heavy usage; and in these months of testing, etcd has shown to be robust under many kinds of harsh failure scenarios. We will continue to run these tests as we iterate on the 2.1 releases.
Leveled logging is supported now. Users can set an expected log level for etcd and its subpackages. In the meantime, we have moved verbose, repeated logging to DEBUG log level, so etcd's default log will be significantly more readable. You can control leveled logging using flags listed here.
etcd 2.1 introduces a new metrics API endpoint that can be used for real-time monitoring and debugging. It exposes statistics about both client behaviors and resource usage. Like the auth API endpoint, this is an experimental feature which may be improved and changed based on user feedback.
We will continue to work to make etcd a fundamental building block for Google-like infrastructure that users can take off the shelf, build upon, and rely on. Get started with etcd, continue to share your feedback, and even help by contributing directly to the code.