The iSulad container engine has many advantages, such as light weight and fast speed. This article focuses on how fast the iSulad container engine is. To prove that iSulad is fast, I choose some other leading container engines as reference, for example, Docker, Podman of Red Hat, and CRI-O.

After setting the reference objects, we need to specify the comparison scope.

Test Scope #

A container engine can be used in the following modes:

• Client mode: This mode is commonly used in personal development, test, and some production scenarios.
• CRI mode for PaaS: In typical cloud computing scenarios, the container runtime interface (CRI) is used to invoke container engine capabilities and manage pod clusters.

To improve the accuracy of the test result, both the client mode and the CRI mode need to be covered.

Client Mode #

CRI-O does not support the client mode. Therefore, only the following three are tested in this mode.

• Docker
• Podman
• iSulad

CRI Mode #

The CRI needs to be tested using cri-tools.

In CRI mode, the following three are selected as the test objects:

• Docker
• CRI-O
• iSulad

Environment Setup #

Machine Environment #

x86 #

ARM #

Installing iSulad #

For details, see the official document.

$ isula version

Client:
  Version:	2.0.3
  Git commit:	3bb24761f07cc0ac399e1cb783053db8b33b263d
  Built:	2020-08-01T09:40:06.568848951+08:00

Server:
  Version:	2.0.3
  Git commit:	3bb24761f07cc0ac399e1cb783053db8b33b263d
  Built:	2020-08-01T09:40:06.568848951+08:00

OCI config:
  Version:	1.0.1
  Default file:	/etc/default/isulad/config.json

Installing cri-tools #

Use the unified client tool to perform the CRI test. Choose V1.15.0 for Kubernetes.

$ git clone https://github.com/kubernetes-sigs/cri-tools
$ cd cri-tools
$ git checkout v1.15.0
$ make
$ export PATH=$PATH:$GOPATH/bin

Installing Docker #

For details, see the official document.

$ docker version

Client:
 Version:           19.03.11
 API version:       1.40
 Go version:        go1.14.3
 Git commit:        42e35e6
 Built:             Sun Jun 7 21:16:58 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.11
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.14.3
  Git commit:       42e35e6
  Built:            Sun Jun 7 00:00:00 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.3.3
  GitCommit:        
 runc:
  Version:          1.0.0-rc10+dev
  GitCommit:        fbdbaf85ecbc0e077f336c03062710435607dbf1
 docker-init:
  Version:          0.18.0
  GitCommit:

Installing kubelet #

V1.15.0 is selected as the test version. Clone the source code at https://github.com/kubernetes/kubernetes.

Obtaining Source Code #

Download the source code.

$ cd $GOPATH/src/k8s.io
$ git clone https://github.com/kubernetes/kubernetes.git
$ cd kubernetes
$ git checkout v1.15.0
$ go mod tidy

Compiling kubelet #

$ make all WHAT=cmd/kubelet

Note:

• Kubernetes has requirements on the Go version. For example, Kubernetes V1.15.0 requires Go 1.12.
• The go mod tidy command can be used to download dependencies. If a repository fails to be authenticated, run the go get -v -insecure command.

Installing kubelet #

$ cp _output/bin/kubelet /usr/local/bin/kubelet
$ kubelet --version
  Kubernetes v1.15.0

Starting kubelet #

$ kubelet --network-plugin=cni --runtime-cgroups=/systemd/system.slice --kubelet-cgroups=/systemd/system.slice --cgroup-driver="systemd"  --fail-swap-on=false  -v 5 --enable-controller-attach-detach=false --experimental-dockershim

Note: cgroup is managed by systemd.

Installing CRI-O #

CRI-O v1.15.4 installed using dnf cannot run properly. Therefore, we need to install it by compiling source code.

$ dnf install glib-2.0 glibc-devel glibc-static container-common
$ git clone https://github.com/cri-o/cri-o.git
$ cd crio
$ make
$ make install
$ mkdir -p /etc/crio && cp crio.conf /etc/crio/

Installing Podman #

Use dnf for installation.

$ dnf install -y podman

$ podman --version
  podman version 2.0.3

Test Scheme #

This article focuses on the container lifecycle performance of the container engines. The test points are as follows:

• Performance of operations such as create, start, stop, rm, and run of a single container;
• Performance of concurrent create, start, stop, rm, and run operations of 100 containers;
• Performance of runp, stopp, and rmp operations of a single pod;
• Performance of run, stop, and rm operations of a single pod containing a single container;
• Performance of concurrent runp, stopp, and rmp operations of 100 pods;
• Performance of concurrent run, stop, and rm operations of 100 pods, each pod containing a single container;

Note: The linux field must be specified in the pod configuration. Otherwise, Docker creates a default NIC for the pod. As a result, the CNI plugin fails to be executed.

{
    "metadata": {
        "name": "nginx-sandbox",
        "namespace": "default",
        "attempt": 1,
        "uid": "hdishd83djaidwnduwk28bcsb"
    },
    // The linux field must be set.
    "linux": {
    }
}

Detailed Design #

Although a single test and a concurrent test are two test scenarios, a single test can be regarded as a special case of concurrent tests. When designing test cases, we can control the number of concurrent operations to distinguish the two scenarios. The following figure shows the detailed design:

Client Mode #

Test Result in the x86 Environment #

Operation performance comparison of a single container

Operation performance comparison of 100 containers

Test Result in the ARM Environment #

Operation performance comparison of a single container

Operation performance comparison of 100 containers

CRI Mode #

Test Result in the x86 Environment #

Operation performance comparison of a single pod

Operation performance comparison of a single pod containing a single container

Operation performance comparison of 100 pods

Operation performance comparison of 100 pods (each containing a single container)

Test Result in the ARM Environment #

Operation performance comparison of a single pod

Operation performance comparison of a single pod containing a single container

Operation performance comparison of 100 pods

Operation performance comparison of 100 pods (each containing a single container)

Summary #

According to the test result, iSulad excels in container lifecycle operations and concurrent operations. Especially on ARM, its concurrency performance is close to that on x86, whereas the performance of other container engines is not satisfactory, which deteriorates by more than 50%.

So, why does iSulad have such a big advantage? In my opinion, the main reasons are as follows:

• iSulad is written in C/C++, while Docker/Podman/CRI-O is written in Go. C/C++ has advantages in speed.
• Compared with Docker, iSulad has a simpler architecture and shorter call chain. Podman is serverless and lags behind in concurrent scenarios.
• In the container creation process, reduced lock granularity and container dependencies (for example, an image management module) improve concurrency performance.

Architecture Comparison #

The following figure shows the iSulad architecture design:

The following figure shows the architecture design of Docker:

This figure does not describe the containerd and runc process involved in Docker daemon, which can be roughly presented as follows:

In the architecture, the Docker container lifecycle process involves the following: RESTful communication from the client to Docker daemon, gRPC communication from daemon to containerd, and runc execution in a forked process. The iSulad process only involves gRPC communication from the client to the server and lxc-start execution in a forked process.

References #

• https://stackoverflow.com/questions/46726216/kubelet-fails-to-get-cgroup-stats-for-docker-and-kubelet-services
• https://github.com/cri-o/cri-o
• https://gitee.com/openeuler/iSulad/blob/master/docs/build_docs/guide/build_guide.md