Containerd communication process using gRPC

Creating a container using Containerd ¶

The code below is an example of creating a container using Containerd. First, connect to Containerd using the UNIX domain socket containerd.sock. Containerd is a daemon process that always runs in the background. After that, through the Client object, Containerd gets the latest Debian Linux image and creates a Debian container. Very intuitive and simple, isn’t it? What you can see here is that it communicates with the process through the Client object. Although it cannot be confirmed in this short code, the communication process between Client and Container is summarized in this post.

func main() {
	// create a new client connected to the default socket path for containerd
	client, err := containerd.New("/run/containerd/containerd.sock")
	
	// create a new context with an "builder" namespace
	ctx := namespaces.WithNamespace(context.Background(), "builder")

	// pull the debian image from DockerHub
	image, err := client.Pull(ctx,
							  "docker.io/library/debian:latest",
							  containerd.WithPullUnpack,
							  containerd.WithPullSnapshotter(snapshotter))
	
	// create a debian container
	container, err := client.NewContainer(
		ctx,
		"debian_container",	// container name
		containerd.WithImage(image),
		containerd.WithSnapshotter(snapshotter),
		// snapshot name should unique across snapshotters, namespaces
		containerd.WithNewSnapshot("debian_snapshot", image), // snapshot name 
		containerd.WithNewSpec(oci.WithImageConfig(image)),
	)
}

Delegates on the client side Client ¶

Put basic information about the newly created container in containers.Container. This structure, defined in containers.go, includes several fields, but since it is before the container is created, it only contains information necessary for creation, i.e., what ID to identify a new container with, and what runtime to create. Here, runtime means things such as runc or gVisor, and the explanation can be found in the previous part. Anyway, if you pass this structure to the ContainerService().Create() method, a container is created.

The NewContainer() method is contained in the client.go file. The receiver of this method is a Client object. That is, a container can be created only when a Client object is created from the standpoint of embedding Containerd. to pkg.go.dev/github.com/containerd/containerd Some of the listed APIs are methods of this Client object, and methods with intuitive names such as GetImage() and LoadContainer() are included. I thought these APIs were from Containerd, which I had to call via gRPC, but they were actually from the Client. To put it simply, the Client is the client’s proxy and establishes a connection with Containerd using gRPC on behalf of the client. Therefore, the developer who implements the client does not need to worry about complex contents such as gRPC.

// NewContainer will create a new container with the provided id.
// The id must be unique within the namespace.
func (c *Client) NewContainer(ctx context.Context, id string, opts ...NewContainerOpts) (Container, error) {
	container := containers.Container{
		ID: id,
		Runtime: containers.RuntimeInfo{
			Name: c.runtime,
		},
	}
	r, err := c.ContainerService().Create(ctx, container)
	if err != nil {
		return nil, err
	}
	return containerFromRecord(c, r), nil
}

ContainerService() is likely to return a ContainerService object, but it returns RemoteContainerStore. Remote? Let’s put the puzzle together. The client will control the ContainerStore of Containerd located beyond gRPC through RemoteContainerStore. That is, both ends of the gRPC connection will be RemoteContainerStore and ContainerStore (actually ContainersService). Therefore, in the codes we will see from now on, gRPC and protobuf-related contents begin to appear. containersapi means containers.pb.go created by api/services/containers/v1/containers.proto, and these two files define the communication protocol between Client and Containerd only for ContainersService. A description of gRPC and protobuf will be omitted.

import (
	containersapi "github.com/containerd/containerd/api/services/containers/v1"
)

// ContainerService returns the underlying container Store
func (c *Client) ContainerService() containers.Store {
	if c.containerStore != nil {
		return c.containerStore
	}
	c.connMu.Lock()
	defer c.connMu.Unlock()
	return NewRemoteContainerStore(containersapi.NewContainersClient(c.conn))
}

Is RemoteContainerService just a shell? ¶

Contents related to remoteContainerStore are in containerstore.go, and you can see in comments that there is an actual containerStore beyond gRPC. There are methods such as Get(), List(), Create(), Update(), and Delete() in this file. As you can guess from the name, creation/deletion of containers, and acquisition/update of previously created container information carry out However, the actual work is only performed by containerStore of Containerd. It’s just a shell.

// NewRemoteContainerStore returns the container Store connected with the provided client
func NewRemoteContainerStore(client containersapi.ContainersClient) containers.Store {
	return &remoteContainers{
		client: client,
	}
}

Let’s just look at the Create() method. As it is a shell, there is no such thing as creating a container. However, it only calls the r.client.Create() method once more. All other methods are like this. r.client is the ContainersClient object of containers.pb.go seen above, and the Create() method of this object will call the Create() method of the ContainersServer object, which is a server-side object beyond gRPC. The contents of these files with the .pb.go extension are codes automatically generated by gRPC, so I will not explain them in this post because it is annoying.

func (r *remoteContainers) Create(ctx context.Context, container containers.Container) (containers.Container, error) {
	created, err := r.client.Create(ctx, &containersapi.CreateContainerRequest{
		Container: containerToProto(&container),
	})
	if err != nil {
		return containers.Container{}, errdefs.FromGRPC(err)
	}

	return containerFromProto(&created.Container), nil

}

Server-side delegate “containers” ¶

The create() method in the code below is a method of the ContainersServer object, which is a server-side object beyond gRPC. So how does gRPC map the create() method of ContainersClient, which is a client-side object, to here? The answer is the RegisterContainersServer() function. The developer of Containerd should implement the ContainersServer type object defined by gRPC and pass it as an argument. As a result, this function maps the method of the object developed by the developer to be called. Since gRPC is in charge of mapping related content, I will not explain it in this post because it is annoying. Anyway, this code is contained in a file called service.go, and the file implements a plug-in called “containers”. Details related to this will be described in the next paragraph.

func init() {
	plugin.Register(&plugin.Registration{
		Type: plugin.GRPCPlugin,
		ID:   "containers",
		Requires: []plugin.Type{
			plugin.ServicePlugin,
		},
		...
	...
}

var _ api.ContainersServer = &service{}

func (s *service) Register(server *grpc.Server) error {
	api.RegisterContainersServer(server, s)
	return nil
}

func (s *service) Create(ctx context.Context, req *api.CreateContainerRequest) (*api.CreateContainerResponse, error) {
	return s.local.Create(ctx, req)
}

The inner story ContainersService plugin finally revealed!!! ¶

The Create() method of the ContainersServer object that we looked at earlier seems to be able to create a container directly, but the Create() function of the local object is offloading the work. Why? Now, I would like to explain the unique part of Containerd. Remember that the above code is implemented in a file called service.go and the code below is implemented in a file called local.go. Let’s take a look at the following four things first.

1. If you look at the architecture of Containerd attached to the end of this post, you can see that it is composed of several Services. These services are all plug-ins, to be more precise, built-in plugins. If there is one thing that stands out, it will be “ContainersService”.

2. Somewhere in the upper part of the two files, there is a code to initialize the plugin by calling the plugin.Register() method along with the init() function.

3. service.go is the content of the service object, which is the end of gRPC, and implements the plug-in named “containers”.

4. local.go is the contents of the local object that creates the actual container, and implements a plugin called services.ContainersService (the string “containers-service” if traced).

In summary, the server object, the shell for gRPC, is contained in the service.go file, which implements the container plugin. On the other hand, the local object is contained in local.go, which is the body that actually works, such as creating an actual container, and implements the ContainersService plugin. This ContainersService plugin is identical to the one we saw in the architecture photo earlier. The container plug-in is simply a pass-through in charge of gRPC for ContainersService, so it doesn’t seem to appear in the picture. no need to come out

One question is, what if the file name of local.go was changed to service.go and service.go to service-grpc.go? This is because local.go implements the ContainersService plugin, but has nothing to do with service.go.

One more thing, the local object is implemented by dragging the ContainersClient type created by gRPC. Strictly speaking, the local object is located at the server-side end of gRPC. I don’t understand logically why ContainersServer was not dragged. Perhaps the server object is already using the ContainersServer type, so it is implemented like this to avoid confusion, but I feel the implementation is more confusing now. Anyway, it’s not important, but if you know the reason, please don’t pass it by and leave a comment.

func init() {
	plugin.Register(&plugin.Registration{
		Type: plugin.ServicePlugin,
		ID:   services.ContainersService,
		Requires: []plugin.Type{
			plugin.EventPlugin,
			plugin.MetadataPlugin,
		},
		...
	...
}

var _ api.ContainersClient = &local{}

func (l *local) Create(ctx context.Context, req *api.CreateContainerRequest, _ ...grpc.CallOption) (*api.CreateContainerResponse, error) {
	var resp api.CreateContainerResponse

	if err := l.withStoreUpdate(ctx, func(ctx context.Context) error {
		container := containerFromProto(&req.Container)

		created, err := l.Store.Create(ctx, container)
		if err != nil {
			return err
		}

		resp.Container = containerToProto(&created)

		return nil
	}); err != nil {
		return &resp, errdefs.ToGRPC(err)
	}

	return &resp, nil
}

Figure Summary: Containerd communication process using gRPC ¶

The communication process between the client and Containerd described so far is illustrated as follows.

첨부 : Containerd의 아키텍처 ¶