Background

For a local private cloud, I’ve always wanted to create a toolkit-level solution, similar to Vultr’s instance creation approach. Initially, I considered integrating a lightweight open-source framework based on KVM/QEMU, like OpenStack. However, given the trends in lightweight and technological iteration, I’ve been using Vagrantfile for a long time and wanted to try something different (see this discussion). OpenStack’s heaviness may not be suitable for small-scale scenarios, especially for my own test development environment.

I also considered whether to replace it with Proxmox VE (PVE). After reading some blog posts and considering my needs, I identified two main directions:

  1. KVM all-in-one with a configurable CLI
  2. LXD

Since I’ve deployed and maintained KVM before, I decided to explore LXD to compare the differences between the two. The deployment of LXD is straightforward, unlike Vagrant which requires setting up and installing many things. The concept of LXD seems inseparable from LXC. To understand their similarities and differences, according to the information, LXD is an enhanced version of LXC. LXD is primarily designed to build complete operating system containers. Therefore, compared to pure Docker images like Ubuntu or CentOS, LXD’s container images have better compatibility at the operating system level. Note that LXD container images and Docker images are two different types of things.

General Knowledge

Basic Concepts

The local test machine uses Ubuntu 20, and LXD 4.0.4 stable version is installed from the source, not the latest 5.0 version. According to the 4.0 version documentation, finding and installing images from the source is very easy. Here is a simple example:

root@agub20 ~ $ lxc remote ls
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| local (current) | unix://                                  | lxd           | file access | NO     | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+

# lxc image list images:  
# lxc image list images: | grep -i ubuntu  
# lxc image list images: | grep -i debian

## Deploy centos7 instance
lxc launch images:centos/7/amd64 test-lxc-centos7

Network Mapping Access

During the initial configuration (init), you need to select the network IP range for the LXD container. You can specify any network segment arbitrarily. For example, using the 192.168.61.1/24 network segment.

## Format: lxc config device <container name> <config name> proxy <listen option> <connect option>

## Temporarily expose container ssh port 22 to host port 10000 for external server access
## lxc config device add test proxy_ssh proxy listen=tcp:0.0.0.0:10000 connect=tcp:0.0.0.0:22

## Map local LXD container test-lxc-centos7 port 22 to host 10.66.253.248 port 10000, and name this mapping rule proxyssh-test-lxc-centos7
lxc config device add test-lxc-centos7 proxyssh-test-lxc-centos7 proxy listen=tcp:10.66.253.248:10000 connect=tcp:192.168.61.119:22

## Further restrictions can be applied, e.g., only allow SSH access to the container via the current host as a jump host
lxc config device add test-lxc-centos7 proxyssh-test-lxc-centos7 proxy listen=tcp:10.66.253.248:10000 connect=tcp:192.168.61.119:22 bind=host

## If you change the IP of the LXD container instance and want to update the configuration, refer to the following commands:
lxc config device list test-lxc-centos7-1   ## Get the current instance's proxy name: proxyssh-test-lxc-centos7
lxc config device set test-lxc-centos7-1 proxyssh-test-lxc-centos7 listen=tcp:10.66.253.248:10000 connect=tcp:192.168.61.127:22

## To change IP while in RUNNING state, refer to:
lxc config device override test-lxc-centos7 eth0 ipv4.address=192.168.61.123
lxc restart test-lxc-centos7
lxc ls

Additionally, LXD containers can also be configured to use the host’s DHCP server, eliminating the need to consider mapping issues.

Container Instance Packaging

## Example: package a virtual container named test-lxc-centos7-1, add --optimized-storage parameter for data compression.
lxc export test-lxc-centos7-1 test-lxc-centos7.tar.gz --optimized-storage
lxc import test-lxc-centos7.tar.gz
lxc start test-lxc-centos7-1

## Directly clone test-lxc-centos7-1 on the host to generate test-lxc-centos7-2, and set IP for the new copy
lxc copy test-lxc-centos7-1 test-lxc-centos7-2 -d eth0,ipv4.address=192.168.61.34

Container instance export actually has two methods: export and publish. Their difference,

After actual testing,

export export and import is a complete migration. After migration runs, when the replica starts, the IP remains the same as before. publish export and import: stored files, services, and packages are all present, but some configurations are missing, and the IP will be reassigned and generated. More subtle differences require further testing.

Of course, you can also publish the current container instance as an image for your own use. Example: submit the instance as an image with alias centos79_dk.

lxc publish test-lxc-centos7-1 --alias centos79_dk

Packaging private images

docker inside lxd

lxd launches a container service test-lxc-centos7-1. Inside this container, you can continue to install docker-ce. If you want to use docker inside lxd, you need to add the following configuration after loading the container service, otherwise there will be permission errors.

lxc config set test-lxc-centos7-1 security.nesting=true

kubernetes inside lxd also has many examples.

lxd sources

For better and more comfortable usage, lxd relies heavily on templates and customized private images. Official and third-party sources only provide standard container images. When using templates privately, you need to perform package customization via publish (commits) yourself.

root@agub20 ~ $ lxc remote ls
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
|      NAME       |                   URL                    |   PROTOCOL    |  AUTH TYPE  | PUBLIC | STATIC |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| images          | https://images.linuxcontainers.org       | simplestreams | none        | YES    | NO     |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| local (current) | unix://                                  | lxd           | file access | NO     | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu          | https://cloud-images.ubuntu.com/releases | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+
| ubuntu-daily    | https://cloud-images.ubuntu.com/daily    | simplestreams | none        | YES    | YES    |
+-----------------+------------------------------------------+---------------+-------------+--------+--------+

lxc image list images: |grep -i centos

## View information of local images
lxc image info 5f0f7a9589c6

## View information of remote repository images
lxc image info mirror-images:01a2eaafd432

lxc-images Tsinghua University Open Source Mirror Site lxc-images tencent Image server infrastructure - News - Linux Containers Forum

For example, if I want to find a container image for centos7.2 or centos7.6, it’s actually hard to find. The official centos7 image is updated to 7.9, and the centos7 image identifier does not specify the exact 7.x version. However, Ubuntu and other image sources have specific version tags. So when you need a specific centos 7.x or a domestic special image like Kylin, Anolis, etc., these are not available, and you need to customize them yourself.

Advantages

What are its advantages? Let me show you with a few examples.

Example 1: Multiple people use one GPU server. How to simply and conveniently allocate the resources each person needs while maintaining isolation?

Previously, one might consider an all-in-one Ubuntu18 host with KVM virtualization. Now, using lxd directly, deployment and allocation efficiency are much faster.

Example 2: Want to quickly virtualize a Linux machine on a server for temporary testing without manually cloning a machine in the cloud interface?

A few simple copy commands at the bottom can directly achieve this. Because lightweight, efficient, and convenient are its characteristics, no need for a bunch of pre-configuration; just a few installation commands and it’s ready to use. It’s truly an out-of-the-box product. If there is an lxc profile template configuration, it becomes even more convenient, similar to Vagrantfile. Based on the configuration file, you can quickly pull up a test machine with the specified configuration.

Disadvantages

Pay attention to the version of lxd, as there are differences in storage setup between version 4 and version 5. This is because it supports drive formats such as btrfs, zfs, lvm, dir, etc. Different disk types have different methods for setup, expansion, and configuration.

Inside an lxd container instance, you cannot see the accurate memory and disk size of the current container instance. Specifically, when you limit the disk size of a container instance, entering the container and running df -h will not show the limited size; it will still display the unlimited size. The same applies to memory. Resource limits are very convenient, but the inability to see the limited size synchronously inside the container is a drawback. Docker images can be easily converted to lxd container images for use, but the reverse is not possible.

Summary

When first encountering the concept of lxd, one often thinks of docker and containerd. In reality, there are some differences between the three, and their use cases are also different. For example, after dockershim became increasingly bloated and difficult to maintain in the Kubernetes community, k8s version 1.20 gradually dropped support for docker and switched to containerd. The maintenance and usage methods of container instances started with docker run and those started with lxd are also very different.

All three are containers, but the differences in specific scenarios create their distinctions. Just like Red Hat’s podman, it is also a container. However, in my usual business scenarios, it is almost never seen, so naturally it is not included in the current comparison list. Based on my limited usage time so far, lxd is very nice and convenient as a replacement for Vagrantfile for temporary test environments. Perhaps because it is too flexible, I am not confident enough to use it in production at the moment.