DBLD is a central tool when it comes to syslog-ng development, but even after multiple blogs about the tool, it is still not much used outside of the developers’ team. So, what is DBLD and how could it be used even by you? The abbreviation stands for Docker BuiLD. Using containers ensures both that you have an easily reproducible build environment, and also that you do not have to “pollute” your base system with development-related software packages. You can use DBLD to build the release tarball or ready-to-use packages for a number of Linux distributions. It can even be used as a development environment with all necessary tools installed.
Still not convinced? Yes, the listed possibilities are mostly interesting for syslog-ng developers and 3rd party packagers. If you are lucky, you will never need DBLD. On the other hand, it can come handy if you reported a problem and the syslog-ng team fixed it. Even if you are not a developer, only a junior sysadmin, you can still easily build fixed syslog-ng packages for testing (and even for production use) until an official release with the fix arrives.
Before you begin
Using DBLD has very few external dependencies. You need Linux, Docker (or Podman & Buildah), Git, GNU make and an active Internet connection. The rest of the magic happens within containers. These tools are available on most modern Linux distributions, but sometimes you need to enable additional repositories before you can install them.
How does it work?
Before we go into details of possible functions, let me give you a quick overview of how DBLD works. To use it, you need to check out the syslog-ng source code from GitHub. As a first step, you either build or download container images. Even if you do not do that separately, it is done in the background. The default base operating system is Ubuntu Bionic, but for most operations you can use other operating systems as well. The advantage of containerization is that no matter which distribution you use for everyday work, you can access the very same environment that syslog-ng team members use as long as Docker is available.
For example, I use openSUSE on my laptop. It is not a tested or supported platform by syslog-ng developers, still DBLD works perfectly in my environment. And I can support syslog-ng users running Debian or Ubuntu, even if I do not have any Debian-based systems.
Once a container image is available locally, the rest of the operations will run in containers. Most functions of DBLD are available from the host command line, but you can also enter the containers. This is especially useful if you want to build syslog-ng manually instead of building a package automatically.
Note that container images use considerable disk space and they are not automatically deleted. This can save you gigabytes of download traffic when working actively with DBLD. After use, you can safely delete container images and rebuild or download them any time you need them again.
Using Podman instead of Docker
Some distributions (for example, RHEL / CentOS 8) removed Docker in favor of Podman completely. Others (like openSUSE and a number of other distributions) are on the way phasing out Docker support and already have Podman available. Luckily, you can also use Podman instead of Docker with DBLD. All you need to do is to replace “docker” with “podman” at a single location in dbld/rules. Search for a line containing “DOCKER=docker”. Note that in this case, generated file names will contain the word “dirty” to show that you use modified sources.
You do not have to change anything if your distro contains compatibility links from Docker to Podman.
Downloading syslog-ng sources
Before you can start using DBLD, you have to download it. It is part of the syslog-ng source code on GitHub. The release tarball does contain DBLD, but it does not work without Git sources. Plus, even if you can download syslog-ng Git sources as a zip file, it does not contain some of the information used by DBLD, so you really need to use Git for source code download. Luckily, it is an easy one-line command to check out the latest source code:
git clone https://github.com/syslog-ng/syslog-ng
Now change to the syslog-ng directory and you are ready to go:
cd syslog-ng
If you need to build from an earlier commit instead of the latest commit on the master branch, you can change to that by looking up the commit ID on GitHub (or the output of “git log”). For example:
git checkout cb21a5623bf68f1034759890ffab530e8f56ac3c
Building container images
In most cases, this step is not needed. The DBLD scripts download the necessary image when you initiate an operation that needs a specific image. But it can come handy if the definition of a container is changed in DBLD and an updated image is not yet available on Docker Hub. When you start the DBLD script from the command line without any arguments, it will print usage information, including the list of supported operating systems. Start it from the root directory of the syslog-ng sources using:
dbld/rules
Now build a container image for one of your platforms. Most syslog-ng users run RHEL / CentOS 7, so that is what I use in my examples:
dbld/rules image-centos-7
Once the image is built, you can list the results:
# docker images REPOSITORY TAG IMAGE ID CREATED SIZE balabit/syslog-ng-centos-7 latest 4a32e99d32bf 14 minutes ago 1.49GB centos 7 5e35e350aded 4 months ago 203MB
Building the release tarball
The release tarball is the syslog-ng source code archive as most Linux distributions and BSD variants use it when building a syslog-ng package. It contains a subset of files from the source code on GitHub, but also includes additional files (for example, configure or man pages built from XML sources). While creating a release tarball should work on most distributions (for example, on CentOS 7), the recommended way is to use Ubuntu Bionic, the default base operating system. You can start to create a release tarball with the following command:
dbld/rules tarball
It will download the Ubuntu Bionic base image if it is not yet available and create a release tarball. You will find the result under the dbld/build subdirectory. When you list the directory, you should look for a .tgz archive, for example: syslog-ng-3.26.1.188.gbe6ae18.tar.gz The name starts with syslog-ng, contains the latest release number, the number of commits since the latest release and the first eight characters of the Git hash.
Building a modified release tarball
When the syslog-ng team fixes a problem, they usually ask the reporter of a problem to test fixes before they are committed on GitHub. Before DBLD, it was lots of manual work and only some power users were able to do it. DBLD makes the process easily available for most people. If you use one of the supported distributions, it can even do package building for you. But even others can receive a fully prepared .tgz archive that they can use in their distributions’ package building tool. This is, for example, how I do test builds for FreeBSD ports.
Let’s create a release tarball containing an April fools’ day code submission. Once you checked out the syslog-ng source code from GitHub, change to the syslog-ng directory and download the patch from GitHub. The PR is available at https://github.com/syslog-ng/syslog-ng/pull/3216, and GitHub provides a nice little trick to download it as a patch. Just append .patch to the above URL: https://github.com/syslog-ng/syslog-ng/pull/3216.patch
Download it, apply it to the source code and build a new tarball:
-
wget https://github.com/syslog-ng/syslog-ng/pull/3216.patch
-
patch -p 1 < 3216.patch
-
dbld/rules tarball
When you list the dbld/build directory, you will find a slightly different file name: syslog-ng-3.26.1.188.gbe6ae18.dirty.tar.gz Here “dirty” means that the build scripts detected that the source code is modified compared to original Git sources.
Building packages for some DEB and RPM distributions
Using DBLD, you can build ready-to-install packages for a number of popular Linux distributions. As mentioned earlier, you can list the currently supported distros running the “dbld/rules” command without any further parameters. The target distro does not depend on what you are currently running (for example, I regularly build CentOS 7 packages on my laptop running openSUSE). And you can even build DEB packages on an RPM distro, or the other way around, too.
Starting a build is easy. You do not even have to execute the previous steps, as DBLD automagically downloads the missing container images, creates a release tarball and builds packages. Of course, if they are already there, DBLD can use the results of the previous steps. The exact parameter depends on the type of the distro for which you want to build. For RPM distros, you have to prefix the distro name with “rpm-”. For DEB distros, the prefix is “deb-”. Here are two examples:
-
dbld/rules rpm-centos-7
-
dbld/rules deb-ubuntu-bionic
You can find the resulting packages in sub directories of “dbld/build” named after the platform for which you compiled. The name of the packages reflect the source used, just as when you generate a tarball:
linux-yv1e:~/syslog-ng # ls dbld/build/ubuntu-bionic/ syslog-ng_3.26.1.188.gbe6ae18-snapshot+20200403T163009_all.deb syslog-ng-core_3.26.1.188.gbe6ae18-snapshot+20200403T163009_amd64.deb syslog-ng-dbg_3.26.1.188.gbe6ae18-snapshot+20200403T163009_amd64.deb syslog-ng-dev_3.26.1.188.gbe6ae18-snapshot+20200403T163009_amd64.deb syslog-ng-mod-add-contextual-data_3.26.1.188.gbe6ae18-snapshot+20200403T163009_amd64.deb syslog-ng-mod-amqp_3.26.1.188.gbe6ae18-snapshot+20200403T163009_amd64.deb [...] linux-yv1e:~/syslog-ng # ls dbld/build/centos-7/ syslog-ng-3.26.1.188.gbe6ae18-1+20200403T141634.el7.src.rpm syslog-ng-3.26.1.188.gbe6ae18-1+20200403T141634.el7.x86_64.rpm syslog-ng-afsnmp-3.26.1.188.gbe6ae18-1+20200403T141634.el7.x86_64.rpm syslog-ng-amqp-3.26.1.188.gbe6ae18-1+20200403T141634.el7.x86_64.rpm syslog-ng-debuginfo-3.26.1.188.gbe6ae18-1+20200403T141634.el7.x86_64.rpm syslog-ng-devel-3.26.1.188.gbe6ae18-1+20200403T141634.el7.x86_64.rpm [...]
What is next?
These are some of the DBLD features even sysadmins can find useful when they want to test a bug fix or a new feature not yet officially released. DBLD also has some features targeting developers. You can experiment by yourself, based on the README in the syslog-ng source code, or wait for my blog next month focusing on the developer shell also available in the DBLD containers.
If you have questions or comments related to syslog-ng, do not hesitate to contact us. You can reach us by email or even chat with us. For a list of possibilities, check our GitHub page under the “Community” section at https://github.com/syslog-ng/syslog-ng. On Twitter, I am available as @PCzanik.