I have been using pkgsrc framework since I have first came to know and use NetBSD, which was around 2005.
There is some good documentation available here https://www.netbsd.org/docs/pkgsrc/creating.html and here https://wiki.netbsd.org/pkgsrc/intro_to_packaging/ I wanted to share some real-life example of creating such custom package (I have chosen CFEngine-3.11.0 which is not present in pkgsrc-current nor the pkgsrc-Q1-2018, the latest version there is cfengine-3.7.3 )
CFEngine is an open source configuration management system, written by Mark Burgess. Its primary function is to provide automated configuration and maintenance of large-scale computer systems, including the unified management of servers, desktops, consumer and industrial devices, embedded networked devices, mobile smartphones, and tablet computers.
I have created the below example on the Debian 7.5 mipsel Ci20 developer board.
1.1) Pre-requisites
- – Bootstrapped pkgsrc environment on the Ci20 (used a 16 Gb SDCard)
- – Basic knowledge of pkgsrc environment
- – Some free time and patience
1.2) Preparing the tools for custom packages build
Make sore you build and install the following packages
# cd /usr/pkgsrc/pkgtools/url2pkg # /usr/pkg/bin/bmake install clean # cd /usr/pkgsrc/pkgtools/pkgdiff # /usr/pkg/bin/bmake install clean
Next we create a WIP (work in progress) directory within the pkgsrc directory structure
# mkdir -p /usr/pkgsrc/wip/cfengine
We have decided to create the latest cfengine package available which is 3.11.0, so we take a note of the url download link and go to the new build directory and call url2pkg to initiate the package preperation
# cd /usr/pkgsrc/wip/cfengine # /usr/pkg/bin/url2pkg https://cfengine-package-repos.s3.amazonaws.com/tarballs/cfengine-3.11.0.tar.gz
Once the script finishes it will open an editor and offer you to add customized options to the Makefile
I Save the edit and exit (vi :wq) you will get this message once the script finishes,
===> Overriding tools for cfengine-3.11.0 ===> Extracting for cfengine-3.11.0 url2pkg> Adjusting the Makefile. Remember to correct CATEGORIES, HOMEPAGE, COMMENT, and DESCR when you're done!
The final file structure of the skeleton cfengine pkgsrc looks as following:
root@mipsbox:/usr/pkgsrc/wip/cfengine# ls -al total 24 drwxr-xr-x 3 root root 4096 Jan 18 20:50 . drwxr-xr-x 10 root root 4096 Jan 18 20:47 .. -rw-r--r-- 1 root root 0 Jan 18 20:47 DESCR -rw-r--r-- 1 root root 392 Jan 18 20:50 Makefile -rw-r--r-- 1 root root 18 Jan 18 20:47 PLIST -rw-r--r-- 1 root root 367 Jan 18 20:49 distinfo drwxr-xr-x 11 root root 4096 Jan 18 20:50 work
I have edited the Makefile in this way
# $NetBSD$
DISTNAME= cfengine-3.11.0
CATEGORIES= wip
MASTER_SITES= https://cfengine-package-repos.s3.amazonaws.com/tarballs/
MAINTAINER= INSERT_YOUR_MAIL_ADDRESS_HERE
HOMEPAGE= https://cfengine.com/product/community/
COMMENT= Tool for automating system administration
LICENSE= gnu-gpl-v3
GNU_CONFIGURE= yes
CONFIGURE_ARGS+= --with-postgresql${BUILDLINK_PREFIX.postgresql}
USE_LIBTOOL= yes
.include "../../databases/postgresql92-client/buildlink3.mk"
.include "../../databases/lmdb/buildlink3.mk"
.include "../../devel/pcre/buildlink3.mk"
.include "../../security/openssl/buildlink3.mk"
.include "../../textproc/libxml2/buildlink3.mk"
.include "../../mk/pthread.buildlink3.mk"
.include "../../mk/bsd.pkg.mk"
Since cfengine-3.11.0 server build depends on either mysql or postgresql I have included the configure option to build it with postresql support (more advanced way is to provide options variables where one could choose the mysql or postgresql options, but Im not gonna cover this here ) as well as lmdb, pcre, openssl and libxml.
Next we will try and actually build the above via bmake and see if we get any errors that need to be addressed
# cd /usr/pkgsrc/wip/cfengine # /usr/pkg/bin/bmake
Since I have quite a huge number of packages already pre-built on my Ci20 I won’t have to wait for long to get all the dependent packages to compile but the make ends with the following error :
DONE: Configuration done. Run make/gmake to build CFEngine Community. => Modifying libtool scripts to use pkgsrc libtool => Modifying libtool scripts to use pkgsrc depcomp ===> Building for cfengine-3.11.0 bmake: "/disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile" line 1099: Variable/Value missing from "export" bmake: Fatal errors encountered -- cannot continue bmake: stopped in /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0 *** Error code 1 Stop. bmake[1]: stopped in /usr/pkgsrc/wip/cfengine *** Error code 1 Stop. bmake: stopped in /usr/pkgsrc/wip/cfengine
In cases like this you need to create some patches that would fix/patch the original code. The error above indicates that there is a wrongly parsed environment variable in the generated Makefile. We can check on line 1099 of the generated Makefile in the following directory
# cat /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile | head -n 1099 | tail -n 1 export TAR_OPTIONS
So bmake environment does not like the TAR_OPTIONS variable defined, lets search for this string only
# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile TAR_OPTIONS = --owner=0 --group=0 export TAR_OPTIONS
The above Makefile gets generated by the configure script, which in fact calls automake at a certain stage which generates a Makefile.in from Makefile.am
# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.in TAR_OPTIONS = --owner=0 --group=0 export TAR_OPTIONS # grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.am TAR_OPTIONS = --owner=0 --group=0 export TAR_OPTIONS
So a quick workaround would be to first try remove the unnecessary variable name (on 2 lines) And prepare a diff that can be used to create a custom patch file. We will be using the pkgvi program to edit the following original file like so
# /usr/pkg/bin/pkgvi /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am
Remove the 2 offending lines containing the TAR_OPTIONS variable and save (vi :wq)
pkgvi: File was modified. For a diff, type: pkgdiff "/disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.am"
To produce a diff run the above command
# pkgdiff "/disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am" $NetBSD$ --- /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am.orig 2017-08-03 15:28:40.000000000 +0000 +++ /disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am @@ -48,8 +48,6 @@ SUBDIRS = libcompat \ # Hide the buildsystem's username, at least with GNU tar. -TAR_OPTIONS = --owner=0 --group=0 -export TAR_OPTIONS EXTRA_DIST = ChangeLog INSTALL README.md LICENSE CFVERSION
The output should be saved to a patch file in the /usr/pkgsrc/wip/cfengine/patches directory
# mkdir -p /usr/pkgsrc/wip/cfengine/patches # /usr/pkg/bin/pkgdiff "/disk/pkgsrc/wip/cfengine/work/cfengine-3.11.0/Makefile.am" > /usr/pkgsrc/wip/cfengine/patches/patch-Makefile.am
Once the patch file called patch-Makefile.am is in place we need to regenerate SHA1 chacksums for the distinfo meta file like so:
# cd /usr/pkgsrc/wip/cfengine # /usr/pkg/bin/bmake distinfo
So lets try and build it one more time and check if the above patch fixes our “TAR” issue
# cd /usr/pkgsrc/wip/cfengine # rm -rf /usr/pkgsrc/wip/cfengine/work # bmake
Again the same error comes ! Magically we have the TAR_OPTIONS in Makefile.in but not in Makefile.am anymore.
# grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.in TAR_OPTIONS = --owner=0 --group=0 export TAR_OPTIONS # grep -r TAR_OPTIONS /disk/pkgsrc/wip/tt/work/cfengine-3.11.0/Makefile.am
I did not have time to investigate the automake voodoo, so we can use a little dirty hack to make sure we patch the generated Makefile.in once the configure script finishes. This is how I have done it – by adding the following line towards the end of the configure script and produced the below patch file
$NetBSD$
--- /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure.orig 2017-08-03 15:29:11.000000000 +0000
+++ /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure
@@ -24850,6 +24854,9 @@ if test -n "$ac_unrecognized_opts" && te
$as_echo "$as_me: WARNING: unrecognized options: $ac_unrecognized_opts" >&2;}
fi
+#Added a small hack here to fix the TAR_ issue in Makefile.in
+sed -i '/TAR_/d' Makefile.in
+sed -i '/TAR_/d' Makefile
{ $as_echo "$as_me:${as_lineno-$LINENO}: result: DONE: Configuration done. Run make/gmake to build CFEngine Community." >&5
$as_echo "DONE: Configuration done. Run make/gmake to build CFEngine Community." >&6; }
Save it as follows
# cd /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0 # /usr/pkg/bin/pkgvi /usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure pkgvi: File was modified. For a diff, type: pkgdiff "work/cfengine-3.11.0/configure" # /usr/pkg/bin/pkgdiff "/usr/pkgsrc/wip/cfengine/work/cfengine-3.11.0/configure" > /usr/pkgsrc/wip/cfengine/patches/patch-configure
Rebuild the SHA1 checksums again
# cd /usr/pkgsrc/wip/cfengine # /usr/pkg/bin/bmake distinfo
And run bmake again, takes approx 30 minutes to build on the Ci20, but this time we are successful !
<.....cut....> Making all in unit Making all in load Making all in acceptance Making all in 25_cf-execd CC cf-execd-rpl-functions.o CCLD cf-execd-test CC mock_package_manager.lo CCLD libmock_package_manager.la CCLD mock_package_manager CC no_fds.o CCLD no_fds CC xml_c14nize-xml-c14nize.o CCLD xml-c14nize root@mipsbox:/usr/pkgsrc/wip/cfengine#
Next we run a test install
# cd /usr/pkgsrc/wip/cfengine # /usr/pkg/bin/bmake stage-install CHECK_FILES=no
Check if the .destdir got populated, following should be visible
<----cut----> /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/translatepath.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/edit_replace_string.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/app_baseline.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/packagesmatching.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/mustache_comments.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/varnet.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/exec_in_sequence.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/process_signalling.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/sort.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/strcmp.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/classmatch.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/isipinsubnet.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/examples/rename.cf /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/share/doc/cfengine/README.md /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-promises /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-net /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-runagent /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-key /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-upgrade /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-agent /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-execd /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-serverd /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/cf-monitord /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/bin/rpmvercmp /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/plugins /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/ppkeys /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/modules /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/inputs /usr/pkgsrc/wip/cfengine/work/.destdir/usr/pkg/var/cfengine/outputs
Make sure you also update the /usr/pkgsrc/wip/cfengine/DESCR
Cfengine, or the "configuration engine" is a very high level language for building expert systems which administrate and configure large computer networks. Cfengine uses the idea of classes and a primitive form of intelligence to define and automate the configuration of large systems in the most economical way possible. Cfengine is designed to be a part of computer immune system. Cfengine 3 is operationally backwards compatible with Cfengine 2, but the language is not. Cfengine 3 is not a drop-in replacement for Cfengine 2.
Once this is done we populate the PLIST inventory
# cd /usr/pkgsrc/wip/cfengine # /usr/pkg/bin/bmake print-PLIST >PLIST
And finally run a proper install
# /usr/pkg/bin/bmake install
Congratulations, your first binary pkgsrc package will land in /usr/pkgsrc/packages/All/ and will get installed into /usr/pkg environment.
