Skip to content

The #newinjessie game: automatic configuration upgrade and other stuff in Debian/Jessie

Here are my contribution for the #newinjessie game, i.e. what new stuff I’ve contributed to the Jessie release of Debian.

See you in 2 years for Stretch release

All the best

Improving creation of debian copyright file

Hello

In my opinion, creating and maintaining Debian copyright file is the most boring task required to create a Debian package. Unfortunately, this file is also one of the most important of a package: it specifies some legal aspect regarding the use of the software.

Debian copyright file is scrutinized by ftp masters gatekeepers when accepting a new package in Debian project: this file must accurately describe the copyright and licenses of all files of a source package, preferably using a specific syntax. (Kudos to the ftp-masters team: reading copyright files must be even more boring than writing them).

The content of the copyright file must reflect accurately the license of all files. This license is often specified in the comments of a source files. The licencecheck command is able to scan sources files and reports the copyright and licenses declared in there. But it does not summarize this information: a copyright line is generated for each file of a package.

licensecheck2dep5 (provided by cdbs package as /usr/lib/cdbs/licensecheck2dep5) does better: the output of licensecheck is consolidated and translated in Debian copyright format. The result is better, but must be heavily edited to be reviewable by ftp-masters team.

The new update subcommand of cme (available with libconfig-model-dpkg-perl 2.061 currently in experimental) goes further than licensecheck2deb:

  • copyright are coalesced when possible (i.e. 2001,2002,2003-2005 is changed to 2001-2005)
  • file entries same copyright owner and license are grouped, group of files may be represented with a wild card (‘*’)
  • license text is filled with actual text for the most popular licenses

For instance, here’s the (slightly edited) output of cme run for pan package starting without debian/copyright file:

$ cme update dpkg-copyright -quiet
Adding dummy license text for license public-domain 
for path pan/general/sorted-vector.h
Adding dummy license text for license BSD-2-clause 
for path pan/usenet-utils/MersenneTwister.h
$ cat debian/copyright
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/

Files: *
Copyright: 1994-2001, Frank Pilhofer. The author may
License: GPL-2+

Files: pan/*
Copyright: 2002-2007, Charles Kerr 
License: GPL-2

Files: pan/data/cert-store.cc
  pan/data/cert-store.h
Copyright: 2011, Heinrich Müller 
  2002-2006, Charles Kerr 
License: GPL-2

Files: pan/general/e-util.cc
  pan/general/e-util.h
Copyright: 2000, 2001, Ximian, Inc
License: LGPL-2

Files: pan/general/locking.h
  pan/general/worker-pool.cc
  pan/general/worker-pool.h
Copyright: 2007, Calin Culianu 
  2002-2007, Charles Kerr 
License: LGPL-2+

Files: pan/general/sorted-vector.h
Copyright: 2002, Martin Holzherr (holzherr@infobrain.com).
License: public-domain
 Please fill license public-domain from header
 of pan/general/sorted-vector.h

[ about 100 more lines including license text for Zlib and several 
  GPL licenses ]

This is a good start, but some modifications must be applied to get a correct license file:

  • add missing upstream information (Upstream-Name, Upstream-Contact and Source items)
  • remove irrelevant text from some copyright owner (e.g. remove “The author may” from Files: * entry).
  • add some missing license text (e.g. include text from sorted-vector.h comments to specify upstream author’s version of public-domain)

These modifications can be done:

  • directly after update by running cme update dpkg-copyright --edit
  • after update by running cme edit dpkg-copyright
  • with your favorite editor

This post has mentioned creation of Debian copyright file, but does not address the problem of updating an existing copyright file when packaging a new version of a software. This will be the subject of a next post.

I hope this new feature of cme will save hours of work for Debian packagers. As usual comments and suggestions are welcome

All the best

Performance improvement for ‘cme check dpkg’

Hello

Thanks to Devel::NYTProf, I’ve realized that Module::CoreList was used in a not optimal way (to say the least) in Config::Model::Dpkg::Dependency when checking the dependency between Perl packages. (Note that only Perl packages with many dependencies were affected by this lack of performance)

After a rework, the performance are much better. Here’s an example comparing check time before and after the modification of libconfig-model-dpkg-perl.

With libconfig-model-dpkg-perl 2.059:
$ time cme check dpkg
Using Dpkg
loading data
Reading package lists... Done
Building dependency tree
Reading state information... Done
checking data
check done

real 0m10.235s
user 0m10.136s
sys 0m0.088s

With libconfig-model-dpkg-perl 2.060:
$ time cme check dpkg
Using Dpkg
loading data
Reading package lists... Done
Building dependency tree
Reading state information... Done
checking data
check done

real 0m1.565s
user 0m1.468s
sys 0m0.092s

 

All in all, a 8x performance improvement on the dependency check.

Note that, due to the freeze, the new version of libconfig-model-dpkg-perl is available only in experimental.

All the best

Looking for help to package Perl6, moar and others for Debian

Let’s face reality: I cannot find the time to properly maintain Perl6 related packages for Debian. Given the recent surge of popularity of rakudo, it would be a shame to let these packages rot.

Instead of throwing the towel, I’d rather call for help to maintain these packages. You don’t need to be a Debian Developer or Maintainer: I will gladly review and upload packages.

The following packages are looking for maintainer:

  • rakudo (currently RC buggy)
  • moar (needs to be packaged, some work has been done by Daniel Dehennin)
  • parrot (up to date)
  • nqp (need to be updated. current version no longer compiles on all arch)

Next step to help Perl6 on Debian is to join:

All the best

 

Status and next step on lcdproc automatic configuration upgrade with Perl and Config::Model

Back in March, I uploaded a Debian’s version of lcdproc with a unique feature: user and maintainer configurations are merged during package upgrade: user customizations and developers enhancements are both preserved in the new configuration file. (See this blog for more details). This avoids tedious edition of the configuration LCDd.conf file after every upgrade of lcdproc package.

At the beginning of June, a new version of lcdproc (0.5.7-1) was uploaded. This triggered another round of automatic upgrades on user’s systems.

According to the popcon rise of libconfig-model-lcdproc-perl, about 100 people have upgraded lcdproc on their system. Since automatic upgrade has an opt-out feature, one cannot say for sure that 100 people are actually using automatic upgrade, but I bet a fair portion are them are.

So far, only one people has complained: a bug report was filed about the many dependencies brought by libconfig-model-lcdproc-perl.

The next challenge for lcdproc configuration upgrade is brought by a bug reported on Ubuntu: the device file provided by imon kernel module is a moving target: The device file created by the kernel can be /dev/lcd0 or /dev/lcd1 or even /dev/lcd2. Static configuration files and moving target don’t mix well.

The obvious solution is to provide a udev rule so that a symbolic link is created from a fixed location (/dev/lcd-imon) to the moving target. Once the udev rule is installed, the user only has to update LCDd.conf file to use the symlink as imon device file and we’re done.

But, wait… The whole point of automatic configuration upgrade is to spare the user this kind of trouble: the upgrade must be completely automatic.

Moreover, the upgrade must work in all cases: whether udev is available (Linux) or not. If udev is not available, the value present in the configuration file must be preserved.

To know whether udev is available, the upgrade tool (aka cme) will check whether the file provided by udev (/dev/lcd-imon) is present or not. This will be done by lcdproc postinst script (which is run automatically at the end of lcdproc upgrade). Which means that the new udev rule must also be
activated in the postinst script before the upgrade is done.

In other words, the next version of lcdproc (0.5.7-2) will:

  • Install a new udev rule to provide lcd-imon symbolic link
  • Activate this rule in lcdproc postinst script before upgrading the configuration (note to udev experts: yes, the udev rule is activated with “--action=change” option)
  • Upgrade the configuration by running “cme migrate” in lcdproc postinst script.

In the lcdproc configuration model installed by libconfig-model-lcdproc-perl, the “imon device” parameter is enhanced so that running cme check lcdproc or cme migrate lcdproc issues a warning if /dev/lcd-imon exists and if imon driver is not configured to use it.

This way, the next installation of lcdproc will deliver a fix for imon and cme will fix user’s configuration file without requiring user input.

The last point is admittedly bad marketing as users will not be aware of the magic performed by Config::Model… Oh well…

In the previous section, I’ve briefly mentioned that “imon_device” parameter is “enhanced” in lcdproc configuration model. If you’re not already bored, let’s lift the hood and see what kind of enhancements was added.

Let’s peek in lcdproc configuration file, LCDd.conf file which is used to generate lcdproc configuration model. You may remember that the formal description of all LCDd.conf parameters and their properties is generated from LCDd.conf to provide lcdproc configuration model. The comments in LCDd.conf follow a convention so that most properties of the parameters can be extracted from the comments. In the example below, the comments show that NewFirmware is a boolean value expressed as yes or no, the latter being the default :

# Set the firmware version (New means >= 2.0) [default: no; legal: yes, no]
NewFirmware=no

Back to the moving target. In LCDd.conf, imon device file parameter is declared this way:

# Select the output device to use
Device=/dev/lcd0

This means that device is a string where the default value is /dev/lcd0.

Which is wrong once the special udev rule provided with Debian packages is activated. With this rule, the default value must be /dev/lcd-imon.

To fix this problem, a special comment is added in the Debian version of LCDd.conf to tune further the properties of the device parameter:

# select the device to use
# {%
#   default~
#   compute
#     use_eval=1
#     formula="my $l = '/dev/lcd-imon'; -e $l ? $l : '/dev/lcd0';"
#     allow_override=1 -
#   warn_if:not_lcd_imon
#     code="my $l = '/dev/lcd-imon';defined $_ and -e $l and $_ ne $l ;"
#     msg="imon device does not use /dev/lcd-imon link."
#     fix="$_ = undef;"
#   warn_unless:found_device_file
#     code="defined $_ ? -e : 1"
#     msg="missing imon device file"
#     fix="$_ = undef;"
#   - %}
Device=/dev/lcd0

This special comment between “{%” and “%}” follows the syntax of Config::Model::Loader. A small configuration model is declared there to enhance the model generated from LCDd.conf file.

Here are the main parts:

  • default~ suppress the default value of the “device” parameter declared in the original LCDd.conf (i.e. “/dev/ldcd0“)
  • compute and the 3 lines below computes a default value for the device file. Since “use_eval” is true, the formula is evaluated as Perl code. This code will return /dev/lcd-imon if this file is found. Otherwise, /dev/lcd0 is returned. Hence, either /dev/lcd-imon or /dev/lcd0 will be used a as default value. allow_override=1 lets the user override this computed value
  • warn_if and the 3 lines below test the configured device file with the Perl instructions provided by the code parameter. There, the device value is available in the $_ variable. This code will return true if /dev/lcd-imon exists and if the configured device does not use it. This will trigger a warning that will show the specified message.
  • Similarly warn_unless and the 3 lines below warns the user if the configured device file is not found.

In both warn_unless and warn_if parts, the fix code snippet is run when by the command cme fix lcdproc and is used to “repair” the warning condition. In this case, the fix consists in resetting the device configuration value so the computed value above can be used.

cme fix lcdproc is triggered during package post install script installed by dh_cme_upgrade.

Come to think of it, generating a configuration model from a configuration file can probably be applied to other projects: for instance, php.ini and kdmrc are also shipped with detailed comments. May be I should make a more generic model generator from the example used to generate lcdproc model…

Well, I will do it if people show interest. Not in the form “yeah, that would be cool”, but in the form, “yes, I will use your work to generate a configuration model for project […]”. I’ll let you fill the blank ;-)

Edit your debian patch header for DEP-3 compliance with cme

While not required by Debian policy, the  patch tagging guidelines (aka DEP-3) recommends to add a structured header to source package patches.

Long story short, patches should begin with something like :

Description: tweak lcdproc config for debian
patch LCDd.conf to:
* use syslog instead of stderr to show message
* run LCDd as root (to read /dev/lcd* file)
.
The latter could be done better by tweaking udev rule
Author: dod
Applied-Upstream: NA

Making sure that the DEP-3 recommendation is respected may not be fun. Do not despair: once libconfig-model-dpkg-perl is installed on your system, you can check whether your patches respect the DEP3 recommendation with the following command:

$ cme check dpkg-patches
loading data
checking data
check done

You can also check individual patches with:

cme check dpkg-patch tweak-conf

Note that bash auto-completion is provided only from version 2.049 of libconfig-model-dpkg-perl package.

If editing the header of your patches with your favorite editor does not strike you as fun, you can also use cme graphical editor provided by libconfig-model-tkui-perl. Once this package is installed, you can run:

cme edit dpkg-patches

You will get this editor to update your patches:

-usr-bin-cme Dpkg::Patches_001

This editor will show you the available parameters and the associated documentation to let you provide relevant information.

Last but not least, patch header check is also performed when you check your whole package with cme check dpkg

All the best

Deprecating experience level and preset value in Config::Model

Hello

The next releases of Config::Model will deprecate 2 (mis)features:

  • Experience level: trying to filter out configuration parameter based on arbitrary experience is a bad idea. This may confuse user by hiding parameter without obvious way to show them back. (actually with the options menu on the graphical interface, but that’s too easy to miss)
  • Preset value: this feature provides a way to specify a default value at runtime. This is now provided with the configuration layer.

The behavior of ‘cme check’ will not change. People using ‘cme edit’ with the GUI will see many more configuration parameters.

All the best

Follow

Get every new post delivered to your Inbox.

Join 41 other followers