Skip to content

Major bug fix for cme update copyright command

Hello

Previous version of libconfig-model-dpkg-perl had 2 bugs related to copyright update command :

  • Too many directory paragraphs (like src/foo/*) were removed during update.
  • Some file paragraph were not merged, leading to needless paragraphs in debian/copyright file. This bug is less severe as no information is lost

Version 2.067 of libconfig-model-dpkg-perl fixes both issues. This version is available in unstable.

To use cme update dpkg-copyright command, the following packages are required:

All the best

Improving update of existing debian/copyright file

Hello

One of my last blog introduced the new cme update dpkg-copyright command and explained how to use this command to create a debian/copyright file from scratch.

We will see in this blog how cme update dpkg-copyright command can also be used if a debian/copyright file already exists.

During the lifetime of a package, creating a new debian/copyright copyright file should only happen once. When updating a package to a new upstream version, the content of the copyright file should be modified to reflect upstream changes by:

  • adding or removing file entries
  • updating © years or owners
  • updating licenses

cme update dpkg-copyright can be run with an existing debian/copyright file: the information extracted by licensecheck are merged in debian/copyright. Unfortunately, this merge may bring back the errors coming from licensecheck that you’ve cleaned up manually last time.

This last part is not optimal as the manual clean up must be repeated after cme update dpkg-copyright is run.

Here’s a way to avoid the last problem: cme update dpkg-copyright supports a way to modify the content of debian/copyright after the merge: the optional file debian/fix.scanned.copyright contains instructions to tweak the content of debian/copyright. (See Config::Model::Dpkg::Copyright man page for more details)

You may remember that Config::Model handles all configuration information as a tree. The information contained in the copyright file is also stored in a tree structure. debian/fix.scanned.copyright contains instructions to navigate this tree and modify the content of the tree.

Imagine that there’s a robot (or a monkey) in that configuration tree. Each instruction tells the robot either to move in the tree or to perform an operation.

The most useful instructions for debian/fix.scanned.copyright are:

  • ! to move to the root of the tree
  • = to assign a value , e.g: short_name=MIT or Copyright="2014, FooBar" (use double quotes when the value white space or other fancy characters).
  • .= to append to a value, e.g Copyright.=" Appended info."
  • =~ to apply a substitution with a Perl regexp, e.g. Copyright=~"/©\s*//g"
  • : to move a Files element, e.g. Files:"foo/baz.c"
  • :~ to remove a Files element, e.g. Files:~"bogus.c"

Otherwise:

  • Comments with # or // must begin at the beginning of the line
  • Instructions are separated by space(s) or new lines

Let’s see a commented example:

# go to the root of the tree (with '!')
# and remove a bogus Files entry (with ":~")
! Files:~"foo/bogus.c"

// go to a Files entry and append (operator '.=') some info to © owner
! Files:"foo/bar.h"
  Copyright.=" Appended info."

# go back to tree root and assign © info
! Files:"foo/baz.c" Copyright="2014, FooBar"

# add a global license and a file entry missed by licensecheck
# global license must be set before using it in Files entry
# note: MIT text license is provided by cme
! License=MIT
! Files:"3rdparty/missed.c" Copyright="2005 FooBar" License short_name=MIT

Other possible instructions for this file are described in Config::Model::Loader doc.

The number of instructions depends on the accuracy of licencecheck command. Hopefully, this number can be reduced by improving licensecheck.

Last but not least, the following packages are required:

You may want to install also libconfig-model-tkui-perl to update the copyright file with cme GUI.

In summary, the command cme updated dpkg-copyright associated with instructions stored in debian/fix.scanned.copyright enable Debian packagers to update with less effort debian/copyright file when updating a package.

All the best

The #newinjessie game: automatic configuration upgrade and other stuff in Debian/Jessie

Here are my contribution for the #newinjessie game, i.e. what new stuff I’ve contributed to the Jessie release of Debian.

See you in 2 years for Stretch release

All the best

Improving creation of debian copyright file

Hello

In my opinion, creating and maintaining Debian copyright file is the most boring task required to create a Debian package. Unfortunately, this file is also one of the most important of a package: it specifies some legal aspect regarding the use of the software.

Debian copyright file is scrutinized by ftp masters gatekeepers when accepting a new package in Debian project: this file must accurately describe the copyright and licenses of all files of a source package, preferably using a specific syntax. (Kudos to the ftp-masters team: reading copyright files must be even more boring than writing them).

The content of the copyright file must reflect accurately the license of all files. This license is often specified in the comments of a source files. The licencecheck command is able to scan sources files and reports the copyright and licenses declared in there. But it does not summarize this information: a copyright line is generated for each file of a package.

licensecheck2dep5 (provided by cdbs package as /usr/lib/cdbs/licensecheck2dep5) does better: the output of licensecheck is consolidated and translated in Debian copyright format. The result is better, but must be heavily edited to be reviewable by ftp-masters team.

The new update subcommand of cme (available with libconfig-model-dpkg-perl 2.061 currently in experimental) goes further than licensecheck2deb:

  • copyright are coalesced when possible (i.e. 2001,2002,2003-2005 is changed to 2001-2005)
  • file entries same copyright owner and license are grouped, group of files may be represented with a wild card (‘*’)
  • license text is filled with actual text for the most popular licenses

For instance, here’s the (slightly edited) output of cme run for pan package starting without debian/copyright file:

$ cme update dpkg-copyright -quiet
Adding dummy license text for license public-domain 
for path pan/general/sorted-vector.h
Adding dummy license text for license BSD-2-clause 
for path pan/usenet-utils/MersenneTwister.h
$ cat debian/copyright
Format: https://www.debian.org/doc/packaging-manuals/copyright-format/1.0/

Files: *
Copyright: 1994-2001, Frank Pilhofer. The author may
License: GPL-2+

Files: pan/*
Copyright: 2002-2007, Charles Kerr 
License: GPL-2

Files: pan/data/cert-store.cc
  pan/data/cert-store.h
Copyright: 2011, Heinrich Müller 
  2002-2006, Charles Kerr 
License: GPL-2

Files: pan/general/e-util.cc
  pan/general/e-util.h
Copyright: 2000, 2001, Ximian, Inc
License: LGPL-2

Files: pan/general/locking.h
  pan/general/worker-pool.cc
  pan/general/worker-pool.h
Copyright: 2007, Calin Culianu 
  2002-2007, Charles Kerr 
License: LGPL-2+

Files: pan/general/sorted-vector.h
Copyright: 2002, Martin Holzherr (holzherr@infobrain.com).
License: public-domain
 Please fill license public-domain from header
 of pan/general/sorted-vector.h

[ about 100 more lines including license text for Zlib and several 
  GPL licenses ]

This is a good start, but some modifications must be applied to get a correct license file:

  • add missing upstream information (Upstream-Name, Upstream-Contact and Source items)
  • remove irrelevant text from some copyright owner (e.g. remove “The author may” from Files: * entry).
  • add some missing license text (e.g. include text from sorted-vector.h comments to specify upstream author’s version of public-domain)

These modifications can be done:

  • directly after update by running cme update dpkg-copyright --edit
  • after update by running cme edit dpkg-copyright
  • with your favorite editor

This post has mentioned creation of Debian copyright file, but does not address the problem of updating an existing copyright file when packaging a new version of a software. This will be the subject of a next post.

I hope this new feature of cme will save hours of work for Debian packagers. As usual comments and suggestions are welcome

Last but not least, the following packages are required:

You may want to install also libconfig-model-tkui-perl to update the copyright file with cme GUI.

All the best

Performance improvement for ‘cme check dpkg’

Hello

Thanks to Devel::NYTProf, I’ve realized that Module::CoreList was used in a not optimal way (to say the least) in Config::Model::Dpkg::Dependency when checking the dependency between Perl packages. (Note that only Perl packages with many dependencies were affected by this lack of performance)

After a rework, the performance are much better. Here’s an example comparing check time before and after the modification of libconfig-model-dpkg-perl.

With libconfig-model-dpkg-perl 2.059:
$ time cme check dpkg
Using Dpkg
loading data
Reading package lists... Done
Building dependency tree
Reading state information... Done
checking data
check done

real 0m10.235s
user 0m10.136s
sys 0m0.088s

With libconfig-model-dpkg-perl 2.060:
$ time cme check dpkg
Using Dpkg
loading data
Reading package lists... Done
Building dependency tree
Reading state information... Done
checking data
check done

real 0m1.565s
user 0m1.468s
sys 0m0.092s

 

All in all, a 8x performance improvement on the dependency check.

Note that, due to the freeze, the new version of libconfig-model-dpkg-perl is available only in experimental.

All the best

Looking for help to package Perl6, moar and others for Debian

Let’s face reality: I cannot find the time to properly maintain Perl6 related packages for Debian. Given the recent surge of popularity of rakudo, it would be a shame to let these packages rot.

Instead of throwing the towel, I’d rather call for help to maintain these packages. You don’t need to be a Debian Developer or Maintainer: I will gladly review and upload packages.

The following packages are looking for maintainer:

  • rakudo (currently RC buggy)
  • moar (needs to be packaged, some work has been done by Daniel Dehennin)
  • parrot (up to date)
  • nqp (need to be updated. current version no longer compiles on all arch)

Next step to help Perl6 on Debian is to join:

All the best

 

Status and next step on lcdproc automatic configuration upgrade with Perl and Config::Model

Back in March, I uploaded a Debian’s version of lcdproc with a unique feature: user and maintainer configurations are merged during package upgrade: user customizations and developers enhancements are both preserved in the new configuration file. (See this blog for more details). This avoids tedious edition of the configuration LCDd.conf file after every upgrade of lcdproc package.

At the beginning of June, a new version of lcdproc (0.5.7-1) was uploaded. This triggered another round of automatic upgrades on user’s systems.

According to the popcon rise of libconfig-model-lcdproc-perl, about 100 people have upgraded lcdproc on their system. Since automatic upgrade has an opt-out feature, one cannot say for sure that 100 people are actually using automatic upgrade, but I bet a fair portion are them are.

So far, only one people has complained: a bug report was filed about the many dependencies brought by libconfig-model-lcdproc-perl.

The next challenge for lcdproc configuration upgrade is brought by a bug reported on Ubuntu: the device file provided by imon kernel module is a moving target: The device file created by the kernel can be /dev/lcd0 or /dev/lcd1 or even /dev/lcd2. Static configuration files and moving target don’t mix well.

The obvious solution is to provide a udev rule so that a symbolic link is created from a fixed location (/dev/lcd-imon) to the moving target. Once the udev rule is installed, the user only has to update LCDd.conf file to use the symlink as imon device file and we’re done.

But, wait… The whole point of automatic configuration upgrade is to spare the user this kind of trouble: the upgrade must be completely automatic.

Moreover, the upgrade must work in all cases: whether udev is available (Linux) or not. If udev is not available, the value present in the configuration file must be preserved.

To know whether udev is available, the upgrade tool (aka cme) will check whether the file provided by udev (/dev/lcd-imon) is present or not. This will be done by lcdproc postinst script (which is run automatically at the end of lcdproc upgrade). Which means that the new udev rule must also be
activated in the postinst script before the upgrade is done.

In other words, the next version of lcdproc (0.5.7-2) will:

  • Install a new udev rule to provide lcd-imon symbolic link
  • Activate this rule in lcdproc postinst script before upgrading the configuration (note to udev experts: yes, the udev rule is activated with “--action=change” option)
  • Upgrade the configuration by running “cme migrate” in lcdproc postinst script.

In the lcdproc configuration model installed by libconfig-model-lcdproc-perl, the “imon device” parameter is enhanced so that running cme check lcdproc or cme migrate lcdproc issues a warning if /dev/lcd-imon exists and if imon driver is not configured to use it.

This way, the next installation of lcdproc will deliver a fix for imon and cme will fix user’s configuration file without requiring user input.

The last point is admittedly bad marketing as users will not be aware of the magic performed by Config::Model… Oh well…

In the previous section, I’ve briefly mentioned that “imon_device” parameter is “enhanced” in lcdproc configuration model. If you’re not already bored, let’s lift the hood and see what kind of enhancements was added.

Let’s peek in lcdproc configuration file, LCDd.conf file which is used to generate lcdproc configuration model. You may remember that the formal description of all LCDd.conf parameters and their properties is generated from LCDd.conf to provide lcdproc configuration model. The comments in LCDd.conf follow a convention so that most properties of the parameters can be extracted from the comments. In the example below, the comments show that NewFirmware is a boolean value expressed as yes or no, the latter being the default :

# Set the firmware version (New means >= 2.0) [default: no; legal: yes, no]
NewFirmware=no

Back to the moving target. In LCDd.conf, imon device file parameter is declared this way:

# Select the output device to use
Device=/dev/lcd0

This means that device is a string where the default value is /dev/lcd0.

Which is wrong once the special udev rule provided with Debian packages is activated. With this rule, the default value must be /dev/lcd-imon.

To fix this problem, a special comment is added in the Debian version of LCDd.conf to tune further the properties of the device parameter:

# select the device to use
# {%
#   default~
#   compute
#     use_eval=1
#     formula="my $l = '/dev/lcd-imon'; -e $l ? $l : '/dev/lcd0';"
#     allow_override=1 -
#   warn_if:not_lcd_imon
#     code="my $l = '/dev/lcd-imon';defined $_ and -e $l and $_ ne $l ;"
#     msg="imon device does not use /dev/lcd-imon link."
#     fix="$_ = undef;"
#   warn_unless:found_device_file
#     code="defined $_ ? -e : 1"
#     msg="missing imon device file"
#     fix="$_ = undef;"
#   - %}
Device=/dev/lcd0

This special comment between “{%” and “%}” follows the syntax of Config::Model::Loader. A small configuration model is declared there to enhance the model generated from LCDd.conf file.

Here are the main parts:

  • default~ suppress the default value of the “device” parameter declared in the original LCDd.conf (i.e. “/dev/ldcd0“)
  • compute and the 3 lines below computes a default value for the device file. Since “use_eval” is true, the formula is evaluated as Perl code. This code will return /dev/lcd-imon if this file is found. Otherwise, /dev/lcd0 is returned. Hence, either /dev/lcd-imon or /dev/lcd0 will be used a as default value. allow_override=1 lets the user override this computed value
  • warn_if and the 3 lines below test the configured device file with the Perl instructions provided by the code parameter. There, the device value is available in the $_ variable. This code will return true if /dev/lcd-imon exists and if the configured device does not use it. This will trigger a warning that will show the specified message.
  • Similarly warn_unless and the 3 lines below warns the user if the configured device file is not found.

In both warn_unless and warn_if parts, the fix code snippet is run when by the command cme fix lcdproc and is used to “repair” the warning condition. In this case, the fix consists in resetting the device configuration value so the computed value above can be used.

cme fix lcdproc is triggered during package post install script installed by dh_cme_upgrade.

Come to think of it, generating a configuration model from a configuration file can probably be applied to other projects: for instance, php.ini and kdmrc are also shipped with detailed comments. May be I should make a more generic model generator from the example used to generate lcdproc model…

Well, I will do it if people show interest. Not in the form “yeah, that would be cool”, but in the form, “yes, I will use your work to generate a configuration model for project […]”. I’ll let you fill the blank ;-)

Follow

Get every new post delivered to your Inbox.

Join 45 other followers