Skip to content

Easier Lcdproc package upgrade with automatic configuration merge

Hello

This blog explains how next lcdproc package provide easier upgrader with automatic configuration merge.

Here’s the current situation: lcdproc is shipped with several configuration files, including /etc/LCDd.conf. This file is modified upstream at every lcdproc release to bring configuration for new lcdproc drivers. On the other hand, this file is always customized to suit the specific hardware of the user’s system. So upgrading a package will always lead to a conflict during upgrade. User will always be required to choose whether to use current version or upstream version.

Next version of libconfig-model-lcdproc-perl will propose user whether to perform automatic merge of the configuration: upstream change are taken into account while preserving user change.

The configuration upgrade shown is based on Config::Model can be applied to other package.

Current lcdproc situation

To function properly, lcdproc configuration must always be adapted to suit the user’s hardware. On the following upgrade, upstream configuration is often updated so user will often be shown this question:

Configuration file '/etc/LCDd.conf'
 ==> Modified (by you or by a script) since installation.
 ==> Package distributor has shipped an updated version.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** LCDd.conf (Y/I/N/O/D/Z) [default=N] ?

This question is asked in the middle of an upgrade and can be puzzling for an average user.

Next package with automatic merge

Starting from lcdproc 0.5.6, the configuration merge is handled automatically by the packaging script with the help of Config::Model::Lcdproc.

When lcdproc is upgraded to 0.5.6, the following changes are visible:
* lcdproc depends on libconfig-model-lcdproc-perl
* user is asked once by debconf whether to use automatic configuration upgrades or not.
* no further question are asked (no ucf style questions).

For instance, here’s an upgrade from lcdproc_0.5.5 to lcdproc_0.5.6:

$ sudo dpkg -i lcdproc_0.5.6-1_amd64.deb 
(Reading database ... 322757 files and directories currently installed.)
Preparing to unpack lcdproc_0.5.6-1_amd64.deb ...
Stopping LCDd: LCDd.
Unpacking lcdproc (0.5.6-1) over (0.5.5-3) ...
Setting up lcdproc (0.5.6-1) ...

Changes applied to lcdproc configuration:
- server ReportToSyslog: '' -> '1' # use standard value
update-rc.d: warning: start and stop actions are no longer supported; falling back to defaults
Starting LCDd: LCDd.
Processing triggers for man-db (2.6.6-1) ...

Note: the automatic upgrade currently applies only to LCDd.conf. The other configuration files of lcdproc are handled the usual way.

Other benefits

User will also be able to:
* check lcdproc configuration with sudo cme check lcdproc
* edit the configuration with a GUI (see Managing Lcdproc configuration with cme for more details)

Here’s a screenshot of the GUI:

GUI to edit lcdproc configuration

More information

* libconfig-model-lcdproc-perl package page. This package provides a configuration model for lcdproc.
* This blog explains how this model is generated from upstream LCDd.conf.
* How to adapt a package to perform configuration upgrade with Config::Model

Next steps

Automatic configuration merge can be applied to other packages. But my free time is already taken by the maintenance of Config::Model and the existing models, there’s no room for me to take over another package.

On the other hand, I will definitely help people who want to provide automatic configuration merge on their packages. Feel free to contact me on:
* config-model-user mailing list
* debian-perl mailing list (where Config::Model is often used to maintain debian package file with cme)
* #debian-perl IRC channel

All the best

Config::Model::OpenSsh now supports OpenSSH 6.4 configuration

Hello

This was long overdue.

I’ve released Config::Model::OpenSsh 1.232. This new release
supports a lot of new parameters that were added to OpenSSH 6.0 and later versions, like AllowAgentForwarding, AuthenticationMethods or AuthorizedKeysCommand.

This new version is also available in Debian.

Happy new year !

LWP::UserAgent https proxy now fixed in Debian

Hello

For more 10 years, opening a https connection throught a proxy was not possible
with LWP::UserAgent.

Thanks to Steffen Ullrich, this bug is now fixed in LWP::UserAgent and LWP::Protocol::https repositories.

In Debian, I’ve updated libwww-perl 6.05-2 and liblwp-protocol-https-perl 6.04-2 to include the same patches. This fix is now available in Debian unstable.

See my previous blog for more details on this story.

All the best

About LWP::UserAgent, https and proxy setup

Hello

The last few weeks, I’ve been banging my head to use uscan, Pithub and JIRA::Client::Automated behind a corporate firewall. They are all written in Perl and use LWP::UserAgent to fetch information from Internet. At $work, using a proxy is mandatory to connect to Internet. But LWP::UserAgent https connection does not work through a proxy. This bug was reported 10 years ago but is still not fixed.

Here’s my plan to fix (or work-around) this issue, at least for Debian.

Before going further, let’s step back to explain briefly what is https and how proxies work.

https is not a protocol by itself. https is plain http over a socket encrypted with SSL (aka TLS). To create a https connection, the agent must first setup the SSL socket with the server. Then the agent uses http protocol to communicate with the server. All the traffic is encrypted on agent side and decrypted on server side by the SSL layer.

When creating a connection through a proxy, things get a little more complicated. Plain http requests (like GET, POST and so on…) are sent to the proxy as if the proxy was the http server. Then, the proxy forwards the request to the actual server.

From what I’ve read, most proxy servers refuse to plainly forward encrypted data to a web server. First a negotiation to create a tunnel towards the web server must be done by sending a http CONNECT request to the proxy server. The encrypted socket is then set up between the user and the web server through the proxy. Once this encrypted tunnel is set up, the usual http communication can be done.

Let’s go back to LWP::UserAgent. To create a connection over SSL, LWP::UserAgent will use a SSL library to setup the socket. This SSL library can be IO::Socket::SSL or Net::SSL. Direct https connection with LWP::UserAgent works fine with either library.

That said, only IO::Socket::SSL is able to perform correctly the verification of the server name. Net::SSL does not check correctly SSL certificates. For more details, see https://github.com/libwww-perl/libwww-perl/pull/51 and https://bugzilla.redhat.com/show_bug.cgi?id=705044 .

When trying to setup a https connection through a proxy, LWP::UserAgent (<= 6.05) tries to use the proxy like a regular http proxy without going through the CONNECT phase. This does not work.

Thanks to Steffen Ullrich, LWP::UserAgent and LWP::Protocol::https are now fixed in github.

In Debian, libwww-perl 6.05-2 and liblwp-protocol-https-perl 6.04-2 contains the same patch to fix https_proxy and are now uploaded in Debian unstable.

Next step is to provide simple patches to let uscan, Pithub and JIRA::Client::Automated correctly connect through proxies without jumping through hoops.

All the best

[ Edited: I've removed some bad ideas from this blog about using Net::SSL ]

Released Perl ptkdb debugger to experimental

Hello

Following Gabor’s post, I’ve updated the Debian package for libdevel-ptkdb-perl. This new package is now available in experimental repository.

Why not in unstable ? This new version of ptkdb (or is it a fork ?) was heavily changed and needs to be tested more before being released to unstable.

I will use it during my regular Perl activities. Nevertheless, I would welcome feedback on this new version to help decide when to upload to unstable.

All the best

Seeking help to update OpenSSH configuration editor (Config::Model)

Hello

OpenSsh 6.0 has been out for more than a year and the OpenSSH configuration editor still lacks the new parameters introduced by this release (like AllowAgentForwarding).

Technically, the task is not difficult, but I lack time to address it: I’m swamped by real life job, maintenance of Config::Model, Debian packaging activities…

So I’m looking for volunteer(s) to help me on Config::Model. Updating OpenSSH model is a great way to start !

If you want to help, please:

  • say so in a comment of this post or on config-model-users at lists.sourceforge.net (to synchronize efforts)
  • Fork Config::Model::OpenSsh repository
  • Follow instructions in the README file to install dependencies
  • Read this wiki page that details how to upgrade OpenSSH model.
  • Update and test the model
  • Send a pull request (Note that I’ll also accept patches sent by mail)

Be sure that you won’t be forgotten in the change log ;-)

Feel free to contact config-model-users at lists.sourceforge.net to get help, exchange ideas, or to discuss how to handle deprecation or upgrade of OpenSSH parameters…

All the best

update: OpenSSH project name was corrected after Rafal’s comment.

Improving error messages of Dpkg dependency parser

Hello

Config::Model::Dpkg project (a Debian source package model based on Config::Model) is partly based on a ParseRec::Descent grammar. This grammar is used to parse the dependency of a Debian source package.

This article will show how such a grammar is written, its limitation regarding error handling and how to improve the situation.

Debian package main data is described in debian/control file. This file can feature a list of dependencies, i.e. a list of package that must be installed for the package to work. These dependencies are declared in fields like “Build-Depends”, or “Depends” as a list of package. For Dpkg model purpose, I needed only to parse one item of a dependency list at a time.

This dependency item can be a simple package name:

foo

or a package name with a version requirement:

foo ( > 1.24 )

or a package name with architectures restrictions:

foo [alpha amd64 hurd-arm linux-armeb]

or both:

foo ( > 1.24 ) [alpha amd64 hurd-arm linux-armeb]

or a list of alternate choices combining the possibilities above:

foo ( > 1.24 ) | bar [ linux-any] | baz ( << 3.14 ) [ ! hurd-armel !hurd-armeb ]

or a variable that is replaced during package build:

${perl-depends}

Writing a Parse::RecDescent grammar to parse this is relatively straightforward.

The first production handles alternate dependencies separated by ‘|’ and raises an error if some text was not “consumed” by the dependencies:

dependency_item: depend(s /\|/) eofile |

A dependency as explained above is expressed as:

depend: pkg_dep | variable

A variable like ${foo} or ${bar}-1.24~ is parsed with:

variable: /\${[\w:\-]+}[\w\.\-~+]*/

This rule handles a package name with optional version or arch restriction:

pkg_dep: pkg_name dep_version(?) arch_restriction(?) 
pkg_name: /[a-z0-9][a-z0-9\+\-\.]+/

The remaining rules are quite simple:

dep_version: '(' oper version ')' 
oper: '<<' | '<=' | '=' | '>=' | '>>'
version: variable | /[\w\.\-~:+]+/

arch_restriction: '[' arch(s) ']'
arch:  /!?[\w-]+/

eofile: /^\Z/

The grammar above works well to parse the dependency. You can test it with this small Perl script:

#!/usr/bin/perl
use strict;
use warnings;
use 5.010 ;
use Parse::RecDescent ;

my $parser = Parse::RecDescent->new(join('',));
my $dep = shift ;
say "parsing '$dep'";
my $ret = $parser->dependency_item($dep) ;

say "result is ", $ret if $ref ;

__DATA__
# insert grammar here !!!

Unfortunately, any error in the optional parts (i.e version requirements and arch restriction) leads to an error message which is not very helpful. The error message only mention that some text could not be parsed:

parsing 'foo ( != 1.24 ) | bar'

       ERROR (line 1): Invalid dependency item: Was expecting /\|/ but found
                       "( != 1.24 ) | bar" instead

or

parsing 'foo [ arm & armel] | bar'

       ERROR (line 1): Invalid dependency item: Was expecting /\|/ but found
                       "[ arm & armel] | bar" instead

The problem comes from the fact that version requirements or arch restrictions are optional. For instance if a version requirement has a syntax error, Parse::RecDescent will try to parse it as an arch restriction. This arch restriction rule will also fail and the last terminal (“eofile”) will fail. So the error message does not hint at the actual syntax problem.

To generate better error messages, I improved the suggestion made in Parse::RecDescent FAQ.

Instead of calling a plain subroutine, I use a sub reference that will store the error messages in a closure. This sub ref is declared in a start-up action. Note that the sub ref explicitly returns undef. I’ll explain why later.

{
    my @dep_errors ;
    my $add_error = sub {
        my ($err, $txt) = @_ ;
        push @dep_errors, "$err: '$txt'" ;
        return ;
    } ;
}

The following production always fails while ensuring that the error list is reset. This production is always run at the beginning of the dependency parsing:

dependency: { @dep_errors = (); }

Here’s the actual “dependency” production that is run when “dependency” method is called on the parser. It will return an array ref containing (1, data) if the dependency is valid or (0, errors) otherwise:

dependency: depend(s /\|/) eofile
  {
    $return = [ 1 , @{$item[1]} ] ;
  }
  |
  {
    push( @dep_errors, "Cannot parse: '$text'" ) unless @dep_errors ;
    $return =  [ 0, @dep_errors ];
  }

The following productions don’t change much:

depend: pkg_dep | variable
variable: /\${[\w:\-]+}[\w\.\-~+]*/
pkg_dep: pkg_name dep_version(?) arch_restriction(?) 
dep_version: '(' oper version ')'

The first rule of this production parses the package name which must be followed by a space, end of string ‘(‘ or ‘['. A positive look-ahead assertion is used so only the package name is consumed. If the first rule fails, the second rule provides a meaningful error message. The second rule will match anything which is not a space and create an error message. Since $add_error returns undef, the second rule returns undef and the production fails. So the text stored in the error message is not consumed:

pkg_name: /[a-z0-9][a-z0-9\+\-\.]+(?=\s|\Z|\(|\[)/
    | /\S+/ { $add_error->("bad package name", $item[1]) ;}

The same trick is used with these productions:

oper: '<<' | '=' | '>>'
    | /\S+/ { $add_error->("bad dependency version operator", $item[1]) ;}

version: variable | /[\w\.\-~:+]+(?=\s|\)|\Z)/
    | /\S+/ { $add_error->("bad dependency version", $item[1]) ;}

The action of this production is a little bit more tricky. The action ensures that '!' are either added before all arch or not at all. Otherwise an error message is generated and added to the list of errors:

arch_restriction: '[' osarch(s) ']'
    {
        my $mismatch = 0;
        # $ref contains ['!',os,arch] or ['',os,arch]
        my $ref = $item[2] ;
        for (my $i = 0; $i < $#$ref -1 ; $i++ ) {
            $mismatch ||= ($ref->[$i][0] xor $ref->[$i+1][0]) ;
        }
        my @a = map { ($_->[0] || '') . ($_->[1] || '') . $_->[2] } @$ref ;
        if ($mismatch) {
            $add_error->("some names are prepended with '!' while others aren't.", "@a") ;
        }
        else {
            $return = 1 ;
        }
    }

The check above is possible only if the "osarch" production returns an array ref containing something like ('!','linux','any') for "!linux-any' or ('','linux','any') for "linux-any":

osarch: not(?) os(?) arch
    {
        $return =  [ $item[1][0], $item[2][0], $item[3] ];
    }
    | /.?(?=\s|\]|\Z)/ { $add_error->("bad arch specification: ", $item[1]) ; }

not: '!'

Here's the remaining of the grammar:

os: /(any|uclibc-linux|linux|kfreebsd|knetbsd|etc...)-/
   | /\w+/ '-' { $add_error->("bad os in architecture specification", $item[1]) ;}

arch: / (any |alpha|amd64 |arm\b |arm64 |etc... )
        (?=(\]| ))
      /x
      | /\w+/ { $add_error->("bad arch in architecture specification", $item[1]) ;}

eofile: /^\Z/

That's all for grammar 2.0...

Before someone yells: "Show me the message ! ", here are some example of bad dependencies and their error message generated by the parser:

parse 'foo ( != 1.24 ) | bar'
result is: 0 bad dependency version operator: '!='

parsing 'foo [ arm & armel] | bar'
result is: 0 bad arch specification: : '&'

parsing 'foo [ arm armel ] | bar [!moo]'
result is: 0 bad arch specification: : ']' bad arch in architecture specification: 'moo'

The 2 first error messages are spot on the actual error. The second one has a false positive (']' is correct) but correctly highlights the wrong arch name ('moo').

Mission accomplished.

In order to keep this post (relatively) simple, I've removed the part that actually store parsed data. They don't really matter for error handling. Nevertheless, you may see the whole grammar in Config::Model::Dpkg::Dependency module.

All the best

Follow

Get every new post delivered to your Inbox.