What name for Config::Model’s new command line ? cfg or something else ?

Hello

As Damyan mentioned last summer in Debconf, the current command interface to Config::Model Perl module is rather cumbersome as it involves quite a lot of typing. First, config-edit command name is too long. Even with the help bash completion, this does not compare well with git or hg command. Then all operation require options. For instance, the -application option is required to specify what you will be working on. A required option is a good example of an oxymoron. Which may be funny but which is not user friendly…

So, here’s the synopsis of the new command I’m working on:

# edit dpkg config with GUI (Tk or curses, or readline)
cfg edit dpkg 
  
# just check the validity of the file, show warnings
# and suggested fixes
# does not modify the file
cfg check dpkg
  
# check the file, remove deprecated parameters,
# migrate data to new parameters and save
cfg migrate dpkg 
  
# like migrate, but also apply all suggested fixes
cfg fix dpkg 

# like migrate and modify configuration 
# with command line argument
cfg modify dpkg source format="quilt (3.0)"
  
# access dpkg parameters through a fuse file
# system. data are saved when fs is unmounted
cfg fusefs dpkg fuse-dir

# dump configuration data in config-model format
cfg dump dpkg   

# list all available applications
cfg list 
  
# more general synopsis
cfg [ global_options ] command application [ options ] arguments

This synopsis use dpkg as an example, but it could be used for any of the available models (e.g. ssh, lcdproc, multistrap …).

The main question I have for you is: Is cfg name good ? I mean it’s short, descriptive enough. But it’s also quite ugly. Does anyone have a better idea ?

And what do you think of the synopsis shown above ? Are the sub-command name descriptive enough ?

All the best

Debian SDL packaging team revival

Hello

Most SDL packages are outdated, people are complaining in the BTS, and the team is mostly inactive.

That’s bad news for my project. What can I do ?

Well, it’s Debian project, I just have to step up.

Thanks to Sam Hocevar, I’m now admin on Debian Alioth SDL packaging project.

I’d like to revive the team. (Note that I don’t have the bandwidth to be the team all by myself).

Here are the first steps I plan (mostly inspired by the way Debian-perl team is working):

• create a SDL-team page on debian wiki (done)
• setup a SDL packaging git repo so that team members can collaborate more easily on packaging (on-going)
• call for volunteers to help. (this message is the first step)
• review and upload packages from team members

Now, there are some ground rules I wish to push forward:

• any team member can work on any package maintained by the team.
• just tell us on this list about any work you are doing (to avoid duplication)
• send a mail on this list if you need a package to be reviewed for upload.
• you can use SDL subversion repository to work on a package
• you can also use SDL git repo to work on a package (please discuss with us for the migration steps)
• once a package is migrated to git, it cannot go back to subversion.

If you don’t agree with the above points, now is the time to say it.

Feel free to join by:

1. Creating an account on Alioth
2. request to join sdl-team
3. Last but not least, send a short introduction mail to pkg-sdl-maintainers AT lists.alioth.debian.org so we can know each other.

See you soon

Debian NM process: it’s not bureaucracy, it’s …

Hello

Before entering Debian NM process (formerly New Maintainer process, now New Member process) , I’ve often read that it was too long, too hard, or too bureaucratic.

It’s certainly true that becoming a Debian Developer takes some time. Even if you’re already actively packaging. You have to answer a lot of questions regarding several aspects of Debian project. Some of them are not fun but necessary:

• Licensing is mostly boring but necessary: a mistake there may put the whole project in jeopardy. On the positive side, you also learn how to protect the software you write
• Debian social rules: Well, Debian project is inviting you to become a member. As they say “When in Rome, do like the Romans”. Learning Debian ways and customs will allow you to better cooperate and benefit from the project.
• Tools and skills: Even though I’ve been actively packaging for Debian-perl team for quite a while, I did have some knowledge holes. I had to do some research to answer questions. Especially regarding security practices and all the packaging tools available in Debian.

All in all, the NM process was time well spent. I’ve learned a lot.

IMHO, Debian NM process isn’t bureaucracy, it’s more like a training. And an interesting one.

So, if you’re thinking about becoming a Debian Member, to paraphrase someone, what the bloody hell are you waiting for ?

Many thanks to all people that helped me become a DD:

• gregoa whose attention to small details is uncanny
• xoswald who had to read all my answers and the bad jokes in there
• Enrico who reviewed the whole stuff and approved my application
• The debian-perl team for their warm welcome and helpful attitude

All the best

Config::Model: How to test configuration models

Hello

With Config::Model, configuration models are created to validate configuration data.

But, before that, how can we validate a model ? This has long been a sore spot of Config::Model. During its development, I’ve progressively enhanced a test script to tests the various models (and their corner cases) shipped with Config::Model. This script is able to test several models with several subtests. Currently, more that 400 unit tests are done on 5 models and about 50 configuration files.

Which is great, but usable only inside Config::Model distribution. Which is not so great …

So I’ve bitten the bullet and created a new Config::Model::Tester class from this test script. This new class is shipped with Config::Model 1.260 (also available on Debian/sid).

For instance, let’s see how this tester is used to test Xorg model (shipped in Config::Model::Xorg).

Xorg features a small test file (t/model_test.t):

use warnings;
use strict;
use Config::Model::Tester ;
use ExtUtils::testlib;

my $arg = shift || '';
my $test_only_model = shift || '';
my $do = shift ;

run_tests($arg, $test_only_model, $do) ;

The xorg files to test are specified by t/model_tests.d/xorg-test-conf.pl:

$conf_file_name = "xorg.conf" ;
$conf_dir = "etc/X11" ;
$model_to_test = "Xorg" ;

@tests = (
    { name => 'fglrx', },
    { name => 'modern', },
    { name => 'vesa', },
    { name => 'xorg', },
    { name => 'xorg-ati', },
);

1;

And the xorg.conf files are there:

$ ls t/model_tests.d/xorg-examples/
fglrx  modern  vesa  xorg  xorg-ati

Here’s the output of the xorg-ati subtest:

$ perl -Ilib t/model_tests.t x xorg xorg-ati
ok 1 - compiled
# Beginning xorg test (t/model_tests.d/xorg-test-conf.pl)
# xorg uses Xorg model on file xorg.conf
# Beginning subtest xorg xorg-ati
ok 2 - Copied xorg example xorg-ati
ok 3 - Read wr_root/test-xorg-ati/etc/X11/xorg.conf and created instance with init() method with warning check 
ok 4 - Ran dump_tree
ok 5 - Dumped xorg config tree in full mode
ok 6 - Dumped xorg config tree in custom mode
ok 7 - xorg write back done
ok 8 - Created instance xorg-test-xorg-ati-w
ok 9 - Dumped xorg 2nd config tree in custom mode
ok 10 - compare original xorg custom data with 2nd instance custom data
ok 11 - check that original xorg file was not clobbered
# End of subtest xorg xorg-ati
# End of xorg test
1..11

Now, it’s quite easy to create non regression tests for bug triggered by err, unexpected, configuration files: Just drop the file in the example directory and edit the *-test-conf.pl file. This saves a lot of typing.

All the best

Perl6 (aka rakudo) is available on Debian

Hello

I’m happy to announce that Perl6 (as rakudo) is now available on Debian unstable.

Thanks also to Gabor Szabo for the Perl6 talk he gave at FOSDEM.

His talk made me think: “Why the hell don’t we have rakudo on Debian ?”. So I stepped up, pushed to get a recent parrot package, ghedo and I created a pkg-rakudo team on Alioth, and we updated the old rakudo package done by ryan52.

Enjoy

I’m a Debian Developer now :-)

He he , after6 months of hard labor (just kidding, it wasn’t difficult) I’m now a brand new Debian Developer.

I plan to continue the work I did before, mainly Perl module packaging, helping getting a recent rakudo in Debian, and improving configuration tools (as upstream author).

I wish to thanks heartfully all the people that helped me before and during the new maintainer process:
gregoa, xoswald and the whole pkg-perl team.

Too bad I can’t go to Banja-Luka.

All the best

Generate a configuration editor from a config template file with Perl: lcdproc example

Hello

Creating a configuration model for Config::Model can be a boring task: When a configuration system has a lot of parameters, a lot of details need to be specified to have a complete configuration validation tool. So, all tricks are permitted to make this task more bearable.

Let’s see how this can be applied to lcdproc. This project requires a configuration file that supports more than 40 drivers. Each driver has it own set of parameters for a total of about 250 parameters. That’s a lot to specify..

Fortunately, LCDd.conf template file is written in a way which makes it relatively easy to parse and get all required information to build a model.

All drivers are listed in this template file, most parameters have default values and legal values are written in comments in a uniform way. Hence this file (and comments) can be parsed to retrieve information required for the model.

This blog will describe how this task can be implemented in Perl using Config::Model modules. The code will be slightly trimmed down to keep this blog (relatively) short. All initialisation code and error checks are removed. You can get the whole script on sourceforge.

The main steps are:

1. prepare a parser for LCDd.conf template
2. parse LCDd.conf template
3. prepare a LCDd model skeleton
4. translate LCDd.conf template information in a format suitable to create a model.
5. Write the resulting LCDd model

First, here’s the necessary incantation to start properly a Perl program:

use strict;
use warnings;

Then load all required libraries. Their use will be described later:

use Config::Model;
use Config::Model::Itself 1.225;
use Config::Model::Backend::IniFile;

use 5.10.0;
use IO::File;
use IO::String;

Perform some initialisation to avoid upsetting the logging system:

use Log::Log4perl qw(:easy);
Log::Log4perl->easy_init($WARN);

Let’s also initialize a model object that will be very useful later:

my $model = Config::Model->new();

Now we can really start the fun part.

Step 1: Create a parser for LCDd.conf (INI file)

Problem: LCDd.conf comments must also be analysed to get more semantic information. The comments must be retrieved and associated with the INI classes and parameters found in this file.

Fortunately, Config::Model::Backend::IniFile is able to associate comments with the relevant parameters. But Config::Model::Backend::IniFile must store its values in a configuration tree. So let’s create a model suitable for LCDd.conf template that accepts any INI class and any INI parameter

Dummy::Class is used to store any parameter found in an INI class. The accept keyword with .* regular expression means that any INI parameter will be accepted in each INI class.

$model->create_config_class(
  name   => 'Dummy::Class',
  accept => [ 
    '.*' => { 
       type => 'leaf',
       value_type => 'uniline'
       }, 
    ],
);

Now the Dummy model must provide a configuration class to hold all INI classes:

$model->create_config_class(
    name   => 'Dummy',
    accept => [ 
       '.*' => { 
           type => 'node', 
           config_class_name => 'Dummy::Class' 
       }, 
    ],
);

Note that an INI backend could be specified directly here. But, some useful parameters are commented out in LCD.conf. Some some processing is required to be able to create a model with these commented parameters. See below for this processing.

Step 2: Parse LCDd.conf

Now the dummy configuration class is created. Let’s create a configuration tree from this dummy class to store the data from LCDd.conf

my $dummy = $model->instance(
    instance_name   => 'dummy',
    root_class_name => 'Dummy',
)-> config_root;

Now, read LCDd.conf:

my $lcd_file 
  = IO::File->new('examples/lcdproc/LCDd.conf');
my @lines    = $lcd_file->getlines;

And perform the pre-processing on LCDd.conf mentioned above. Just un-commenting commented parameters is required:

foreach (@lines) { s/^#(\w+=)/$1/ }

Store the pre-processed LCDd.conf in a IO::Handle usable by INI backend

my $ioh = IO::String->new( join( '', @lines ) );

Create an INI backend to read the data from pre-processed LCDd.conf and store it in the configuration tree:

my $ini_backend = Config::Model::Backend::IniFile
  ->new( node => $dummy );
$ini_backend->read( io_handle => $ioh );

Step 3: Prepare LCDd configuration model

Your attention please. You may remember that we invoked instance on object to create the “dummy” configuration tree to hold LCDd.conf data. The instance method used the Dummy configuration class.

Now, we must invoke instance again on object. But this time, instance will use Config::Model::Itself. This model is Config::Model’s way of eating its own dog food. The configuration tree created there will be able to hold LCDd model. Hence the root of this special configuration tree will be called “meta_root” to avoid confusion with the other configuration root defined above (for the “Dummy” model).

Still with me?

Good, let’s create this “meta” tree that will contain LCDd model:

my $meta_root = $model->instance(
    root_class_name => 'Itself::Model',
    instance_name   => 'meta_model',
) -> config_root;

Create the main LCDd configuration class and store the first comment from LCDd.conf as class description:

my $first_comment = $dummy->annotation ;
$meta_root
  ->grab("class:LCDd class_description")
  ->store( $first_comment );

Append my own text in this description:

my $extra_description =
    "Model information extracted from template /etc/LCDd.conf"
  . "\n\n=head1 BUGS\n\nThis model does not support to load "
  . "several drivers. Loading several drivers is probably a "
  . "marginal case. Please complain to the author if this "
  . "assumption is false";
$meta_root->load(
     qq!class:LCDd 
        class_description.="\n\n$extra_description"!
);

Add legal stuff in the model:

$meta_root->load( qq!
  class:LCDd 
    copyright:0="2011, Dominique Dumont" 
    copyright:1="1999-2011, William Ferrell and others" 
    license="GPL-2"
!
);

Add INI backend so configuration object created from LCDd model will be able to read user’s INI files:

$meta_root->load( qq!
  class:LCDd 
    read_config:0 
      backend=ini_file 
      config_dir="/etc" 
      file="LCDd.conf"
!
);

Note that all the load calls above could be done as one call. They were split for better clarity.

Step 4:Extracting LCDd.conf information

Now that all information from LCDd.conf template file is stored in $dummy configuration tree, it’s time to use this information to enhance the skeleton of LCDd model created above.

Let’s first find all the INI classes defined in LCDd.conf:

my @ini_classes = $dummy->get_element_name;

Now before actually mining LCDd.conf information, we must prepare several subs to handle them. This is done using a dispatch table:

my %dispatch;

First create the default entry which will be used for most parameters. This subs is passed: the INI class name, the INI parameter name, the comment attached to the parameter, the INI value, and an optional value type. It will return a string describing a model for simple elements. For instance:

class:"LCDd::MD8800" element:Device type=leaf value_type=uniline default="/dev/ttyS1" description="device to use [default: /dev/ttyS1]"

Here’s the subroutine invoked by default. Let’s create a an anonymous subroutine and store it in the dispatch table:

$dispatch{_default_} = sub {
    my ( $ini_class, $ini_param, $ini_note, 
         $ini_v, $value_type ) = @_;
    $value_type ||= 'uniline';

Now, prepare a string that will be used later to create the ini_class model. This string uses Config::Model’s serialisation format described in Config::Model::Loader:

    my $load = qq!class:"$ini_class" 
                  element:$ini_param type=leaf !;

Get semantic information from LCDd.conf template comments (written between square brackets):

    my $info = $ini_note =~ /\[(.*)\]/ ? $1 : ''; 
    $info =~ s/\s+//g;

Use this semantic information to better specify the parameter:

    $load .=
        $info =~ /legal:(\d+)-(\d+)/ 
           ? " value_type=integer min=$1 max=$2"
      : $info =~ /legal:([\w\,]+)/ 
           ? " value_type=enum choice=$1"
      :    " value_type=$value_type";

    if ( $info =~ /default:(\w+)/ ) {
        # specify upstream default value 
        # if it was found in the comment
        $load .= qq! upstream_default="$1"! 
           if length($1);
    }
    else {
        # or use the value found in INI 
        # file as default
        $ini_v =~ s/^"//g;
        $ini_v =~ s/"$//g;
        $load .= qq! default="$ini_v"! 
           if length($ini_v);
    }

The model (in serialised format) is ready, let’s return it and exit the subroutine:

    
    return $load;
};

Now let’s take care of the special cases. This one deals with “Driver” parameter found in INI [server] class. This sub will create a string for an enum element. The possible values of the enum are extracted from the comment found in the LCDd.conf file:

$dispatch{"LCDd::server"}{Driver} = sub {
    my ( $class, $elt, $info, $ini_v ) = @_;
    my $load = qq!class:"$class" 
                    element:$elt 
                      type=leaf 
                      value_type=enum
    !;
    my @drivers = split /\W+/, $info;
    while ( @drivers 
            and ( shift @drivers ) !~ /supported/ ) {
      # nothing, the work is done by shift operator 
      # in loop test to skip all words in $info until
      # 'The following drivers are supported:'.
      # ok, that's a hack
    }
    $load .= 'choice=' 
          . join( ',', @drivers ) . ' ';

    return $load;
};

This sub invokes the default sub and add a small feature to the model to ensure that DriverPath ends with ‘/’. This is ensured by the match="/$" snippet:

$dispatch{"LCDd::server"}{DriverPath} = sub {
    my ( $class, $elt, $info, $ini_v ) = @_;
    return   $dispatch{_default_}->(@_) 
           . q! match="/$" !;
};

Likewise, this sub invokes the default sub with the optional 4th parameter that forces the value_type of the element. In this case, we enforce an integer element:

 $dispatch{"LCDd::server"}{WaitTime} 
 = $dispatch{"LCDd::server"}{ReportLevel} 
 = $dispatch{"LCDd::server"}{Port}  
 = sub {
    my ( $class, $elt, $info, $ini_v ) = @_;
    return $dispatch{_default_}->( @_, 'integer' );
  };

This sub returns a string for a list element with default values:

  $dispatch{"LCDd::server"}{GoodBye} 
= $dispatch{"LCDd::server"}{Hello} 
= sub {
    my ( $class, $elt, $info, $ini_v ) = @_;
    my $ret = qq( class:"$class" 
                    element:$elt 
                      type=list
                ) ;
    $ret .= 'cargo type=leaf 
             value_type=uniline - ' ;  
    $ret .= 'default_with_init:0="\" '.$elt.'\"" ' ; 
    $ret .= 'default_with_init:1="\" LCDproc!\""'; 
    return $ret ;
};

Now that the dispatch table is ready, let’s really mine LCDd.conf information.

First handle all INI classes in the foreach loop:

foreach my $ini_class (@ini_classes) {

Let’s retrieve parameters for this INI class from the LCDd.conf template:

  my $ini_obj = $dummy->grab($ini_class);

This will be the name of the lcdproc driver configuration class (e.g. LCDd::imonlcd):

  my $config_class   = "LCDd::$ini_class";

The configuration class will also be created in a serialised format. Let’s create config class in case there’s no parameter in INI file:

  $meta_root->load(qq!class:"LCDd::$ini_class"!);

Now, loop over all INI parameters and create LCDd::$ini_class elements:

  my @all_ini_params = $ini_obj->get_element_name ;  
  foreach my $ini_param ( @all_ini_params ) {

Retrieve the value provided in template LCDd.conf:

    my $ini_v = $ini_obj->grab_value($ini_param);

Retrieve the comment attached to the value provided in template LCDd.conf:

    my $ini_comment 
      = $ini_obj->grab($ini_param)->annotation;
    # escape embedded quotes
    $ini_comment =~ s/"/\\"/g;

Retrieve the correct sub from dispatch table:

    my $sub = $dispatch{$config_class}{$ini_param}
            || $dispatch{_default_};

Run the sub to get the model string:

    my $model_spec 
      = $sub->( $config_class, $ini_param, 
                   $ini_comment, $ini_v );

Attach the comment from LCDd.conf as class description:

    $model_spec .= qq! description="$ini_comment"!
      if length($ini_comment);

Then, last but not least, load class specification in the model, and end the loop aroung INI parameters:

    $meta_root->load($model_spec);
  }

Now create a an $ini_class element in LCDd class (to link LCDd class and LCDd::$ini_class):

  my $driver_class_spec = qq!
    class:LCDd 
      element:$ini_class 
  ! ;

Since server and menu are not drivers, they have a special treatment:

  if (   $ini_class eq 'server' 
      or $ini_class eq 'menu' ) {
    $driver_class_spec .= qq! 
      type=node 
      config_class_name="LCDd::$ini_class" 
    ! ;
  }

Here’s the treatment for LCDd drivers. Setup the model with a warped node so that a driver configuration class is shown only if the driver was selected in the [server] class:

  else {
    $driver_class_spec .= qq! 
      type=warped_node 
      config_class_name="LCDd::$ini_class"
      level=hidden
      follow:selected="- server Driver"
      rules:"\$selected eq '$ini_class'" 
        level=normal
    !;
  }

Now we can load the LCDd driver model in meta root:

  $meta_root->load($driver_class_spec);
}

Step 5: save LCDd model

Still with me ?

Good. Here’s the last part, now that the model was created in memory, let’s save it in files. Config::Model::Itself constructor returns an object to read or write the data:

my $rw_obj = Config::Model::Itself
     ->new( model_object => $meta_root );
say "Writing all models in file (please wait)";
$rw_obj->write_all( 
    model_dir => 'lib/Config/Model/models/'
);

That’s it. Once this script is run, the directory lib/Config/Model/models/ contains a bunch of files:

• The main LCDd model
• All models for the drivers

You can see the resulting editor in my first blog on lcdproc.

Conclusion and future steps (with you?)

As mentioned at the end of this blog, the result of this model generation process is far from perfect, because the information retrieved from LCDd.conf template are not complete for this task (which is normal since these comments were targeted for users).

Now, I’d really like to discuss with project developers to improve configuration template to generate better models.

For project owner, such an improvement would give them:

• command lines to validate configuration
• graphical configuration editor with integrated help
• Configuration doc generated from the information stored in the template file

Please leave a comment on this blog (or send a message to config-model-users at lists.sourceforge.net) if you are interested in configuration tool generation for a project you contribute to.

All the best