I've done various bits with ARMMP and LAVA on this blog already, usually waiting until I've got all the issues ironed out before writing it up. However, this time I'm just going to do a dump of where it's at, how it works and what can be done.

I'm aware that LAVA can seem mysterious at first, the package description has improved enormously recently, thanks to exposure in Debian: LAVA is a continuous integration system for deploying operating systems onto physical and virtual hardware for running tests. Tests can be simple boot testing, bootloader testing and system level testing, although extra hardware may be required for some system tests. Results are tracked over time and data can be exported for further analysis.

The LAVA documentation has a glossary of terms like result bundle and all the documentation is also available in the lava-server-doc package.

The goal is to validate the dtbs built for the Debian ARMMP kernel. One of the most accessible ways to get the ARMMP kernel onto a board for testing is tftp using the Debian daily DI builds. Actually using the DI initrd can come later, once I've got a complete preseed config so that the entire install can be automated. (There are some things to sort out in LAVA too before a full install can be deployed and booted but those are at an early stage.) It's enough at first to download the vmlinuz which is common to all ARMMP deployments, supply the relevant dtb, partner those with a minimal initrd and see if the board boots.

The first change comes when this process is compared to how boards are commonly tested in LAVA - with a zImage or uImage and all/most of the modules already built in. Packaged kernels won't necessarily raise a network interface or see the filesystem without modules, so the first step is to extend a minimal initramfs to include the armmp modules.

apt install pax u-boot-tools

The minimal initramfs I selected is one often used within LAVA:

wget http://images.armcloud.us/lava/common/linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot

It has a u-boot header added, as most devices using this would be using u-boot and this makes it easier to debug boot failures as the initramfs doesn't need to have the header added, it can simply be downloaded to a local directory and passed to the board as a tftp location. To modify it, the u-boot header needs to be removed. Rather than assuming the size, the u-boot tools can (indirectly) show the size:

$ ls -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
-rw-r--r-- 1 neil neil  5179571 Nov 26  2013 linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot

$ mkimage -l linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot
Image Name:   linaro-image-minimal-initramfs-g
Created:      Tue Nov 26 22:30:49 2013
Image Type:   ARM Linux RAMDisk Image (gzip compressed)
Data Size:    5179507 Bytes = 5058.11 kB = 4.94 MB
Load Address: 00000000
Entry Point:  00000000

Referencing http://www.omappedia.com/wiki/Development_With_Ubuntu, the header size is the file size minus the data size listed by mkimage.

5179571 - 5179507 == 64

So, create a second file without the header:

dd if=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz.u-boot of=linaro-image-minimal-initramfs-genericarmv7a.cpio.gz skip=64 bs=1

decompress it:

gunzip linaro-image-minimal-initramfs-genericarmv7a.cpio.gz

Now for the additions:

dget http://ftp.uk.debian.org/debian/pool/main/l/linux/linux-image-3.14-1-armmp_3.14.12-1_armhf.deb

(Yes, this process will need to be repeated when this package is rebuilt, so I'll want to script this at some point.):

dpkg -x linux-image-3.14-1-armmp_3.14.12-1_armhf.deb kernel-dir
cd kernel-dir

Pulling in the modules we need for most needs, comes thanks to a script written by the Xen folks. The set is basically disk, net, filesystems and LVM.

find lib -type d -o -type f -name modules.\*  -o -type f -name \*.ko \
 \( -path \*/kernel/lib/\* -o  -path \*/kernel/crypto/\* \
 -o  -path \*/kernel/fs/mbcache.ko -o  -path \*/kernel/fs/ext\* \
 -o  -path \*/kernel/fs/jbd\* -o  -path \*/kernel/drivers/net/\* \
 -o  -path \*/kernel/drivers/ata/\* -o  -path \*/kernel/drivers/scsi/\* \
 -o -path \*/kernel/drivers/md/\* \) | pax -x sv4cpio \
 -s '%lib%/lib%' -d -w >../cpio
gzip -9f cpio

original Xen script (GPL-3+)

I found it a bit confusing that i is used for extract by cpio, but that's how it is. Extract the minimal initramfs to a new directory:

sudo cpio -id < ../linaro-image-minimal-initramfs-genericarmv7a.cpio

Extract the new cpio into the same location. (Yes, I could do this the other way around and pipe the output of find into the already extracted location but that's for when I get a script to do this):

sudo cpio --no-absolute-filenames -id < ../ramfs/cpio

CPIO Manual

Use newc format, the new (SVR4) portable format, which supports file systems having more than 65536 i-nodes. (4294967295 bytes) (41M)

find . | cpio -H newc -o > ../armmp-image.cpio

... and add the u-boot header back:

mkimage -A arm -T ramdisk -C none -d armmp-image.cpio.gz debian-armmp-initrd.cpio.gz.u-boot

Now what?

Now send the combination to LAVA and test it.

Results bundle for a local LAVA test job using this technique. (18k)

submission JSON - uses file:// references, so would need modification before being submitted to LAVA elsewhere.

complete log of the test job (72k)

Those familiar with LAVA will spot that I haven't optimised this job, it boots the ARMMP kernel into a minimal initramfs and then expects to find apt and other tools. Actual tests providing useful results would use available tools, add more tools or specify a richer rootfs.

The tests themselves are very quick (the job described above took 3 minutes to run) and don't need to be run particularly often, just once per board type per upload of the ARMMP kernel. LAVA can easily run those jobs in parallel and submission can be automated using authentication tokens and the <code>lava-tool</code> CLI. ` lava-tool <https://packages.debian.org/jessie/lava-tool>`_ can be installed without <code>lava-server</code>, so can be used in hooks for automated submissions.


That's just one DTB and one board. I have a range of boards available locally:

  • iMX6Q Wandboard (used for this test)
  • iMX.53 Quick Start Board (needs updated u-boot)
  • Beaglebone Black
  • Cubie2
  • CubieTruck
  • arndale (no dtb?)
  • pandaboard

Other devices available could involve ARMv7 devices hosted at www.armv7.com and validation.linaro.org - as part of a thank you to the Debian community for providing the OS which is (now) running all of the LAVA infrastructure.

That doesn't cover all of the current DTBs (and includes many devices which have no DTBs) so there is plenty of scope for others to get involved.

Hopefully, the above will help get people started with a suitable kernel+dtb+initrd and I'd encourage anyone interested to install lava-server and have a go at running test jobs based on those so that we start to build data about as many of the variants as possible.

(If anyone in DI fancies producing a suitable initrd with modules alongside the DI initrd for armhf builds, or if anyone comes up with a patch for DI to do that, it would help enormously.)

This will at least help Debian answer the question of ` what the Debian ARMMP package can actually support <https://lists.debian.org/debian-devel/2014/07/msg00833.html>`_

For help on LAVA, do read through the documentation and then come to us at #linaro-lava or the linaro-validation mailing list or file bugs in Debian: reportbug lava-server.

Going to debconf14

so you can ask me.

I'm giving one talk on the LAVA software and there will be a BoF on validation and CI in Debian.