Sep 05 2012
1 Comment
By jclehner /var/blog

Faster emerge on the Raspberry Pi, Part I

Introduction

Running Gentoo on your Raspberry Pi can be quite a pain after realizing that that emerge you’ve started 5 minutes ago is still struggling with src_configure in the first ebuild. Even having your PORTDIR in a SquashFS image and using a reasonably fast SD card, things still aren’t fast. Using an external USB HDD might very well save that SD card of yours from croaking a tad too early, but it’s still slow:

root@pi # hdparm -t /dev/mmcblk0
/dev/mmcblk0:
Timing buffered disk reads: 56 MB in 3.00 seconds = 18.66 MB/sec

root@pi # hdparm -t /dev/sda
/dev/sda:
Timing buffered disk reads: 64 MB in 3.02 seconds = 21.21 MB/sec

So you start looking for alternatives, and things like cross-compiling with crossdev or a qemu-user chroot come to mind. The first one is indeed the fastest option, but quite a few ebuilds will need a little tinkering in order to get them working, as the dreaded configure: error: cannot run test program while cross compiling is all too common. The second option looks nice and easy on paper but, as one user in the RPi forums put it, it’s abysmally slow.

I wasn’t able to find an armv6j binhost, but even if there was one, things are much less gentooey if you have to accept whatever USE flags a certain package has been blessed with. And then there’s that moment when you realize that you’ve had another ARM powered computer at your disposal for quite some time now, without giving it too much thought: your Android phone (yes, that also applies to your iPhone, but no, it won’t do for our purposes). Building packages on your Android phone admittedly sounds a bit crazy – and it is – but this is more of a proof-of-concept post than a true alternative.

To give you an idea of the speeds I’m talking about, here are the results of running time emerge sys-apps/parted on various machines, showing only the real time part.

  •  1m50.324s:crossdev on Intel Core2 Duo E6750 @ 2.66 GHz, 4GB RAM, ordinary HDD
  • 10m38.965s:HTC Desire running Sandvold’s ICS ROM (v0.17.1)  w/ stock kernel, smartassv2 governor, max. CPU frequency of 1075 MHz
  • 10m58.522s:crossdev on Intel Pentium III 733 MHz, 256 MB RAM, ordinary HDD
  • 26m29.839s:Raspberry Pi, not overclocked, w/ Samsung Plus 8GB SDHC Card
  • 27m19.574s:qemu-user chroot on the Intel Core2 Duo machine

The above samples were collected with CFLAGS="-O2 -pipe -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard" and MAKEOPTS="-j1".

The ingredients required to cook a delicious chroot for your phone are quite straightforward:

  1. A rooted phone (surprise!) with an ARM CPU ≥ ARMv6 with VFP (if you want to use hardfloat)
  2. 1-2 GB of space left on your SD card
  3. A kernel with FUSE, and loop-device support (optional, but highly recommended)

Requirement 3. is not vital, but it sure makes things a lot easier:

  • FUSE support enables you to mount the portage tree using a SquashFS image (via squashfuse) – first of all this saves quite a lot of disk space, and secondly, it’ll significantly reduce the number of  read/write cycles your SD card has to endure. If your kernel supports squashfs out of the box, even better – still, having FUSE is cool as it allows for a few neat tricks later on.
  • Having a loop device allows you to store the disk image of your chroot on /sdcard/, rather than having to store it on /sd-ext/ or /data/ or any other place you can think of that might just have enough storage left.

Code, or GTFO!

1. Checking requirements

Enough of the blabbering, let’s get to it. Firstly, we’ll need to verify that our phone meets the specified requirements. For this, you’ll need to install the Android SDK, which provides adb.

user@pc $ adb -d shell cat /proc/cpuinfo | grep -P '^Processor|^Features'
Processor	: ARMv7 Processor rev 2 (v7l)
Features	: swp half thumb fastmult vfp edsp thumbee neon vfpv3

CPU ≥ ARMv6 w/ VFP – we’re in business, baby! Now how about supported filesystems?

user@pc> $ adb -d shell cat /proc/filesystems | grep -oP '(fuse|squashfs|ext\d|yaffs\d*)\W'
ext3
ext2
ext4
fuse
yaffs
yaffs2

We have useable filesystems (any ext or yaffs), now let’s check loop device support. Note that piping the output from adb shell does not seem to work, so we pull the file and zcat it later.

user@pc adb pull /proc/config.gz /tmp/config.gz
165 KB/s (13027 bytes in 0.076s)
user@pc zcat /tmp/config.gz | grep CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_LOOP=y

2. Creating an image from a stage3 tarball

This section should not require lengthy explanations. We’re using ext4 for our image, but any other ext or yaffs will do. Note that we run tar as root, otherwise not all permissions (suid for example) will be preserved. If your phone’s kernel doesn’t support FUSE or SquashFS, or you don’t want to run out of space too early, consider creating an image larger than 1 GB, as things might get pretty crammed.

user@pc $ mkdir gentoo-android-chroot && cd gentoo-android-chroot
user@pc $ wget ftp://[...]/source/releases/arm/autobuilds/[...]/stage3-armv6j_hardfp-YYYYMMDD.tar.bz2
[a lot of noise]
user@pc $ dd if=/dev/zero of=chroot.img bs=1M count=1024 
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 10.4049 s, 103 MB/s
user@pc $ mkfs.ext4 -Fq chroot.img
user@pc $ mkdir temproot
user@pc $ mount -o loop -t ext4 chroot.img temproot/
user@pc $ sudo tar xpjf stage3-armv6j_hardfp-YYYYMMDD.tar.bz2 -C temproot/
user@pc $ umount temproot/

Now let’s push the image to the phone’s SD card. Using a cardreader here might speed up things dramatically, as adb push’s speed is roughly 1200 KB/s on my phone.

user@pc $ adb push chroot.img /sdcard/
1260 KB/s (1073741824 bytes in 831.667s)
user@pc $ adb shell

The following code assumes that you have busybox installed on your phone, otherwise most commands won’t be available. If you don’t, you could compile busybox with USE="static" on your Pi and then push the binary to your phone.

user@phone $ su
root@phone # cd /mnt/sdcard/
root@phone # mkdir gentoo-chroot
root@phone # LOOPDEV=$(losetup -f)
root@phone # losetup $LOOPDEV chroot.img
root@phone # mount -t ext4 $LOOPDEV gentoo-chroot/

The image is mounted, but we still need to mount some stuff to make it all work!

root@phone # cd gentoo-chroot/
root@phone # mount -o bind /dev dev
root@phone # mkdir -p dev/shm
root@phone # mount -t tmpfs shm dev/shm
root@phone # mount -t proc none proc
root@phone # mount -t sys none sysfs

Android does not provide /etc/resolv.conf. We can either get dns servers from getprop, or by doing a DNS query using nslookup. Or if you’re lazy, you could use something like OpenDNS.

root@phone # getprop | grep .dns
[dhcp.wlan0.dns1]: [1.1.1.1]
[dhcp.wlan0.dns2]: [2.2.2.2]
[dhcp.wlan0.dns3]: []
[dhcp.wlan0.dns4]: []
[dhcp.wlan0.ndns]: [3.3.3.3 4.4.4.4]
[net.change]: [net.dnschange]
[net.dns1]: [5.5.5.5]
[net.dns2]: [6.6.6.6]
[net.dnschange]: [18]
[net.rmnet0.dns1]: [7.7.7.7]
[net.rmnet0.dns2]: [8.8.8.8]
[net.wlan0.dns1]: []
[net.wlan0.dns2]: [9.9.9.9]
root@phone # nslookup example.com | head -n1 | awk '{ print $2 }'
2.2.2.2
root@phone # echo nameserver 2.2.2.2 > etc/resolv.conf

Inside the chroot, commands will not work as expected at first, as Android uses a non-standard PATH. Luckily, /usr/sbin/env-update does the trick.

root@phone # chroot . /bin/bash
root@chroot # /usr/sbin/env-update && source /etc/profile
[ some errors you can safely ignore]
root@chroot # ln -sf /etc/mtab /proc/mounts
root@chroot # cp /usr/share/zoneinfo/[your timezone] /etc/localtime

3. The chroot is up – now what?

Now it’s time to edit /etc/make.conf. Here’s mine (headless system used as router, firewall, soundserver and NAS using samba).

CHOST="armv6j-hardfloat-linux-gnueabi"

CFLAGS="-O2 -pipe -mcpu=arm1176jzf-s -mfpu=vfp -mfloat-abi=hard"
CXXFLAGS="${CFLAGS}"

# /usr/portage/ will be mounted readonly, as we're using
DISTDIR="/usr/portage-distfiles/"
PKGDIR="/usr/portage-packages/"

ACCEPT_KEYWORDS="~arm"

FEATURES="buildpkg nodoc"
USE="minimal udev bindist avahi ipv6 alsa pulseaudio dbus \
-gpm -introspection -glib -svg -gdbm -nls -fortran -python \
-perl -X -qt -qt3 -qt4 -kde -kde3 -kde4 -gtk -gnome"

Also, we need to set a profile…

root@chroot # eselect profile list | grep armv6j
  [25]  default/linux/arm/10.0/armv6j
  [26]  default/linux/arm/10.0/armv6j/desktop
  [27]  default/linux/arm/10.0/armv6j/desktop/gnome
  [28]  default/linux/arm/10.0/armv6j/desktop/kde
  [29]  default/linux/arm/10.0/armv6j/developer
  [30]  default/linux/arm/10.0/armv6j/server
root@chroot # eselect profile set 25

To continue, we need the current portage tree. If your kernel doesn’t support fuse, you’ll have to bite the bullet and emerge --sync (or extract a portage snapshot).

3.1. Portage tree on SquashFS

We need to compile fuse and, if your kernel doesn’t support squashfs, squashfuse too. If neither applies to you, just skip to the next block of code.

root@chroot # mkdir build/
root@chroot # wget http://sourceforge.net/projects/fuse/files/fuse-2.X/2.9.1/fuse-2.9.1.tar.gz
[...]
root@chroot # tar xfz fuse-2.9.1.tar.gz && cd fuse-2.9.1
root@chroot # ./configure && make && make install
[...]
root@chroot # cd ..
root@chroot # wget https://github.com/downloads/vasi/squashfuse/squashfuse-0.1.tar.gz
[...]
root@chroot # tar xfz squashfuse-0.1.tar.gz && cd squashfuse.tar.gz
root@chroot # ./configure && make && make install
[...]

Up-to-date SquashFS’d snapshots of the portage tree can be downloaded from http://binhost.ossdl.de/.

root@chroot # mkdir -p /usr/portage/ /usr/portage-squashfs
root@chroot # wget http://binhost.ossdl.de/portage-YYYYMMDD.squashfs40 -O /usr/portage-squashfs/snapshot.squashfs

On a kernel with squashfs support, run

root@chroot # mount -o loop /usr/portage-squashfs/portage-squashfs/snapshot.squashfs /usr/portage/

otherwise, run

root@chroot # squashfuse /usr/portage-squashfs/snapshot.squashfs /usr/portage/

We can now remove the temporary versions of fuse and squashfuse and install proper ebuilds instead.

root@chroot # cd /root/build/fuse-* && make uninstall
[...]
root@chroot # cd /root/build/squashfuse-* && make uninstall
[...]

Running emerge now is guaranteed to fail, as portage downloads distfiles as portage:portage, not root:root, and Android by default prevents all users but root from accessing the net. Instead, it uses the groups aid_inet and aid_inet_raw to hand out network access to selected users.

root@chroot # groupadd --gid 3003 aid_inet
root@chroot # groupadd --gid 3004 aid_inet_raw
root@chroot # usermod -aG aid_inet portage

Now install proper ebuilds for fuse and squashfuse.

root@chroot # emerge fuse
[...]

Currently, squashfuse is not in the portage tree, so we will create an overlay and put our ebuild there:

root@chroot # echo PORTDIR_OVERLAY="/usr/local/portage" >> /etc/make.conf
root@chroot # mkdir -p /usr/local/portage/sys-fs/squashfuse/
root@chroot # cat > /usr/local/portage/sys-fs/squashfuse/squashfuse-0.1.ebuild <<EOF
EAPI=4

DESCRIPTION="SquashFS for FUSE"
HOMEPAGE="https://github.com/vasi/squashfuse"
SRC_URI="https://github.com/vasi/squashfuse/downloads/${P}.tar.gz"

LICENSE="BSD-2"
SLOT="0"
KEYWORDS="arm"
IUSE="+lzma zlib lzo"

DEPEND="
	>=sys-fs/fuse-2.8.6
	lzma? ( >=app-arch/xz-utils-5.0.4 )
	zlib? ( >=sys-libs/zlib-1.2.5-r2 )
	lzo? ( >=dev-libs/lzo-2.06 )
"

RDEPEND="${DEPEND}"

pkg_setup() {
	if ! use lzma && ! use zlib && ! use lzo; then
		die "Must specify one of lzma/zlib/lzo use-flags!"
	fi
}
EOF
root@chroot # ebuild /usr/local/portage/sys-fs/squashfuse/* digest
>>> Creating Manifest for /usr/local/portage/sys-fs/squashfuse/squashfuse-0.1.ebuild
root@chroot # emerge squashfuse
[...]

If you want to be able to mount squashfs images with mount -t squashfs, you can create
/usr/sbin/mount.squashfs with the following contents:

#!/bin/bash

if grep -q 'squashfs' /proc/filesystems; then
	exec mount -i $*
else
	dev=$1; shift
	mnt=$1; shift
	exec squashfuse $*,ro,nonempty "${dev}" "${mnt}"
fi

Be sure to chmod a+x it, otherwise mount will fail silently. Now, we can update /etc/fstab:

root@chroot # echo '/usr/portage-squashfs/snapshot.squashfs /usr/portage squashfs auto 0 0' >> /etc/fstab
3.2 I’m Ready to rock – no wait! I’m running out of space!

A stage3 will set you back by about 500 MB, so if you were as cheap as I was and created a 1 GB image, it’s time for some janitorial action. Some directories to consider are:

/usr/share/doc
/usr/share/man
/usr/share/info
/usr/share/gtk-doc
/usr/share/{gcc,binutils}-data/*/*/locale
/usr/share/i18n/locale

Clearing these directories will give you ~150 MB of additional space, but if you do some digging, there sure is more stuff you’ll never need in a chroot environment. Also consider USE="-nls" and FEATURES="nodoc". If that still isn’t enough, there are more exotic options such as mounting your PKGDIR using sshfs so the packages you’re building won’t eat up even more space.

3.3 Setting up sshd

If you’ve made it this far, this should require no explanations. If you want to fire it up using the init script, be sure to create /lib/rc/init.d/softlevel first.

root@chroot # touch /lib/rc/init.d/softlevel
root@chroot # /etc/init.d/sshd --nodeps start

4. I’m done. So how exactly does this affect my Raspberry Pi?

Okay, admittedly it was all about a Gentoo chroot on your Android phone for now, but we’re getting there. You now have a gentoo system on your Android phone, which will enable you to build binpkgs and push them to the Pi – after all, that emerge was twice as fast on the phone. Also, if you have some quadcore powered monster of a phone with a shitload of RAM and storage, things are guaranteed to be even faster. After all, my HTC Desire is now 2½ years old.

The next post will be is about how you can (ab)use the chroot we just created to build packages for your Pi on demand. Or, if you’re really brave and crazy, and you don’t wanna wait, you might try something like

root@chroot # cd $(portageq envvar PORTDIR)
root@chroot # ALL_EBUILDS="$(find . -maxdepth 2 | sed -re 's:^\./::')"
root@chroot # ALL_EBUILDS=$(grep -vP 'eclass|licenses|metadata|profiles|virtual' <<< "$ALL_EBUILDS")
root@chroot # emerge $ALL_EBUILDS

but I’d suggest you don’t 😉

Tagged , , ,

One thought on “Faster emerge on the Raspberry Pi, Part I”

  1. Bosstiger says:
    September 7, 2012 at 09:24

    Reblogged this on Gigable – Tech Blog.

Comments are closed.